Sim-to-Real Intuition Lab IIRandomize to Generalize

You can't fix the world, so stop training in only one

Chapter 2.6 handed you a measuring instrument and a verdict. You took a policy raised in a flawless simulator, injected the perturbations that dominate the reality gap, and watched it degrade. The lesson landed as a question: a policy trained in one clean world has no defense it was never asked to learn — so what do you do about it?

You cannot fix the real world to match your simulator. The robot's motors will always be a little weaker than the model, its floor a little more or less grippy, its payload never exactly the nominal mass. So do the opposite. Stop making the simulator one fixed world. Randomize its dynamics every episode — sample a different mass, a different friction, a different gravity each time the policy resets — and the policy is forced to learn a behavior that works across the whole range instead of memorizing the one point it was handed. Meet the reality gap in sim, where falling over is free.

That is domain randomization, and it is the workhorse behind the famous sim-to-real results — Tobin's randomized rendering for object detection, OpenAI's in-hand cube that trained entirely in a randomized sim and transferred to a real Shadow Hand. This chapter builds it from scratch on the quadruped, trains two policies that differ in exactly one thing, and then does the part that separates engineering from wishful thinking: it measures whether it worked.

Two policies, one difference

We train two PPO policies on the quadruped — the same from-scratch PPO you wrote in chapter 2.1, unchanged. The only thing that differs between them is the world they train in:

• NARROW trains on the nominal dynamics, every episode. The mass, friction, and gravity are always the book values.
• RANDOMIZED resamples the torso/leg masses, the foot friction, and gravity from a band around nominal at the start of every episode.

Domain randomization, in code, is nothing more than scaling the simulator's physical parameters in place — on top of the contact solver the quadruped env pins. We touch mass, foot friction, and gravity; we never touch the solver iterations or the contact-cone settings the env README fixes for determinism, so the whole thing stays bitwise-reproducible per seed.

# Domain randomization = scaling the env's PHYSICAL parameters, in place, on top# of the pinned contact solver. We touch only mass/inertia (heavier or lighter# robot), foot friction (a slick vs grippy floor), and gravity (a heavier or# lighter world) — never the solver iterations/cone the quadruped README pins, so# the bitwise-CPU determinism guarantee still holds per fixed seed. Half-widths# below are the nominal band at --dr_width 1.0; dr_width scales them (0 => narrow).MASS_HW, FRIC_HW, GRAV_HW = 0.4, 0.5, 0.25   # mass +-40%, friction +-50%, gravity +-25%SCALE_FLOOR = 0.2                            # keep every scale physical (no zero-mass / frictionless / zero-g)  def capture_nominal(env: QuadrupedEnv) -> dict:    """Snapshot the env's as-built physical params so every episode scales from    the SAME nominal baseline (scaling in place would otherwise compound)."""    return {"mass": env.model.body_mass.copy(), "inertia": env.model.body_inertia.copy(),            "friction": env.model.geom_friction.copy(), "gravity": float(env.model.opt.gravity[2])}  def sample_scales(rng: np.random.Generator, width: float) -> dict:    """Draw one (mass, friction, gravity) scale triple for an episode, uniform in    the band around 1.0. width 0 collapses the band to a point => all scales 1.0    => the NARROW policy trains on nominal dynamics every episode."""    def draw(half_width: float) -> float:        return max(SCALE_FLOOR, float(rng.uniform(1.0 - width * half_width, 1.0 + width * half_width)))    return {"mass": draw(MASS_HW), "friction": draw(FRIC_HW), "gravity": draw(GRAV_HW)}  def apply_scales(env: QuadrupedEnv, nominal: dict, scales: dict) -> None:    """Write scaled params back into the compiled MjModel (body 0 is the world —    leave its zero mass alone; friction column 0 is the tangential coefficient)."""    env.model.body_mass[1:] = nominal["mass"][1:] * scales["mass"]    env.model.body_inertia[1:] = nominal["inertia"][1:] * scales["mass"]    env.model.geom_friction[:, 0] = nominal["friction"][:, 0] * scales["friction"]    env.model.opt.gravity[2] = nominal["gravity"] * scales["gravity"]

A single knob, --dr_width, scales the band. --dr_width 0 collapses it to a point — that is the narrow policy, so both policies run through the identical code path and differ only in one number. Re-seeding both trainings from the same --seed means they start from the identical network initialization and see the same reset noise; the only variable in the entire experiment is whether the dynamics move. That is a controlled experiment, not a demo.

def train_policy(width: float, label: str) -> Agent:    """Train ONE PPO policy end to end and return it. Called twice — once with    width 0 (NARROW) and once with args.dr_width (RANDOMIZED). Re-seeds torch and    the env/DR streams from args.seed at entry, so the two policies start from the    IDENTICAL init and see the same reset noise; the ONLY difference between them    is whether the dynamics get randomized. That is the controlled experiment."""    torch.manual_seed(args.seed)    envs = [QuadrupedEnv() for _ in range(args.num_envs)]    nominal = [capture_nominal(e) for e in envs]    dr_rng = np.random.Generator(np.random.PCG64(args.seed + 1234))  # the DR draw stream, seeded    episode_count = np.zeros(args.num_envs, dtype=np.int64)     def env_reset(i: int) -> np.ndarray:        # each env/episode gets its own reset seed, then fresh randomized dynamics        obs = envs[i].reset(seed=args.seed + i * 1000 + int(episode_count[i]))        apply_scales(envs[i], nominal[i], sample_scales(dr_rng, width))        episode_count[i] += 1        return obs     agent = Agent(OBS_DIM, ACT_DIM, args.hidden_dim).to(device)    optimizer = torch.optim.Adam(agent.parameters(), lr=args.lr, eps=1e-5)    obs_buf = torch.zeros((args.num_steps, args.num_envs, OBS_DIM), device=device)    act_buf = torch.zeros((args.num_steps, args.num_envs, ACT_DIM), device=device)    logp_buf = torch.zeros((args.num_steps, args.num_envs), device=device)    rew_buf = torch.zeros((args.num_steps, args.num_envs), device=device)    val_buf = torch.zeros((args.num_steps, args.num_envs), device=device)    term_buf = torch.zeros((args.num_steps, args.num_envs), device=device)   # fell (terminal)    done_buf = torch.zeros((args.num_steps, args.num_envs), device=device)   # fell OR time-limit    boot_buf = torch.zeros((args.num_steps, args.num_envs), device=device)   # V(real next state) at episode ends     next_obs = np.stack([env_reset(i) for i in range(args.num_envs)])    ep_return = np.zeros(args.num_envs, dtype=np.float64)    recent: list[float] = []    for iteration in range(1, num_iterations + 1):        optimizer.param_groups[0]["lr"] = args.lr * (1.0 - (iteration - 1.0) / num_iterations)  # anneal        for step in range(args.num_steps):            obs_t = torch.as_tensor(next_obs, dtype=torch.float32, device=device)            obs_buf[step] = obs_t            with torch.no_grad():                action, logp, _, value = agent.get_action_and_value(obs_t)            act_buf[step], logp_buf[step], val_buf[step] = action, logp, value            action_np = action.cpu().numpy()            next_obs = np.empty_like(next_obs)            for i in range(args.num_envs):                obs_i, reward, done, info = envs[i].step(action_np[i])                rew_buf[step, i] = reward                term_buf[step, i] = float(info["terminated"])  # a FALL is terminal; a time-limit is not                done_buf[step, i] = float(done)                ep_return[i] += reward                if done:                    with torch.no_grad():  # value of the state we're leaving — bootstrap on truncation, masked on a fall                        boot_buf[step, i] = agent.get_value(                            torch.as_tensor(obs_i, dtype=torch.float32, device=device).unsqueeze(0)).item()                    recent.append(ep_return[i])                    ep_return[i] = 0.0                    obs_i = env_reset(i)  # autoreset with fresh randomized dynamics                next_obs[i] = obs_i         with torch.no_grad():            next_value = agent.get_value(torch.as_tensor(next_obs, dtype=torch.float32, device=device))        advantages = torch.zeros_like(rew_buf)        last_gae = torch.zeros(args.num_envs, device=device)        for t in reversed(range(args.num_steps)):  # GAE; 1-terminated masks the bootstrap so a FALL earns no future value            next_v = next_value if t == args.num_steps - 1 else val_buf[t + 1]            next_v = torch.where(done_buf[t].bool(), boot_buf[t], next_v)            delta = rew_buf[t] + args.gamma * next_v * (1.0 - term_buf[t]) - val_buf[t]            last_gae = delta + args.gamma * args.gae_lambda * (1.0 - done_buf[t]) * last_gae            advantages[t] = last_gae        returns = advantages + val_buf         b_obs, b_act = obs_buf.reshape(-1, OBS_DIM), act_buf.reshape(-1, ACT_DIM)        b_logp, b_adv = logp_buf.reshape(-1), advantages.reshape(-1)        b_ret = returns.reshape(-1)        for _ in range(args.update_epochs):            for start in range(0, batch_size, minibatch_size):                mb = torch.randperm(batch_size, device=device)[start:start + minibatch_size]                _, new_logp, entropy, new_val = agent.get_action_and_value(b_obs[mb], b_act[mb])                ratio = (new_logp - b_logp[mb]).exp()                adv = (b_adv[mb] - b_adv[mb].mean()) / (b_adv[mb].std() + 1e-8)  # normalize advantages                pg_loss = torch.max(-adv * ratio,                                    -adv * torch.clamp(ratio, 1 - args.clip_coef, 1 + args.clip_coef)).mean()                v_loss = 0.5 * ((new_val.flatten() - b_ret[mb]) ** 2).mean()                loss = pg_loss + args.vf_coef * v_loss                optimizer.zero_grad()                loss.backward()                nn.utils.clip_grad_norm_(agent.parameters(), args.max_grad_norm)                optimizer.step()        mean_return = float(np.mean(recent[-50:])) if recent else float("nan")        if args.rerun:            rr.set_time("global_step", sequence=iteration * batch_size)            rr.log(f"train/{label}/episodic_return", rr.Scalars([mean_return]))        if iteration % 10 == 0 or iteration == num_iterations:            print(f"  [{label}] iter {iteration:3d}/{num_iterations}  mean_return {mean_return:6.1f}")    return agent

Evaluate across the gap — with error bars

Training return is not the point; generalization is. So we sweep a test-dynamics axis — by default the robot's mass — from light to crushingly heavy, and evaluate BOTH policies at every point on held-out seeds. For each point we report the mean return, its spread, and the survival rate: the fraction of episodes the robot stayed up for the full ten seconds. Survival is the honest binomial the reality gap moves first (chapter 1.6's discipline: a rate is a band, not a number).

def rollout(agent: Agent, env: QuadrupedEnv, seed: int) -> tuple[float, bool]:    """One deterministic (policy MEAN, no sampling) episode; return (total_reward,    survived). survived = reached the time limit without falling — the honest    binary signal a brittle policy misses (return folds in walking too, but staying    up is what the reality gap threatens first)."""    obs, done, total = env.reset(seed=seed), False, 0.0    info = {"truncated": False}    while not done:        with torch.no_grad():            action = agent.actor_mean(torch.as_tensor(obs, dtype=torch.float32, device=device).unsqueeze(0))        obs, reward, done, info = env.step(action[0].cpu().numpy())        total += reward    return total, bool(info["truncated"])  # truncated == survived the full horizon (never fell)  def evaluate_at(agent: Agent, knob: str, scale: float) -> dict:    """Return mean+-std return AND survival rate over held-out episodes at ONE    fixed test-dynamics point. The std / rate are the ch1.6 honesty: a single point    estimate hides the spread, and survival is the binomial the reality gap moves."""    env = QuadrupedEnv()    nominal = capture_nominal(env)    scales = {"mass": 1.0, "friction": 1.0, "gravity": 1.0}    scales[knob] = scale  # move ONE axis; hold the others nominal so the curve is attributable    returns, survived = [], []    for ep in range(args.eval_episodes):        # Pin the test dynamics on the MODEL (mj_resetData in rollout resets the        # DATA — qpos/qvel — but never these model params, so the scale persists).        apply_scales(env, nominal, scales)        ret, ok = rollout(agent, env, EVAL_SEED0 + args.seed + ep)        returns.append(ret)        survived.append(ok)    return {"scale": scale, "mean": float(np.mean(returns)), "std": float(np.std(returns)),            "survival": float(np.mean(survived))}  def sweep(agent: Agent) -> list[dict]:    """Evaluate one policy across the whole test-dynamics grid (the 'gap')."""    return [evaluate_at(agent, args.sweep_knob, s) for s in sweep_grid]

The sweep deliberately runs past the randomization band, into the gap where neither policy trained. That is the "break-the-policy" playground the map promises: crank the test dynamics until something falls, and see which policy falls first.

Run it

python curriculum/phase2_reinforcement/ch2.7_dr/dr.py --seed 0

This trains both policies from scratch and runs the full sweep across the mass gap. Here is the generalization curve on the default --seed 0 (mean return ± std | survival rate, 16 held-out episodes):

  mass_scale        0.8           1.0           1.2           1.4           1.6
  narrow       206±1|1.00   206±0|1.00   112±82|0.44    32±1|0.00    27±1|0.00
  randomized   203±0|1.00   203±0|1.00   100±79|0.38    29±1|0.00    24±0|0.00

On this seed the two curves lie almost on top of each other — DR bought nothing. But hold that thought: seeds 1 and 2 tell a different story, and the difference between them is the point. Now read it honestly, because the honest reading is the whole chapter.

The result the map didn't promise

Here is what the measurement actually says, and it is subtler than "randomization wins."

Near nominal, both policies stand, and randomization buys nothing the narrow policy didn't already have. Standing near the book dynamics is a stable equilibrium — the PD servos hold a crouch, and a small change in mass is a small push the same feedback absorbs. The narrow policy generalizes across the easy part of the range for free; there is nothing there for domain randomization to fix, and both policies survive every episode at 0.8× and 1.0× mass.

Deeper in the gap, domain randomization sometimes pays off spectacularly — and sometimes not at all. On seed 1, the randomized policy holds a full-survival, ~200-return stance all the way out to 1.4× mass, a load under which the narrow policy has already collapsed (0.38 survival). That is the textbook result: the narrow policy overfit the one stance that works at nominal, and the randomized policy, forced to cope with heavier and lighter bodies in training, learned a sturdier one that carries into the gap. It even shows the "1.2× ceiling" was never physical — it was the narrow policy's ceiling, and DR pushed past it. But run seed 0, or seed 2, and the edge evaporates: the randomized policy converges to an ordinary nominal stand and falls right alongside the narrow one. Across three seeds the survival edge across the gap is −0.02, +0.22, −0.09.

That spread IS the finding. The average benefit lives inside the seed band — exactly the trap chapter 1.6 taught you to catch. If you had trained one seed, seen +0.22, and written "domain randomization extends the robust range by 40%," you would have published noise. The honest statement is: at this compute budget, DR can produce a dramatically more robust policy, but it does so unreliably — because randomizing the dynamics makes the training problem strictly harder, and 400k steps is not always enough to solve the harder problem. Randomization is not free insurance; it is a harder problem that needs more compute to pay off dependably. (Cut the step budget below this and it is worse still: the randomized policy sometimes fails to learn even the nominal stand.)

So when does domain randomization help — reliably?

This is the judgment the chapter exists to build. Domain randomization pays off, and pays off dependably, when two conditions both hold:

• the narrow policy is genuinely brittle to the axis you randomize — the optimal behavior is dynamics-specific. A walking gait tuned to one friction slips on another; an in-hand manipulation tuned to one mass distribution drops a different one. There, the narrow policy overfits sim-specific dynamics and DR is the fix, not a coin flip. A passively-stable stand is the weakest case: most of its range needs no adaptation at all.
• you spend enough compute to actually solve the harder randomized problem. The famous transfers — OpenAI's in-hand cube, sim-to-real locomotion — randomize hard and train long. Under-train the randomized policy and you get the seed 0 / seed 2 story: the extra difficulty, none of the payoff.

Notice how cleanly the negative half agrees with chapter 2.6: there, the worst perturbation for these simple policies was latency, not model mismatch — they already shrugged off mass and gravity. Domain randomization is the cure for model mismatch. Where model mismatch was never the disease, the cure does little. Measure the gap, and measure it with error bars, before you trust the method.

The randomization band is a hyperparameter (exercise 2)

If a wider band always meant more robustness, you would just crank --dr_width to the sky. The second exercise makes you test that folklore: sweep the band from 0 (no DR) to double-wide and watch what it buys and where it stops. A wider band does buy more robustness through the middle of the gap — on average, though noisily — and it does so without denting nominal return. But the deepest gap point stays pinned on the floor at every width: by 1.6× mass the ±12 Nm servos simply cannot hold, and no amount of sampling near an impossible load makes it possible. Randomization widens the range you can reach; it never lifts the ceiling of what the motors can do. "Randomize harder" is a real dial with a real limit, not a free path to transfer.

Determinism and honesty

Everything is seeded — torch, every env reset, and the domain-randomization draw stream — so a fixed --seed reproduces byte-for-byte on CPU (CI checks the smoke twice over), and randomizing on top of the pinned contact solver preserves that guarantee. But the conclusion wobbles seed to seed, exactly as chapter 1.6 warned: on one seed the randomized policy looks a touch more reliable, on another a touch worse. That is the tell. When the narrow-vs-randomized difference lives inside the seed band, the honest report is that you have not demonstrated a benefit — and this chapter reports that, rather than cherry-picking the seed where DR happened to win.

Read the real thing

Three papers built the intuition this chapter distills, and reading them against what you just measured is the point. Tobin et al. (2017) introduced domain randomization by randomizing a simulator's rendering — textures, lighting, camera pose — so an object detector trained purely in sim transferred to real images. Peng et al. (2018) moved the idea from pixels to physics, randomizing the dynamics — masses, friction, latency — exactly the axes dr.py scales, and transferred a manipulation policy to a real arm. OpenAI's Dactyl (2018/2019) is the headline case: an in-hand cube-reorientation policy trained entirely in a randomized sim and transferred to a physical Shadow Hand.

Read them for the contrast with what happened here. Every one of those results randomizes a behavior that is genuinely dynamics-brittle — contact-rich manipulation, a dexterous regrasp — where the optimal action really does change with the physics, so the narrow policy has no choice but to overfit and DR is the fix. And every one trains long, spending the compute to actually solve the harder randomized problem. Those are the two conditions a passively stable quadruped stand at a free-tier budget does not meet — which is precisely why your measurement came back inside the seed band. The method is real; the substrate here is honest about its limits.