Your policy has only ever lived in a perfect world
Every policy you have trained grew up somewhere flawless. The PPO balancer from chapter 2.1 read the pole's exact angle, the instant it asked, in a world whose gravity and masses were the true ones — because in a simulator they are the only ones. The SAC reacher from 2.2 had the same charmed childhood. Both score near-perfectly on the task they were trained on, and both are, in a specific and measurable way, spoiled.
A real robot lives in none of that. Its sensors are noisy. Its observations arrive a few control steps late — a camera exposes, a filter smooths, a USB bus and a message queue each cost a millisecond, and by the time the policy sees the world the world has moved. And the physical robot's masses, joint friction, and gravity never quite match the numbers the simulator used. That collection of discrepancies is the reality gap, and it is why a policy that scores 500/500 in sim can tip over on the bench.
This chapter builds hardware intuition without hardware. We take a policy trained in the clean sim, and we re-run it while injecting the three perturbations that dominate the gap — one family at a time — and we measure how far it degrades. The goal is not a number; it is a reflex. By the end you should look at a new task and guess which perturbation will hurt it most, the way an experienced roboticist does.
Three perturbations, built from scratch
There is no wrapper hiding the injection. Each perturbation is a few lines you can read, and each acts on the policy's interface with the world — never on its weights. The reality gap is entirely outside the network.
Sensor noise adds gaussian noise to the observation the policy reads. Latency delays that observation through a ring buffer, so the policy acts on a stale snapshot — where the world was, not where it is. Model mismatch scales the eval env's mass, joint damping, and gravity away from the values training saw, by editing the compiled MuJoCo model in place.
# Three injectable perturbations. Sensor noise and latency wrap the OBSERVATION# stream (what the policy reads); model mismatch scales the ENV's physics (what# the policy acts on). Nothing here touches the policy weights — the reality gap# is entirely outside the network. class ObsDelay: """A ring buffer that hands the policy an observation from `steps` control steps ago — the whole latency model in one data structure. steps=0 is a pass-through. Primed with the first observation so the first `steps` actions see the start state rather than an empty buffer (a real robot boots the same way: the estimator has no history yet).""" def __init__(self, steps: int, first_obs: np.ndarray): self.buf: deque = deque([first_obs.copy()] * (steps + 1), maxlen=steps + 1) def push_pop(self, obs: np.ndarray) -> np.ndarray: self.buf.append(obs.copy()) return self.buf[0] # oldest in the window == delayed by `steps` def apply_mismatch(env, mass_scale: float, damping_scale: float, gravity_scale: float) -> None: """Scale the eval env's physical parameters away from what training saw — an in-place edit of the compiled MjModel. mj_step reads these every step, so the dynamics change immediately; the policy is unaware its world moved. (On the planar pusher, gravity is perpendicular to the arm's motion, so gravity_scale is deliberately a NO-OP there — an honest reminder that which knob bites depends on the task's geometry.)""" env.model.body_mass[:] *= mass_scale env.model.body_inertia[:] *= mass_scale # keep mass and inertia consistent env.model.dof_damping[:] *= damping_scale env.model.opt.gravity[:] *= gravity_scale def perceive(raw_obs: np.ndarray, delay: ObsDelay, obs_noise: float) -> np.ndarray: """The full sensor pipeline for one step: delay first (the reading is stale), then add gaussian noise (the reading is also imprecise).""" obs = delay.push_pop(raw_obs) if obs_noise > 0.0: obs = obs + noise_rng.normal(0.0, obs_noise, size=obs.shape).astype(np.float32) return obsA design decision worth pausing on: the policy is always a plain
obs -> action function, and the perturbations wrap that observation stream (or
the env's physics). This is why the chapter's scripted fallback controller — the
same hand-tuned balancer and IK-reacher from the common/ envs, rebuilt here to
read the observation instead of the raw simulator state — degrades under noise
and latency exactly as a learned policy does. A controller that only ever sees
observations has no privileged access to truth, and neither does your robot.
Load a policy you trained, and perturb it
No policy binary ships in this repo (the .pt files are gitignored — root
invariant 5). You point perturb.py at a checkpoint you trained with chapter
2.1 or 2.2; with no checkpoint (and always under --smoke, for hermetic CI) it
falls back to the scripted baseline, which tells the same story.
# The policy is ALWAYS a plain obs->action function. A perturbation acts on the# observation the policy consumes (or on the env's physics) — so a learned net# and the scripted baseline are perturbed identically, through the exact same# obs the real robot's autonomy stack would see. That is why the scripted# fallback is a faithful stand-in: it, too, only ever reads observations. def load_learned(task: str, ckpt: Path): """Rebuild the eval (deterministic) forward pass of a ch2.1/2.2 policy and load its weights. We infer hidden width from the checkpoint tensors, so the same loader reads PPO's width-64 actor and SAC's width-256 actor unchanged. Only the action-producing path is reconstructed — the critic, log-std, and sampling branches are training machinery and eval never touches them.""" sd = torch.load(ckpt, map_location=device, weights_only=True) if task == "cartpole": # PPO: action = actor_mean(obs), a 3-layer Tanh MLP hidden = sd["actor_mean.0.weight"].shape[0] net = nn.Sequential( nn.Linear(TASK["obs_dim"], hidden), nn.Tanh(), nn.Linear(hidden, hidden), nn.Tanh(), nn.Linear(hidden, TASK["act_dim"]), ).to(device) net.load_state_dict({k.removeprefix("actor_mean."): v for k, v in sd.items() if k.startswith("actor_mean.")}) def act(obs: np.ndarray) -> np.ndarray: o = torch.as_tensor(obs, dtype=torch.float32, device=device).unsqueeze(0) return net(o)[0].cpu().numpy() return act # SAC: deterministic action = tanh(mean(trunk(obs))), a squashed-Gaussian mean hidden = sd["trunk.0.weight"].shape[0] # Linear(obs, hidden) weight is (hidden, obs) trunk = nn.Sequential( nn.Linear(TASK["obs_dim"], hidden), nn.ReLU(), nn.Linear(hidden, hidden), nn.ReLU(), ).to(device) mean = nn.Linear(hidden, TASK["act_dim"]).to(device) trunk.load_state_dict({k.removeprefix("trunk."): v for k, v in sd.items() if k.startswith("trunk.")}) mean.load_state_dict({k.removeprefix("mean."): v for k, v in sd.items() if k.startswith("mean.")}) def act(obs: np.ndarray) -> np.ndarray: o = torch.as_tensor(obs, dtype=torch.float32, device=device).unsqueeze(0) return torch.tanh(mean(trunk(o)))[0].cpu().numpy() return act def scripted_cartpole(obs: np.ndarray) -> np.ndarray: """The ch2.1 linear balancer, rebuilt to read the OBSERVATION (so latency and noise reach it). obs = [cart_pos, cart_vel, cos(theta), sin(theta), angvel]; same gains as common/envs/cartpole.balance_action.""" cart_pos, cart_vel, cos_t, sin_t, angvel = obs theta = math.atan2(sin_t, cos_t) u = 10.0 * theta + 2.0 * angvel + 0.4 * cart_pos + 0.8 * cart_vel return np.array([np.clip(u, -1.0, 1.0)], dtype=np.float32) def scripted_pusher(obs: np.ndarray) -> np.ndarray: """The ch2.2 analytic-IK + PD reacher, rebuilt to read the OBSERVATION. We recover the joint angles and (via forward kinematics) the target world position from the obs, then run the same closed-form IK + PD as common/envs/pusher_reach.reach_action.""" cos_sh, sin_sh, cos_el, sin_el, sh_vel, el_vel, dx, dy = obs sh, el, L = math.atan2(sin_sh, cos_sh), math.atan2(sin_el, cos_el), PusherReachEnv.LINK_LEN ftip_x = L * cos_sh + L * math.cos(sh + el) # forward kinematics of the fingertip ftip_y = L * sin_sh + L * math.sin(sh + el) tx, ty = ftip_x + dx, ftip_y + dy # target world position = fingertip + (dx, dy) cos_e = np.clip((tx * tx + ty * ty - 2.0 * L**2) / (2.0 * L**2), -1.0, 1.0) el_des = math.acos(cos_e) # elbow-down IK branch sh_des = math.atan2(ty, tx) - math.atan2(L * math.sin(el_des), L * (1.0 + cos_e)) err = np.array([_wrap(sh_des - sh), _wrap(el_des - el)]) return np.clip(25.0 * err - 2.0 * np.array([sh_vel, el_vel]), -1.0, 1.0).astype(np.float32) def _wrap(a: float) -> float: """Wrap an angle error to [-pi, pi).""" return (a + math.pi) % (2.0 * math.pi) - math.pi def build_policy(task: str, ckpt: Path | None) -> tuple: """Return (policy_fn, label). A real checkpoint -> the trained policy; no checkpoint (or --smoke) -> the scripted baseline (hermetic, needs no .pt).""" if ckpt is not None and ckpt.is_file(): return load_learned(task, ckpt), f"learned({ckpt.name})" scripted = scripted_cartpole if task == "cartpole" else scripted_pusher return scripted, "scripted-baseline"The eval loop is deliberately close to the ones you already know — reset on held-out seeds, act, step — with the perturbation spliced into perception:
def evaluate(policy, episodes: int, obs_noise: float = 0.0, latency_steps: int = 0, mass_scale: float = 1.0, damping_scale: float = 1.0, gravity_scale: float = 1.0) -> dict: """Roll out `episodes` held-out episodes under one perturbation config and return {success_rate, metric}. success_rate is the task-agnostic degradation axis (fraction of episodes that still succeed); metric is the task's natural secondary number (cartpole return, pusher final distance).""" env = TASK["env"]() apply_mismatch(env, mass_scale, damping_scale, gravity_scale) successes, metrics = [], [] for ep in range(episodes): raw = env.reset(seed=EVAL_SEED0 + args.seed + ep) delay = ObsDelay(latency_steps, raw) done, ret = False, 0.0 info = {"truncated": False, "dist": float("nan")} while not done: action = policy(perceive(raw, delay, obs_noise)) raw, reward, done, info = env.step(action) ret += reward if args.task == "cartpole": successes.append(float(info["truncated"])) # survived to the horizon (never fell) metrics.append(ret) # return == steps balanced else: successes.append(float(info["dist"] < 0.05)) # within the ch2.2 solve bar at episode end metrics.append(info["dist"]) return {"success_rate": float(np.mean(successes)), "metric": float(np.mean(metrics))}Run it: the degradation curves
python curriculum/phase2_reinforcement/ch2.6_perturb/perturb.py \
--seed 0 --task cartpole --ckpt outputs/ch2.1-ppo/ppo_agent.pt
The headline run sweeps each perturbation family from clean to broken and reports the success rate falling along the way. On a CPU laptop the full three-family sweep takes about 2.77 min (measured). For the chapter-2.1 PPO balancer (seed 0):
clean baseline: success 1.00 mean_return 500.0000
sensor_noise ( obs_noise) success@ 0:1.00 0.01:1.00 0.02:1.00 0.05:1.00 0.1:1.00 0.2:1.00
latency ( latency_steps) success@ 0:1.00 1:1.00 2:1.00 4:0.00 8:0.00
model_mismatch ( gravity_scale) success@ 0.5:1.00 0.75:1.00 1:1.00 1.25:1.00 1.5:1.00 2:1.00
worst perturbation for this policy: latency (success drop +1.00 from clean baseline)
Read that carefully, because it is more interesting than "the policy gets worse." This policy shrugs off sensor noise — even a large jitter on the pole angle barely moves it — and it shrugs off gravity mismatch, balancing fine under half or double the gravity it trained on. It learned a genuinely robust feedback controller. But it falls off a cliff at observation latency: solid through 2 steps of delay, and dead by 4 (80 ms). Balance is a stability problem, and stability problems die on delay — there is a delay margin, and past it the corrections arrive too late and amplify instead of damp.
The mirror: run the reacher too
Switch tasks and the lesson inverts. The chapter-2.2 SAC reacher, perturbed the same way (seed 0):
clean baseline: success 0.70 mean_final_dist 0.0386
sensor_noise ( obs_noise) success@ 0:0.70 ... 0.1:0.75 0.2:0.35
latency ( latency_steps) success@ 0:0.70 ... 4:0.70 8:0.50
model_mismatch ( mass_scale) success@ 0.5:0.70 1:0.70 1.5:0.70 2:0.70 3:0.70
worst perturbation for this policy: sensor_noise
The reacher is the cartpole's opposite. It tolerates latency — a settling task can afford to arrive a little late — and it is instead brittle to sensor noise, which jitters its notion of where the target is. Gravity mismatch is a genuine no-op here: the arm moves in the plane perpendicular to gravity, so scaling gravity exerts no torque on it at all (mass is its sharp knob, and even that the learned controller mostly absorbs).
That is the whole intuition this chapter exists to build: which perturbation bites is a property of the task's dynamics, not of the algorithm. Nobody could have told you "fear latency" or "fear noise" in general; you have to know what kind of control problem you are solving. A balance problem fears delay. A reach problem fears noise. You now have a way to find out for any policy, before it ever touches a robot.
Break it (optional, not graded)
--break pins the observation latency to an extreme 16 steps (320 ms) and runs a
single eval. The clean-trained policy has no memory to fill that gap and fails
outright. It is a teaching toggle, not a graded bug-hunt: per the RL doctrine
(the chapter-2.1 spike), single-run effects are noise, so the graded exercises
assert a seed-robust structural fact instead — which perturbation wins, and
that the latency cliff is real across seeds.
Determinism and honesty
Perturbed eval is noisy in two senses, and we are honest about both. The
perturbation RNG is its own seeded stream, so a fixed --seed gives
byte-identical results twice over on CPU (CI checks this). But the degradation
numbers still wobble seed to seed — the exact breaking latency moves between 2
and 4 steps depending on which clean-sim policy you trained. So this chapter, like
chapter 1.6 taught, reports the shape (solid → knee → dead) as the robust
finding, and leaves the exact cliff step as something you read, not a single
number you trust.
The seam into 2.7
There is a reason this is "Lab I." You have now measured that a policy trained in one clean world is defenseless against a gap it was never shown — and, tellingly, that the gap it fears most (latency, for the balancer) is not one you can noise- filter your way across. No amount of better sensing gives a policy back a stability margin it never learned to have.
Chapter 2.7 is the answer: domain randomization. Instead of training in one perfect world, train across a distribution of imperfect ones — randomize the masses, the delays, the noise during training — so the policy meets the reality gap in sim, where falling over is free. You have just built the measuring instrument that will prove whether it worked.
Read the real thing
The reading that pairs with this chapter is the transfer literature the perturbation knobs stand in for: the system-identification and latency work on why a policy trained in a clean sim meets the reality gap on hardware, and how production robot stacks measure and compensate for observation delay. Read it for the vocabulary that turns the mass, delay, and noise sliders you just swept into named, measured quantities on a real robot.