Everything you built, in one loop
This is the convergence chapter. Trace what you already own:
- A base policy by cloning demos (BC, 1.1) — and the covariate-shift death you measured when it drifted into states no demo covered.
- Corrections (4.2): you rolled out that policy, caught it failing, and had an expert — the scripted stand-in for your own teleop hand — label the states it actually visits. On-policy corrections, exactly where the policy is weak.
- An offline prior (the primer): given a fixed log, don't just clone it — fit a Q-function and extract the above-average actions with advantage-weighted regression (AWAC). The reward BC threw away, put to work.
- Off-policy RL (2.2): SAC, with its replay buffer, twin-Q critics, target networks, and squashed-Gaussian policy — learning from every transition many times.
HIL-SERL (Human-in-the-Loop Sample-Efficient RL, Luo et al.) is those four things composed into one loop. Take the corrections. Turn them into an offline prior with AWAC. Then fine-tune that prior online with SAC, keeping the corrections in the replay buffer the whole time — so the policy reaches a good return in far fewer online environment samples than RL starting from scratch. Sample-efficiency is the whole point, and this chapter measures it.
Why "sample-efficient" is the right obsession: online RL's currency is environment interaction, and on a real robot every environment step is a real second of real hardware wearing out. RL-from-scratch on a manipulation task can cost millions of steps. If your correction data can buy back most of that, that is the difference between "trains overnight on a robot in a lab" and "never."
The setup, honestly scaled
We run the real algorithm on the public pusher_reach env (2.2's — dense reward,
a 2-link arm reaching a seeded target) so every line is readable and reproducible
on a laptop. The map's capstone suite is hidden-seed graded
(grader/hidden_seeds); the leaderboard number is the gated capstone (4.4). This
chapter teaches the mechanism; the leaderboard scores it. Nothing here claims a
strong trained robot — it is the algorithm, shown honestly at free-tier scale.
The corrections are scripted-expert transitions — the analytic IK reacher driving the task to the goal, our offline stand-in for a human teleoperator taking over on the policy's failures (4.2's mechanism, exactly). This is the only data the offline prior ever sees.
# THE CORRECTIONS. ch4.2's mechanism: the scripted expert (reach_action, the# offline stand-in for a human teleoperator) takes over on the task and drives it# to the goal; we record its full transitions — a human-intervention log, exactly# HIL-SERL's input. This is the ONLY data the offline prior sees; the same# transitions are later pre-loaded into the online replay and STAY there (RLPD).def collect_corrections(num_episodes: int) -> dict[str, torch.Tensor]: env = PusherReachEnv() cols: dict[str, list] = {k: [] for k in ("obs", "action", "reward", "next_obs", "terminated")} returns = [] for ep in range(num_episodes): obs = env.reset(seed=args.seed + ep) # episode ep uses a distinct, reproducible seed ep_return, done = 0.0, False while not done: action = reach_action(env) # the correction: what the expert/teleop would do HERE next_obs, reward, done, info = env.step(action) for key, val in zip(cols, (obs, action, reward, next_obs, float(info["terminated"]))): cols[key].append(val) obs = next_obs ep_return += reward returns.append(ep_return) data = {k: torch.tensor(np.array(v), dtype=torch.float32, device=device) for k, v in cols.items()} data["reward"], data["terminated"] = data["reward"].unsqueeze(1), data["terminated"].unsqueeze(1) print(f"corrections: {num_episodes} episodes / {len(data['obs'])} transitions " f"expert return {np.mean(returns):.2f} (the prior's whole data)") return dataBuild the prior: corrections → AWAC
The prior is the primer's AWAC, run on the corrections. A twin-Q critic learns the
reward-to-go on the fixed correction transitions; then the policy is extracted by
advantage-weighted regression — behavior cloning's MSE onto the correction action,
but each sample scaled by exp(A/beta), where A = Q(s, a_corr) - Q(s, pi(s)) is
how much better the correction was than what the current policy would do. This
anchors the policy to actions the corrections actually contain (no
out-of-distribution extrapolation, the failure the primer's --naive showed) and
warm-starts both the actor and the critics that SAC will continue from.
# THE OFFLINE PRIOR (the primer's AWAC, on the corrections). A twin-Q critic learns# the reward-to-go on the fixed corrections; the policy is extracted by advantage-# weighted regression — ch1.1's BC loss on the correction action, each sample scaled# by exp(A/beta), A = Q(s, a_corr) - Q(s, pi(s)). This anchors the policy to actions# the corrections contain (no OOD extrapolation) and warm-starts BOTH the actor AND# the critics online SAC continues from — the "corrections as prior."def train_prior(corr: dict[str, torch.Tensor]): torch.manual_seed(args.seed) # net init + sampling reproducible regardless of upstream RNG use actor = Actor(args.hidden_dim).to(device) q1, q2 = Critic(args.hidden_dim).to(device), Critic(args.hidden_dim).to(device) q1_targ = copy.deepcopy(q1).requires_grad_(False) q2_targ = copy.deepcopy(q2).requires_grad_(False) actor_opt = torch.optim.Adam(actor.parameters(), lr=args.lr) critic_opt = torch.optim.Adam(list(q1.parameters()) + list(q2.parameters()), lr=args.lr) o, a, r, no, t = (corr[k] for k in ("obs", "action", "reward", "next_obs", "terminated")) n = len(o) for step in range(args.prior_steps): idx = torch.randint(0, n, (args.batch_size,)) obs, action, reward, next_obs, term = o[idx], a[idx], r[idx], no[idx], t[idx] with torch.no_grad(): # critic: TD toward r + gamma * min_targ Q(s', pi(s')) next_action, _, _ = actor.sample(next_obs) min_next_q = torch.min(q1_targ(next_obs, next_action), q2_targ(next_obs, next_action)) target_q = reward + args.gamma * (1.0 - term) * min_next_q critic_loss = F.mse_loss(q1(obs, action), target_q) + F.mse_loss(q2(obs, action), target_q) critic_opt.zero_grad() critic_loss.backward() critic_opt.step() _, _, pred = actor.sample(obs) # deterministic mean action with torch.no_grad(): # AWAC advantage weight q_data = torch.min(q1(obs, action), q2(obs, action)) value = torch.min(q1(obs, pred), q2(obs, pred)) weight = torch.exp((q_data - value) / args.beta).clamp(max=args.weight_clip) actor_loss = (weight * (pred - action).pow(2).mean(1, keepdim=True)).mean() actor_opt.zero_grad() actor_loss.backward() actor_opt.step() soft_update(((q1, q1_targ), (q2, q2_targ))) if args.rerun and step % 100 == 0: rr.set_time("prior_step", sequence=step) rr.log("prior/critic_loss", rr.Scalars([critic_loss.item() / 2])) return actor, q1, q2, q1_targ, q2_targ def soft_update(pairs): """Polyak-average targets a hair toward online (ch2.2).""" with torch.no_grad(): for online, targ in pairs: for p, tp in zip(online.parameters(), targ.parameters()): tp.mul_(1.0 - args.tau).add_(args.tau * p)Note one small but load-bearing detail in the actor: we initialize log_std
small, so a freshly-primed policy is near-deterministic. A wide-open
stochastic policy would explore off the prior's good manifold on the very first
online steps and unlearn it — the offline-to-online collapse. Starting quiet is
half the battle of not throwing the prior away.
Fine-tune online: SAC with the corrections still in the replay
Now the online phase. It is ch2.2's SAC, unchanged — one env step, one gradient step on a replay batch — with two differences that make it HIL-SERL:
- The actor and critics start from the prior, not from random.
- The replay buffer is pre-loaded with the corrections, and they never get evicted (RLPD-style). Every online gradient step samples a mix of fresh online transitions and the original corrections — the corrections keep pulling the critic toward known-good actions while online data refines it.
The same loop runs the from-scratch baseline: identical algorithm, but cold nets and an empty replay with a random warmup. The only differences between the two arms are the starting weights and the replay's initial contents. That is the experiment.
# ONE online SAC loop, called TWICE: warm (HIL-SERL — primed nets + corrections# pre-loaded in replay) and cold (from-scratch — fresh nets, empty replay, random# warmup). Identical algorithm; the ONLY differences are the starting weights and# the replay's initial contents. It returns the sample-efficiency curve, the FIRST# online step it clears the threshold, and the BEST checkpoint over the run (ch4.2's# return-the-best: online can dip before recovering, and the prior may be best).def run_online(actor, q1, q2, q1_targ, q2_targ, buffer, total_steps, primed: bool, tag: str): actor_opt = torch.optim.Adam(actor.parameters(), lr=args.lr) critic_opt = torch.optim.Adam(list(q1.parameters()) + list(q2.parameters()), lr=args.lr) env = PusherReachEnv() episode = 0 obs = env.reset(seed=100_000 + args.seed + episode) episode += 1 curve, sts = [], None best_dist, best_state = float("inf"), copy.deepcopy(actor.state_dict()) if primed: # evaluate the prior at ZERO online samples — the head start corrections buy d0, _, _ = evaluate(actor, args.eval_episodes) curve.append((0, round(d0, 5))) best_dist = d0 sts = 0 if d0 < args.threshold else None for step in range(1, total_steps + 1): if not primed and step <= args.learning_starts: # cold start: random warmup fills the buffer action = rng.uniform(-1.0, 1.0, size=act_dim).astype(np.float32) else: with torch.no_grad(): action, _, _ = actor.sample(torch.as_tensor(obs, dtype=torch.float32, device=device).unsqueeze(0)) action = action[0].cpu().numpy() next_obs, reward, done, info = env.step(action) buffer.add(obs, action, reward, next_obs, info["terminated"]) obs = env.reset(seed=100_000 + args.seed + episode) if done else next_obs episode += 1 if done else 0 if primed or step > args.learning_starts: # one SAC gradient step per env step sac_update(actor, q1, q2, q1_targ, q2_targ, actor_opt, critic_opt, buffer) if step % args.eval_interval == 0 or step == total_steps: eval_dist, _, _ = evaluate(actor, args.eval_episodes) curve.append((step, round(eval_dist, 5))) if eval_dist < best_dist: best_dist, best_state = eval_dist, copy.deepcopy(actor.state_dict()) if sts is None and eval_dist < args.threshold: sts = step if args.rerun: rr.set_time("online_step", sequence=step) rr.log(f"eval/{tag}/dist", rr.Scalars([eval_dist])) print(f" [{tag}] step {step:6d}/{total_steps} eval_dist {eval_dist:.4f}m best {best_dist:.4f}m") actor.load_state_dict(best_state) # return the BEST policy over the run (prior included) return curve, sts, best_dist def sac_update(actor, q1, q2, q1_targ, q2_targ, actor_opt, critic_opt, buffer): """One SAC gradient step on a replay batch (ch2.2), fixed entropy temperature. For HIL-SERL the batch mixes fresh online transitions with the pre-loaded corrections, which keep pulling the critic toward known-good actions.""" obs, action, reward, next_obs, terminated = buffer.sample(args.batch_size) with torch.no_grad(): next_action, next_logp, _ = actor.sample(next_obs) min_next_q = torch.min(q1_targ(next_obs, next_action), q2_targ(next_obs, next_action)) - args.alpha * next_logp target_q = reward + args.gamma * (1.0 - terminated) * min_next_q critic_loss = F.mse_loss(q1(obs, action), target_q) + F.mse_loss(q2(obs, action), target_q) critic_opt.zero_grad() critic_loss.backward() critic_opt.step() new_action, logp, _ = actor.sample(obs) actor_loss = (args.alpha * logp - torch.min(q1(obs, new_action), q2(obs, new_action))).mean() actor_opt.zero_grad() actor_loss.backward() actor_opt.step() soft_update(((q1, q1_targ), (q2, q2_targ)))What we measure: samples-to-threshold
The headline is a curve: eval reach-distance versus online environment samples,
for both arms. We read off samples-to-threshold — the first online step at
which the eval mean final distance drops below the bar (--threshold, 0.10 m;
random leaves ~0.176 m).
The result, stated truthfully. HIL-SERL clears the threshold at zero online samples on every seed — because the offline prior, built entirely from the corrections with no environment interaction, is already past the bar (eval distance 0.061 / 0.074 / 0.069 m across seeds 0–2, well under the 0.10 m threshold). From-scratch SAC has to discover the reach from nothing and needs ~10,000–11,000 online samples to reach the same line. That horizontal gap on the sample axis is the sample efficiency, and it is large and seed-robust. On the pooled held-out eval, HIL-SERL's success rate is at least from-scratch's on all three seeds, and the difference interval excludes zero on two of them.
And the honest part, which is itself the lesson: on this small dense-reward task, online fine-tuning holds the prior rather than beating it. The prior already sits near what SAC can reach on this task in a short horizon (2.2's own SAC needs ~18k steps just to solve it), so there is little headroom left for online RL to add — and the offline-to-online transition can wobble the policy before it recovers, which is exactly why we (a) keep the corrections in the replay to anchor it and (b) return the best checkpoint over the run (4.2's return-the-best idiom), prior included. On harder tasks — the real HIL-SERL paper's contact-rich manipulation, and the gated capstone suite — the prior is far from optimal and the online phase is where the policy actually gets good. Here, the corrections do the work, and the honest headline is the sample axis.
The ablation: what each piece buys
# Build the prior from corrections, run BOTH arms, grade with error bars. The# headline is sample-efficiency: samples-to-threshold, HIL-SERL vs scratch.corr = collect_corrections(args.corr_episodes)prior_actor, q1, q2, q1_targ, q2_targ = train_prior(corr)prior_dist, prior_k, _ = evaluate(prior_actor, args.eval_episodes, args.n_seeds)print(f"\noffline prior (AWAC on corrections): eval_dist {prior_dist:.4f}m (0 online samples)") hil_buffer = ReplayBuffer(args.buffer_size)hil_buffer.preload(corr) # the corrections STAY in the replay (RLPD)print(f"\nHIL-SERL: primed + {hil_buffer.size} corrections in replay, fine-tune {args.hil_steps} online steps")hil_curve, hil_sts, hil_best = run_online(prior_actor, q1, q2, q1_targ, q2_targ, hil_buffer, args.hil_steps, True, "hil") torch.manual_seed(args.seed) # from-scratch SAC: fresh nets, empty replay, random warmupscr_actor = Actor(args.hidden_dim).to(device)s1, s2 = Critic(args.hidden_dim).to(device), Critic(args.hidden_dim).to(device)s1_targ, s2_targ = copy.deepcopy(s1).requires_grad_(False), copy.deepcopy(s2).requires_grad_(False)print(f"\nfrom-scratch SAC: cold, empty replay, {args.scratch_steps} online steps")scr_curve, scr_sts, scr_best = run_online(scr_actor, s1, s2, s1_targ, s2_targ, ReplayBuffer(args.buffer_size), args.scratch_steps, False, "scratch") n_pool = args.n_seeds * args.eval_episodes_, hil_pk, _ = evaluate(prior_actor, args.eval_episodes, args.n_seeds)_, scr_pk, _ = evaluate(scr_actor, args.eval_episodes, args.n_seeds)gap = diff_ci(hil_pk, n_pool, scr_pk, n_pool) # HIL - scratchgap_real = gap[0] > 0 or gap[1] < 0 print(f"\n[headline: sample-efficiency] samples-to-threshold (eval_dist < {args.threshold}m):")print(f" HIL-SERL (primed) : {hil_sts if hil_sts is not None else '>' + str(args.hil_steps)} online samples (prior clears it at 0; best {hil_best:.4f}m)")print(f" SAC from scratch : {scr_sts if scr_sts is not None else '>' + str(args.scratch_steps)} online samples (best {scr_best:.4f}m)")if hil_sts == 0 and scr_sts: print(f" -> corrections-as-prior clear the bar with ZERO online samples; scratch needs ~{scr_sts}. That gap IS the sample efficiency.") print(f"\n[ablation: what each piece buys] pooled N={n_pool} held-out rollouts each:")for name, k, dist in (("prior alone (offline, 0 online)", prior_k, prior_dist), ("scratch (online, no prior/corr)", scr_pk, scr_best), ("HIL-SERL (prior+corr+online)", hil_pk, hil_best)): print(f" {name:<34s} success {k}/{n_pool}={k/n_pool:.2f} {tuple(round(x,2) for x in wilson_ci(k,n_pool))} best_dist {dist:.4f}m")print(f" diff CI (HIL-SERL - scratch): {gap[0]:+.2f}..{gap[1]:+.2f} ({'SIGNIFICANT' if gap_real else 'not significant at this N'})")print(f" (random ~{RANDOM_DIST}m; prior & HIL-SERL sit near this task's short-horizon SAC ceiling, so online fine-tuning")print(" HOLDS the prior rather than beating it — the corrections buy the efficiency. Online post-training on harder")print(" tasks is scored on the GATED capstone suite [ch4.4, hidden-seed].)") if args.rerun: for name, k in (("prior", prior_k), ("scratch", scr_pk), ("hil", hil_pk)): rr.log(f"final/{name}/success_rate", rr.Scalars([k / n_pool])) torch.save(prior_actor.state_dict(), args.out / "serl_actor.pt") # HIL-SERL best checkpointtorch.save(scr_actor.state_dict(), args.out / "scratch_actor.pt") # from-scratch SAC baseline (site ONNX demo)metrics = { "threshold": args.threshold, "corr_episodes": args.corr_episodes, "prior_eval_dist": round(prior_dist, 5), "prior_success_rate": round(prior_k / n_pool, 6), "hil_steps_to_threshold": hil_sts, "scratch_steps_to_threshold": scr_sts, "hil_best_dist": round(hil_best, 5), "scratch_best_dist": round(scr_best, 5), "hil_success_rate": round(hil_pk / n_pool, 6), "scratch_success_rate": round(scr_pk / n_pool, 6), "diff_ci_lo": round(gap[0], 6), "diff_ci_hi": round(gap[1], 6), "gap_significant": bool(gap_real), "hil_curve": hil_curve, "scratch_curve": scr_curve, "n_pooled": n_pool, "seed": args.seed, "smoke": bool(args.smoke),}(args.out / "metrics.json").write_text(json.dumps(metrics, indent=2, sort_keys=True) + "\n")print(f"\nmetrics: {args.out / 'metrics.json'}")if args.rerun: print(f"recording: {args.out / 'serl.rrd'} — open it with: rerun {args.out / 'serl.rrd'}")Read the three arms together. Prior alone ≈ HIL-SERL ≫ from-scratch at the same small online budget (measured pooled success 0.37 / 0.33 / 0.33 for the prior and for HIL-SERL, versus 0.10 / 0.07 / 0.27 for scratch across seeds 0–2). The corrections-as-prior are the sample-efficiency win; online fine-tuning maintains it. If a piece did not help on the free tier, we report that — not paper over it (the ch2.1 spike doctrine). The mechanism is what transfers: corrections become a prior, the prior gives you a running start, and RL post-training refines from there.
Wall-clock (free tier)
| cpu-laptop | expected wall-clock on cpu-laptop: ~9.67 min (measured) | measured |
| mps | wall-clock on mps: not yet measured | pending |
| t4 | wall-clock on t4: not yet measured | pending |
| 4090 | wall-clock on 4090: not yet measured | pending |
Read the real thing
The single-file HIL-SERL you just built is the mechanism with the scaffolding stripped away. Two papers are the natural graduation. The HIL-SERL paper (Luo et al., "Precise and Dexterous Robotic Manipulation via Human-in-the-Loop RL") is the grown-up version of this exact loop: a real human teleoperator in the loop online instead of a scripted stand-in, a learned reward classifier instead of a dense analytic reward, and contact-rich tasks where the prior is far from optimal and the online phase genuinely earns its keep. RLPD (Ball et al., "Efficient Online RL with Offline Data") is where the corrections-in-replay trick comes from, plus the stabilizers this file leaves out — explicit symmetric sampling, a high update-to-data ratio, and LayerNorm critics — which are what let online fine-tuning improve a prior on hard tasks rather than merely hold it. (The exact upstream commits are pinned by the read-the-real-thing build step.)
Exercises
Two, both predict-then-run and both graded on a seed-robust signal rather than one lucky run.
- Predict the sample-efficiency gap (
ex1_predict_sample_efficiency.py): does HIL-SERL clear the threshold in fewer online samples than from-scratch, on every one of seeds 0–2? Predict before you run. - Investigate the corrections (
ex2_investigate_corrections.py): starve the correction data and watch the prior — and HIL-SERL's head start — weaken. The corrections are the prior; the prior is the sample efficiency.