The setting: you can't collect more
Every learning method so far got to act. Behavior cloning (1.1) had a dataset, but PPO and SAC (2.1, 2.2) learned by stepping the env millions of times, and even the corrections chapter (4.2) closes the loop — you fly the policy, catch it failing, and record the fix. Now take that away. You have a fixed pile of logged data — some good demonstrations, some clumsy or failed attempts — and you cannot collect one more transition while you learn. No rollouts, no exploration, no on-policy correction. Just the log.
This is not a contrived constraint; it is the normal one. Robot data is expensive and sometimes dangerous to collect, and the data you already have — teleop sessions, logged corrections, a fleet's worth of mixed-quality trajectories — is the data you get. The question this chapter answers by measuring: given a fixed, mixed-quality log, what is the best policy you can squeeze out of it?
Why behavior cloning leaves reward on the table
BC has an answer: clone the data. Regress a network from observation to the action the log took, with plain MSE (1.1). But look at what MSE does to a mixed dataset. In a given state it minimizes squared error to every logged action at once, so it converges to their average. Average a clean expert action with a random flail and you get a watered-down action that is neither. BC's ceiling is therefore the dataset's average quality — and it has no way to do better, because it never looks at the reward. It cannot tell a good action from a bad one; it only knows they were both in the log.
We build the dataset to make this bite. It is the canonical "expert + random"
mix: a --expert_frac slice is the scripted IK reacher from 2.2 (near-optimal),
and the rest is a uniform-random policy (flails, rarely reaches). Measured
behavior returns: expert ~-2.3, random ~-16.0 — genuinely mixed quality.
The random half is not only junk to average over; it also covers the
action space, which the offline learner will need.
# THE FIXED DATASET. This is the whole premise: we build it ONCE, then never# touch the env again during learning. It is MIXED QUALITY on purpose — a# `--expert_frac` slice is the clean scripted IK reacher (near-optimal), and the# rest is a RANDOM policy (uniform torques — flails, rarely reaches). This is the# canonical offline mix (the D4RL "expert + random" recipe): some good# demonstrations buried in a lot of junk. BC averages over ALL of it and gets# dragged toward the junk; offline RL uses the reward to tell the two apart. The# random half also does the offline learner a favor — it COVERS the state-action# space, so the critic sees what bad actions cost. We store full transitions# (obs, action, reward, next_obs, terminated): BC needs only (obs, action), the# critic needs the rest.def build_dataset(num_episodes: int) -> dict[str, torch.Tensor]: env = PusherReachEnv() n_expert = int(round(args.expert_frac * num_episodes)) cols: dict[str, list] = {k: [] for k in ("obs", "action", "reward", "next_obs", "terminated")} ret_expert, ret_noisy = [], [] for ep in range(num_episodes): obs = env.reset(seed=args.seed + ep) # episode ep uses a distinct, reproducible seed is_expert = ep < n_expert ep_return, done = 0.0, False while not done: action = (reach_action(env) if is_expert # scripted IK reach — the expert else rng.uniform(-1, 1, size=act_dim).astype(np.float32)) # the junk half next_obs, reward, done, info = env.step(action) cols["obs"].append(obs) cols["action"].append(action) cols["reward"].append(reward) cols["next_obs"].append(next_obs) cols["terminated"].append(float(info["terminated"])) obs = next_obs ep_return += reward (ret_expert if is_expert else ret_noisy).append(ep_return) data = {k: torch.tensor(np.array(v), dtype=torch.float32, device=device) for k, v in cols.items()} data["reward"] = data["reward"].unsqueeze(1) data["terminated"] = data["terminated"].unsqueeze(1) print(f"dataset: {num_episodes} episodes / {len(data['obs'])} transitions " f"({n_expert} expert, {num_episodes - n_expert} random)") print(f" behavior return: expert {np.mean(ret_expert):7.2f} random {np.mean(ret_noisy):7.2f} " f"(BC clones the mix of these; offline RL should beat it)") return data data = build_dataset(args.episodes)N = len(data["obs"]) def sample_batch() -> tuple[torch.Tensor, ...]: idx = torch.randint(0, N, (args.batch_size,)) # seeded global torch RNG -> reproducible on CPU return (data["obs"][idx], data["action"][idx], data["reward"][idx], data["next_obs"][idx], data["terminated"][idx])What offline RL does instead: use the reward
Offline RL's move is to not treat all logged actions equally. It uses the reward to prefer the actions that were above average, and it does this in two steps you already know how to build.
Step one: a critic. Fit a Q-function on the fixed data — the reward-to-go of
taking action a in state s. This is exactly 2.2's twin-Q critic: two
Q-networks, clipped double-Q (bootstrap from the min), and slow-moving target
networks. The only change from SAC is that there is no replay buffer being filled
by a live policy — the "buffer" is the frozen dataset, sampled forever.
Step two: advantage-weighted policy extraction. Now extract a policy. Here is the whole trick, and it is one line different from BC. BC regresses toward the logged action with a uniform weight. AWAC regresses toward the same logged action, but weights each sample by
weight = exp( A(s, a) / beta ), A(s, a) = Q(s, a) - Q(s, pi(s))
the advantage of the logged action over what the current policy would do. An
action the reward says beat the policy gets weight > 1 and is pulled toward; a
bad action gets a weight near zero and is ignored. That is the entire difference
between "clone the data" and "extract the best policy the data supports." beta
is the temperature: small sharpens the weighting toward pure filtering, large
melts it back into BC.
# Policy: obs -> action MLP, tanh-bounded to the env's [-1, 1] torque range. This# is deliberately the SAME network BC and offline RL both train — the algorithms# differ ONLY in the loss (see region: train), which is the whole lesson. Critic:# a Q(obs, action) net; we build TWO (twin critics, ch2.2's clipped double-Q) and# always bootstrap from the MIN, the brace that fights Q overestimation.class Policy(nn.Module): def __init__(self, hidden_dim: int): super().__init__() self.net = nn.Sequential( nn.Linear(obs_dim, hidden_dim), nn.ReLU(), nn.Linear(hidden_dim, hidden_dim), nn.ReLU(), nn.Linear(hidden_dim, act_dim), ) def forward(self, obs: torch.Tensor) -> torch.Tensor: return torch.tanh(self.net(obs)) # actions live in [-1, 1] class Critic(nn.Module): def __init__(self, hidden_dim: int): super().__init__() self.net = nn.Sequential( nn.Linear(obs_dim + act_dim, hidden_dim), nn.ReLU(), nn.Linear(hidden_dim, hidden_dim), nn.ReLU(), nn.Linear(hidden_dim, 1), ) def forward(self, obs: torch.Tensor, action: torch.Tensor) -> torch.Tensor: return self.net(torch.cat([obs, action], dim=1))The two learners share the same network and the same regression, and differ only in that weight — so read their losses side by side:
AWAC is BC's regression toward the logged action, but each sample scaled by the exponentiated advantage of that action over what the policy would do — and here it is in code:
# The two learners, side by side. BC and offline RL share the policy network and# the SAME regression — predict the dataset action — and differ in ONE line: the# per-sample weight. That is the entire point of the chapter, so read the losses# together.def train_bc() -> Policy: """Behavior cloning: uniform-weighted MSE to the dataset action (ch1.1). The ceiling is the data's average quality — it cannot tell a good action from a bad one, because it never looks at the reward.""" policy = Policy(args.hidden_dim).to(device) opt = torch.optim.Adam(policy.parameters(), lr=args.lr) for step in range(args.steps): obs, action, *_ = sample_batch() loss = F.mse_loss(policy(obs), action) opt.zero_grad() loss.backward() opt.step() if args.rerun and step % 50 == 0: rr.set_time("step", sequence=step) rr.log("bc/loss", rr.Scalars([loss.item()])) return policy def train_offline() -> tuple[Policy, float]: """Offline RL. First a twin-Q critic learns the reward-to-go on the FIXED data (bootstrapping from the policy's own next action, ch2.2 idiom). Then the policy is extracted: AWAC advantage-weighted regression — the BC loss, but each sample scaled by exp(A/beta) where A = Q(s,a_data) - Q(s,pi(s)). Actions the reward says beat the current policy get pulled toward; bad ones get ignored. naive (--naive) NO constraint: maximize Q(s, pi(s)) directly. Offline, the policy walks to out-of-distribution actions where Q is a fantasy — the overestimation the constraint exists to prevent. Worst on narrow data (--expert_frac 1.0). This is the Break It. """ policy = Policy(args.hidden_dim).to(device) q1, q2 = Critic(args.hidden_dim).to(device), Critic(args.hidden_dim).to(device) q1_targ = copy.deepcopy(q1).requires_grad_(False) q2_targ = copy.deepcopy(q2).requires_grad_(False) policy_opt = torch.optim.Adam(policy.parameters(), lr=args.lr) critic_opt = torch.optim.Adam(list(q1.parameters()) + list(q2.parameters()), lr=args.lr) mean_q = float("nan") for step in range(args.steps): obs, action, reward, next_obs, terminated = sample_batch() # --- critic: TD-regress both Q's toward r + gamma * min_targ Q(s', pi(s')) --- with torch.no_grad(): next_action = policy(next_obs) min_next_q = torch.min(q1_targ(next_obs, next_action), q2_targ(next_obs, next_action)) target_q = reward + args.gamma * (1.0 - terminated) * min_next_q q1_loss, q2_loss = F.mse_loss(q1(obs, action), target_q), F.mse_loss(q2(obs, action), target_q) critic_opt.zero_grad() (q1_loss + q2_loss).backward() critic_opt.step() # --- policy extraction: the ONE line that separates the algorithms --- pred = policy(obs) if args.naive: # maximize Q with no anchor to the data -> divergence offline policy_loss = -torch.min(q1(obs, pred), q2(obs, pred)).mean() else: # AWAC: advantage-weighted regression toward the dataset action with torch.no_grad(): q_data = torch.min(q1(obs, action), q2(obs, action)) # Q of the logged action value = torch.min(q1(obs, pred), q2(obs, pred)) # V(s) = Q(s, pi(s)) baseline weight = torch.exp((q_data - value) / args.beta).clamp(max=args.weight_clip) policy_loss = (weight * (pred - action).pow(2).mean(1, keepdim=True)).mean() policy_opt.zero_grad() policy_loss.backward() policy_opt.step() # --- soft target update (Polyak), ch2.2 idiom --- with torch.no_grad(): for online, targ in ((q1, q1_targ), (q2, q2_targ)): for p, tp in zip(online.parameters(), targ.parameters()): tp.mul_(1.0 - args.tau).add_(args.tau * p) mean_q = 0.5 * (q1(obs, action) + q2(obs, action)).mean().item() if args.rerun and step % 50 == 0: rr.set_time("step", sequence=step) rr.log("offline/critic_loss", rr.Scalars([(q1_loss + q2_loss).item() / 2])) rr.log("offline/policy_loss", rr.Scalars([policy_loss.item()])) rr.log("offline/mean_q", rr.Scalars([mean_q])) # watch this EXPLODE under --naive return policy, mean_qWhy naive offline RL is not enough (and why the constraint exists)
A reasonable objection: if we have a Q-function, why the weighting ceremony — why
not just train the policy to maximize Q, the way DDPG or SAC do? Run --naive
and find out. Maximizing Q with no anchor to the data lets the policy walk to
out-of-distribution actions — actions the log never contains — where the
Q-function is pure extrapolation. Offline, there is no env step to check that
fantasy, so the critic overestimates those actions and the policy happily
chases the overestimate.
Here is the honest, measured subtlety, and it is the best part of the lesson.
How badly this bites depends on how narrow your data is. On the broad
expert+random mix, the random half covers the action space, the critic stays
calibrated, and even naive maximize-Q survives. But on narrow data —
expert-only, --naive --expert_frac 1.0, which is the shape of real demo and
correction logs — the naive critic inflates: measured mean |Q| ~7x the
AWAC critic's while its eval collapses toward random. AWAC's advantage weighting
is the fix that does not depend on coverage: it only ever asks the critic
about actions the data actually contains, so there is nothing to extrapolate
into. That is why offline RL is its own algorithm and not just "Q-learning on a
log."
The result: measured, honest
Train both on the same fixed dataset, then grade them with 1.6's error bars — a Wilson interval on each success rate and a difference CI that decides whether the gap is real. On the default free-tier config (mixed dataset, held out over 100 rollouts), across seeds 0, 1, 2:
| learner | success rate | mean final dist |
|---|---|---|
| behavior cloning | 0.03 / 0.07 / 0.05 | 0.154 / 0.147 / 0.144 m |
| offline RL (AWAC) | 0.27 / 0.20 / 0.21 | 0.109 / 0.100 / 0.089 m |
The difference CI (offline − BC) excludes zero on every seed — the win is significant, not noise, and it holds across seeds. Offline RL beat BC on the same data by using the reward BC ignored.
Be honest about the size of it. Offline RL reaches ~0.09–0.11 m (random leaves ~0.176 m), still well short of the scripted expert's ~0.0001 m — this is a primer on a tiny free-tier budget, not a leaderboard entry. And BC does not improve much even as you clean the data up (exercise 2 sweeps 15% → 60% expert): it stays ~0.03–0.05 success as long as the log still carries a meaningful junk fraction, because averaging any junk into the regression target corrupts it. The mechanism is the lesson, not the number: reward-aware extraction beats cloning, and the advantage constraint is what makes it safe to do offline.
Now the boundary that locates the whole mechanism, and it is the most important
honesty in the chapter. Take the junk away entirely — --expert_frac 1.0, a
clean expert log and nothing else — and AWAC's edge over BC evaporates. Measured
across seeds 0, 1, 2: BC jumps to ~0.2 success (there is no bad action left to
average in, so cloning the expert is the right move) and actually holds a
slightly lower final distance than AWAC (~0.053 m vs ~0.076 m); the success-rate
difference CI is marginal and not seed-robust — a tie on seed 0, a hair above zero
on 1 and 2. AWAC lost nothing; it simply had nothing left to reweight. That is the
real headline, stated carefully: offline RL beats cloning when the log carries
suboptimal actions worth down-weighting — the mixed, correction-shaped data of
the real world. On a pristine expert log, cloning already does the reweighting for
free, and the honest reading is a wash. This is why the default dataset is a mix,
and why the win to trust is the seed-robust one on that mix, not a blanket claim
that "AWAC beats BC."
# Train both, then grade them with error bars. The headline is the difference CI:# does offline RL BEAT BC on the SAME fixed dataset, and is the gap real at this N?bc_policy = train_bc()offline_policy, offline_mean_q = train_offline()mode = "naive-maxQ" if args.naive else "AWAC"print(f"\ntrained BC and offline ({mode}); offline final mean|Q| over data = {abs(offline_mean_q):.1f}") bc_succ, bc_dist = evaluate(bc_policy)off_succ, off_dist = evaluate(offline_policy)n_pool = args.n_seeds * args.eval_episodesbc_k, off_k = int(bc_succ.sum()), int(off_succ.sum())gap = diff_ci(off_k, n_pool, bc_k, n_pool) # offline - BC; excludes 0 => realgap_real = gap[0] > 0 or gap[1] < 0verdict = ("offline RL BEATS BC" if gap[0] > 0 else "BC beats offline RL" if gap[1] < 0 else "tie (diff CI spans 0)") print(f"\n[headline] BC vs offline RL on the same fixed dataset (N={n_pool} rollouts each):")print(f" BC success {bc_k}/{n_pool} = {bc_k/n_pool:.2f} {tuple(round(x,2) for x in wilson_ci(bc_k,n_pool))}" f" mean final dist {bc_dist.mean():.4f} m")print(f" offline success {off_k}/{n_pool} = {off_k/n_pool:.2f} {tuple(round(x,2) for x in wilson_ci(off_k,n_pool))}" f" mean final dist {off_dist.mean():.4f} m")print(f" diff CI (offline - BC): {gap[0]:+.2f}..{gap[1]:+.2f} -> {verdict}" f" ({'SIGNIFICANT' if gap_real else 'not significant at this N'})")print(f" (random baseline ~{RANDOM_DIST} m; lower dist is better)")if args.naive: print(f"\n[Break It] naive offline RL (maximize Q, no data constraint): mean|Q| over data = {abs(offline_mean_q):.1f}.") print(" On narrow data (--expert_frac 1.0) this inflates ~7x vs AWAC's ~1 while eval collapses toward random:") print(" the policy chases OOD actions the critic OVERESTIMATES. On the broad expert+random mix, coverage") print(" keeps the critic honest and naive survives — which is exactly why narrow correction data needs the constraint.") if args.rerun: for name, k, dist in (("bc", bc_k, bc_dist), ("offline", off_k, off_dist)): lo, hi = wilson_ci(k, n_pool) rr.log(f"eval/{name}/success_rate", rr.Scalars([k / n_pool])) rr.log(f"eval/{name}/success_ci", rr.Scalars([lo, hi])) rr.log(f"eval/{name}/mean_dist", rr.Scalars([float(dist.mean())])) torch.save(offline_policy.state_dict(), args.out / "offline_policy.pt")torch.save(bc_policy.state_dict(), args.out / "bc_policy.pt") # the BC baseline arm, for the site ONNX demometrics = { "mode": mode, "episodes": args.episodes, "expert_frac": args.expert_frac, "steps": args.steps, "beta": args.beta, "n_pooled": n_pool, "bc_success_rate": round(bc_k / n_pool, 6), "offline_success_rate": round(off_k / n_pool, 6), "bc_mean_final_dist": round(float(bc_dist.mean()), 6), "offline_mean_final_dist": round(float(off_dist.mean()), 6), "diff_ci_lo": round(gap[0], 6), "diff_ci_hi": round(gap[1], 6), "gap_significant": bool(gap_real), "offline_mean_abs_q": round(abs(offline_mean_q), 4), "naive": bool(args.naive), "seed": args.seed, "smoke": bool(args.smoke),}(args.out / "metrics.json").write_text(json.dumps(metrics, indent=2, sort_keys=True) + "\n")print(f"\nmetrics: {args.out / 'metrics.json'}")if args.rerun: print(f"recording: {args.out / 'offline.rrd'} — open it with: rerun {args.out / 'offline.rrd'}")Why this is the prior 4.3 builds on
The capstone's RL post-training (4.3, HIL-SERL) fine-tunes an RL policy starting from your correction data as a prior — and sample efficiency is the whole point, so it cannot afford to start from scratch. That prior is precisely what this chapter builds: a policy extracted from a fixed log of mixed-quality behavior, using the reward, without the extrapolation error that sinks naive offline RL. Correction data is narrow — exactly the regime where the naive approach fails and the advantage constraint earns its keep. When 4.3 says "offline-primed," this is the priming.
Running it
python curriculum/phase4_capstone/ch4_offline_primer/offline.py --seed 0
# Break it (narrow data): ... --seed 0 --naive --expert_frac 1.0
# Investigate: ... --seed 0 --expert_frac 0.6
Determinism: --seed fixes the dataset, both trainings, and every eval reset, so
two CPU runs produce byte-identical metrics.json. CPU is the one configuration
this book promises is bitwise-reproducible; on a GPU or an Apple mps backend the
same seed reproduces the result statistically, not bitwise — the numbers above
are --device cpu, and the exercises force it.
Read the real thing
The AWAC you just built is the honest core of a real algorithm, not a toy of one. Two readings show you where the production implementations go from here.
The AWAC paper — Nair et al. 2020, Accelerating Online Reinforcement Learning
with Offline Datasets — is where the advantage weight comes from. The
exp(A / beta) you wrote is not a heuristic someone tuned; it falls out as the
closed-form solution to a KL-constrained policy-improvement step, which is why the
weight is an exponential of the advantage and why beta is a temperature. Read
it for the derivation the code can only assert.
IQL — Kostrikov's reference implementation, ikostrikov/implicit_q_learning —
is the "what the grown-up version does differently" reading. Find the one place
our critic is still asked about an action it never saw: the Q(s, pi(s)) baseline
in the advantage. IQL removes even that, replacing it with an expectile
regression that estimates the state value without ever querying an out-of-sample
action — closing the last crack the advantage constraint left open. The expectile
loss is a single line; find it, and you have the whole idea.