Behavior CloningThe Dumbest Thing That Works

The problem

In chapter 0.4 you drove this pusher yourself, with your own two hands on the keys, and you succeeded at a task you could not begin to write down. Try it: write the rule you used for deciding when to stop pushing the bar and start nudging the stem. You rotated the block when it needed rotating. How did you know? You'd have to say something about the angle, and the contact point, and where the pusher happened to be — and by the third clause you're describing a controller you never consciously ran.

The scripted expert in curriculum/common/envs/pusht/scripted_expert.py is what writing it down actually costs: a two-phase state machine, contact-point selection in the block's body frame, a detour circle so the approach never plows through the block, a steering offset proportional to yaw error. It took real engineering hours, it is full of constants like 0.045 and 1.15, and it works for exactly one task on exactly one block. Nobody writes that controller for the thousandth task.

Behavior cloning skips the describing. You already produced the only artifact it needs — a dataset of (observation, action) pairs from chapter 0.4's teleop session — and the claim this chapter tests is almost insulting in its simplicity: fit a function from one column to the other with mean squared error, and driving skill falls out. What we build: a single file that loads your demos, fits an MLP, measures it honestly in rollouts, and ships the result to the browser. Then we break it in a way no loss curve will admit to.

Build

Four regions in dependency order: data, model, train, eval. The whole file is bc.py, about 270 lines, and every line of it is on the page — there is no framework underneath.

Setup

One thing to look for: every source of randomness in this file — the train/val split, the weight init, the batch shuffle — flows from the one --seed flag through set_seed.

import argparseimport jsonimport shutilimport sysfrom pathlib import Path import numpy as npimport torchimport torch.nn as nn # Chapter artifacts run as loose scripts from the repo root; put the root on# sys.path so `curriculum.common` resolves (same pattern as tests/).sys.path.insert(0, str(Path(__file__).resolve().parents[3])) from curriculum.common.assert_parity import assert_parity  # noqa: E402from curriculum.common.device import banner, detect_device  # noqa: E402from curriculum.common.envs.pusht import PushTEnv, gen_demos  # noqa: E402from curriculum.common.export_onnx import export_policy  # noqa: E402from curriculum.common.seeding import set_seed  # noqa: E402 parser = argparse.ArgumentParser(description=__doc__)parser.add_argument("--data", type=Path, default=Path("outputs/pusht-demos"),                    help="LeRobot-format demo dataset (your ch0.4 teleop session, or gen_demos.py output)")parser.add_argument("--out", type=Path, default=Path("outputs/ch1.1-bc"))parser.add_argument("--epochs", type=int, default=600)  # cpu-laptop: minutes | smoke: 3parser.add_argument("--batch_size", type=int, default=128)parser.add_argument("--lr", type=float, default=1e-3, help="peak Adam lr; cosine-decays to 0 over --epochs")parser.add_argument("--hidden_dim", type=int, default=512)  # width is NOT the bottleneck — see the model regionparser.add_argument("--seed", type=int, default=0, help="seeds the split, the init, and the shuffle")parser.add_argument("--eval_episodes", type=int, default=50)  # T4: 50 | smoke: 5parser.add_argument("--normalize", choices=("full", "narrow"), default="full",                    help="narrow = stats from a lopsided slice of the demos; the Break It flag")parser.add_argument("--device", choices=("cpu", "cuda", "mps"), default=detect_device())  # T4: cuda | Mac: mps | cpu: deterministic (statistical repro on GPU/mps)parser.add_argument("--smoke", action="store_true",                    help="tiny self-contained CPU run for CI; two runs must produce byte-identical metrics.json")parser.add_argument("--rerun", dest="rerun", action="store_true", default=True)parser.add_argument("--no-rerun", dest="rerun", action="store_false", help="skip .rrd recording (CI smoke)")args = parser.parse_args() rng = set_seed(args.seed)  # seeds python/numpy/torch; returns the numpy Generator the split draws fromif args.smoke:  # smoke pins everything the CI byte-compare depends on    args.epochs, args.eval_episodes, args.device = 3, 5, "cpu"banner("ch1.1-bc", device=args.device)  # report the device the run ACTUALLY uses (after --smoke pins cpu)args.out.mkdir(parents=True, exist_ok=True)device = torch.device(args.device)if args.rerun:    import rerun as rr    rr.init("zero2robot/ch1.1-bc", spawn=False)    rr.save(str(args.out / "bc.rrd"))

The flags follow the house convention: free-tier defaults, a --smoke mode that runs tiny and fixed so CI can byte-compare two runs, and --no-rerun to skip recording. Two flags are new. --normalize you will meet properly in Break It. --device picks the fastest thing your machine has (cuda, then mps, then cpu); the reference numbers below are from --device cpu, the one configuration we can promise is bitwise-deterministic under --seed. On a GPU or an Apple mps backend the same seed reproduces the result statistically, not bitwise — your success rate will land near the number printed here, not exactly on it — and this book does not pretend otherwise.

Data

Look for what gets split: episodes, never frames.

# Two dataset paths, same format: the reference is `lerobot/pusht` from the# HF Hub (human demos); the path this chapter assumes is YOURS — the teleop# session you recorded in ch0.4, or the scripted-expert set from gen_demos.py.if args.smoke:    # Smoke runs are hermetic: CI regenerates its own tiny deterministic set.    # REGENERATE it every run (never reuse a leftover dir): a cache from a    # different --seed would train on seed-0 demos while metrics.json records    # seed 1 — silent wrong data. gen_demos is deterministic, so same seed ->    # bit-identical dataset whether it was just built or rebuilt.    args.data = args.out / "smoke-demos"    if args.data.exists():        shutil.rmtree(args.data)    gen_demos.main(["--episodes", "6", "--seed", str(args.seed),                    "--out", str(args.data), "--no-video"])if not (args.data / "meta" / "info.json").is_file():    sys.exit(f"no dataset at {args.data} — record one in ch0.4, or generate demos first:\n"             f"  python curriculum/common/envs/pusht/gen_demos.py "             f"--episodes 500 --seed 0 --out {args.data} --no-video") from lerobot.datasets.lerobot_dataset import LeRobotDataset  # noqa: E402  (heavy import — after the cheap failures) frames = LeRobotDataset("local/pusht-demos", root=args.data).hf_dataset.with_format("numpy")obs = np.stack(frames["observation.state"])  # (N, 10) — layout documented in pusht_env.pyactions = np.stack(frames["action"])         # (N, 2)  — pusher velocity in [-1, 1]episode_ids = np.asarray(frames["episode_index"])  # (N,) which demo each frame came from # Split by EPISODE, never by frame. Frames 0.1 s apart are near-duplicates;# a frame-level split puts one twin in train and one in val, and val loss# becomes a memorization test you can only pass. Episode-level keeps val# honest: whole trajectories the network has never seen.episode_order = rng.permutation(np.unique(episode_ids))val_episodes = episode_order[: max(1, len(episode_order) // 10)]train_episodes = episode_order[len(val_episodes):]in_val = np.isin(episode_ids, val_episodes) # Normalization stats: per-dim min/max, mapping each dim to [-1, 1] — the# same scheme the real PushT policies use (diffusion_policy, LeRobot).# full: stats over every training frame, so by construction no training# input ever leaves [-1, 1]. narrow: stats over only the ~20% of training# episodes whose block STARTS closest to the target — the demos you'd record# first while testing your teleop rig, block placed gently near the goal.# Both splits share the same stats either way, which is exactly why no loss# curve will flag the difference (Break It).if args.normalize == "narrow":    def block_start_distance(episode: int) -> float:        tee_x, tee_y = obs[episode_ids == episode][0][2:4]  # episode's first frame        return float(np.hypot(tee_x, tee_y))    easiest = sorted(train_episodes, key=block_start_distance)[: max(1, len(train_episodes) // 5)]    stats_frames = np.isin(episode_ids, easiest)else:    stats_frames = ~in_valobs_min, act_min = obs[stats_frames].min(0), actions[stats_frames].min(0)obs_range = obs[stats_frames].max(0) - obs_minact_range = actions[stats_frames].max(0) - act_min# The 4 target dims are constant in this phase -> range 0. A constant carries# no information; give it range 1 so it maps to a constant instead of a# division by zero.obs_range = np.where(obs_range < 1e-4, np.float32(1.0), obs_range)act_range = np.where(act_range < 1e-4, np.float32(1.0), act_range) train_obs = torch.from_numpy(obs[~in_val]).to(device)train_actions = torch.from_numpy(actions[~in_val]).to(device)val_obs = torch.from_numpy(obs[in_val]).to(device)val_actions = torch.from_numpy(actions[in_val]).to(device)print(f"dataset: {len(episode_order)} episodes / {len(obs)} frames "      f"({len(train_episodes)} train / {len(val_episodes)} val episodes), normalize={args.normalize}")

The dataset is LeRobot-format — the same format your ch0.4 teleop session produced, the same format lerobot/pusht's human demonstrations ship in, and the same format the reference generator writes. This chapter's numbers use the scripted expert's demos so they're exactly reproducible on your machine:

python curriculum/common/envs/pusht/gen_demos.py --episodes 500 --seed 0 \
    --out outputs/pusht-demos --no-video

Two decisions in this region carry most of the chapter's honesty. First, the split. Consecutive frames are 0.1 s apart; the block barely moves between them. Split frame-wise and nearly every validation frame has a near-twin in training — val loss becomes a memorization test you can only pass, and it will happily stay low while your policy learns nothing transferable. Splitting by episode keeps validation what it claims to be: whole trajectories the network has never seen.

Second, normalization. Raw observations mix meters (±0.35) with sin/cos (±1); raw actions are velocities. We rescale each dimension to [-1, 1] using its min and max over the training frames — the same scheme the real PushT policies use. The stats are computed from the data, which sounds like a triviality and is actually a loaded gun: the stats are a claim about what the world looks like, and the code will believe that claim long after it stops being true. --normalize full computes them from all training episodes. The other setting exists for Break It.

Model

Look at how little there is to look at.

class BCPolicy(nn.Module):    """3-layer MLP, obs float32[10] -> action float32[2]. Deliberately boring.     The ceiling of behavior cloning is the data, not the network: this policy    can never act better than the demonstrations it averages over, and past a    point extra width just buys a sharper copy of the same mistakes (exercise    3 makes you measure where that point is). Normalization lives INSIDE the    model, as buffers: the checkpoint and the ONNX export carry their own    stats, so the playground (tensor contract v1) feeds raw observations.    """     def __init__(self, hidden_dim: int, obs_min, obs_range, act_min, act_range):        super().__init__()        self.net = nn.Sequential(            nn.Linear(PushTEnv.OBS_DIM, hidden_dim), nn.ReLU(),            nn.Linear(hidden_dim, hidden_dim), nn.ReLU(),            nn.Linear(hidden_dim, PushTEnv.ACT_DIM),        )        for name, stat in [("obs_min", obs_min), ("obs_range", obs_range),                           ("act_min", act_min), ("act_range", act_range)]:            self.register_buffer(name, torch.from_numpy(stat))  # saved with the weights, but never trained     def forward(self, obs: torch.Tensor) -> torch.Tensor:        # (B, 10) raw -> (B, 10) in [-1, 1] -> (B, 2) in [-1, 1] -> (B, 2) raw        normalized_obs = 2.0 * (obs - self.obs_min) / self.obs_range - 1.0        # The clamp guards the net against inputs outside the range it trained        # on. When the stats cover the demos (they define [-1, 1]) it never        # moves a training value — remember it exists. Break It wakes it up.        normalized_action = self.net(normalized_obs.clamp(-1.0, 1.0))        return (normalized_action + 1.0) / 2.0 * self.act_range + self.act_min  policy = BCPolicy(args.hidden_dim, obs_min, obs_range, act_min, act_range).to(device)

Three linear layers and two ReLUs. This is deliberate, and it is not modesty. The ceiling of behavior cloning is the data, not the network: the policy can never act better than the demonstrations it averages over, so capacity spent past "can represent the demonstrator" buys you nothing but a sharper copy of the same mistakes. Exercise 3 makes you measure this instead of trusting me — you'll 10x the network's training two different ways and watch which one the success rate responds to.

One structural decision deserves attention: the normalization stats live inside the model, as buffers. The forward pass takes raw observations, normalizes, predicts, denormalizes. That means the checkpoint and the ONNX export carry their own stats — the browser playground can feed the model observations straight from the simulator, and there is no separate stats file to lose, version-skew, or apply twice. Exercise 1 is about the "apply twice" failure, which every robotics codebase commits eventually.

The clamp in the forward pass never moves a training value — the training data defines [-1, 1], so nothing in it can leave. Remember that it exists.

Train

The loop is short enough to read in one breath.

# A plain loop — no DataLoader, no scheduler, no early stopping. The whole# dataset sits in memory as two tensors; "batching" is indexing a shuffled# permutation. MSE says: predict the average action the demonstrator took in# this state. Defensible (it's maximum likelihood under a Gaussian), flawed# (when demos disagree, the average of two good actions can be a bad one —# that flaw is chapter 1.2's opening problem).optimizer = torch.optim.Adam(policy.parameters(), lr=args.lr)# One concession to optimization reality: decay the lr to 0 over the run, or# the last epochs bounce around the minimum instead of settling into it# (measured: +6 points of success rate at the default config).scheduler = torch.optim.lr_scheduler.CosineAnnealingLR(optimizer, T_max=args.epochs)shuffle = torch.Generator().manual_seed(args.seed)  # torch-side RNG: same seed -> same batch ordertrain_loss, global_step = float("nan"), 0for epoch in range(args.epochs):    epoch_loss, num_batches = 0.0, 0    for batch in torch.randperm(len(train_obs), generator=shuffle).split(args.batch_size):        loss = nn.functional.mse_loss(policy(train_obs[batch]), train_actions[batch])        optimizer.zero_grad()        loss.backward()        optimizer.step()        epoch_loss, num_batches = epoch_loss + loss.item(), num_batches + 1        if args.rerun:            rr.set_time("step", sequence=global_step)            rr.log("policy/loss/train", rr.Scalars([loss.item()]))        global_step += 1    scheduler.step()    train_loss = epoch_loss / num_batches    with torch.no_grad():        val_loss = nn.functional.mse_loss(policy(val_obs), val_actions).item()    if args.rerun:        rr.log("policy/loss/val", rr.Scalars([val_loss]))    if epoch % 50 == 0 or epoch == args.epochs - 1:        print(f"epoch {epoch:3d}  train_loss {train_loss:.5f}  val_loss {val_loss:.5f}") torch.save(policy, args.out / "bc_policy.pt")  # whole module — reloadable only where THIS file's BCPolicy is importable# TorchScript carries its own code, so assert_parity's CLI reloads it in a# fresh interpreter with no BCPolicy on the path (the cross-process gate):#   python curriculum/common/assert_parity.py <out>/bc_policy.onnx <out>/bc_policy.ts.pttorch.jit.script(policy.eval()).save(str(args.out / "bc_policy.ts.pt"))

No DataLoader, no gradient clipping, no early stopping: the dataset is two tensors in memory, and a "batch" is a slice of a shuffled index permutation. The one concession to optimization reality is the cosine decay on the learning rate — without it the last hundred epochs bounce around the minimum instead of settling into it, and the difference is measurable: +6 points of success rate at the default configuration (62% with the decay, 56% without).

MSE deserves a sentence of defense and a sentence of prosecution. Defense: minimizing squared error on actions is maximum likelihood under a Gaussian — it's not a hack, it's the textbook estimator for "predict the average action the demonstrator took in this state". Prosecution: it predicts the AVERAGE action. When demonstrations disagree — approach the block clockwise or counterclockwise, push the near tip or the far one — the average of two good actions can be a bad action, sometimes a catastrophic one. Even our scripted expert disagrees with itself: its action depends on an internal phase (approaching vs. mid-stroke) that the observation does not fully reveal, so the same observation carries different labels and the MLP splits the difference. Hold that thought; it is the entire reason chapter 1.2 exists.

Eval

Loss measured how well we imitate on the dataset's states. Rollouts ask the question we actually care about.

# Loss measured how well we imitate on the dataset's states. Rollouts measure# what we actually care about: does the block reach the target when the POLICY# picks the states it visits? Those are different questions — Break It shows# just how different.env = PushTEnv()policy.eval()successes, episode_returns = 0, []for episode in range(args.eval_episodes):    # 10_000 + offset: demo episode i used reset seed (seed + i), so eval    # seeds are held out by construction — never graded on a start we trained on.    obs_now = env.reset(seed=10_000 + args.seed + episode)    episode_return, done, info = 0.0, False, {}    while not done:        with torch.no_grad():  # the rollout loop: obs -> action -> step (exercise 2 blanks this)            obs_batch = torch.from_numpy(obs_now).to(device).unsqueeze(0)  # (10,) -> (1, 10)            action = policy(obs_batch)[0].cpu().numpy()        obs_now, reward, done, info = env.step(action)        episode_return += reward        if args.rerun:            # offset each episode by the max horizon so traces don't overlap on one timeline            rr.set_time("sim_time", duration=episode * (PushTEnv.MAX_STEPS / PushTEnv.CONTROL_HZ) + env.data.time)            rr.log("policy/action", rr.Scalars(action.astype(np.float64)))            rr.log("eval/pos_err", rr.Scalars([info["pos_err"]]))    successes += bool(info["success"])    episode_returns.append(episode_return)    if args.rerun:        rr.log("eval/success", rr.Scalars([float(info["success"])]))        rr.log("eval/episode_return", rr.Scalars([episode_return]))success_rate = successes / args.eval_episodesprint(f"eval: {successes}/{args.eval_episodes} episodes succeeded "      f"(success rate {success_rate:.2f}), mean return {np.mean(episode_returns):.3f}") # The full loop ends in the browser: export to ONNX (tensor contract v1),# then prove torch and onnxruntime agree before the file goes anywhere.onnx_path = export_policy(policy, PushTEnv.OBS_DIM, PushTEnv.ACT_DIM, args.out / "bc_policy.onnx")parity_delta = assert_parity(policy, onnx_path, PushTEnv.OBS_DIM)print(f"exported {onnx_path} — torch/onnx parity delta {parity_delta:.2e}") metrics = {    "epochs": args.epochs,    "eval_episodes": args.eval_episodes,    "final_train_loss": round(train_loss, 6),    "final_val_loss": round(val_loss, 6),    "mean_episode_return": round(float(np.mean(episode_returns)), 6),    "normalize": args.normalize,    "parity_delta": round(parity_delta, 6),  # rounds to 0.0 in practice; the gate already asserted < 1e-4    "seed": args.seed,    "smoke": bool(args.smoke),    "success_rate": round(success_rate, 6),}(args.out / "metrics.json").write_text(json.dumps(metrics, indent=2, sort_keys=True) + "\n")print(f"metrics: {args.out / 'metrics.json'}")if args.rerun:    print(f"recording: {args.out / 'bc.rrd'} — open it with: rerun {args.out / 'bc.rrd'}")

The distinction matters because the policy chooses its own future inputs. One slightly-off action leads to a state slightly off the demonstration manifold, where the policy is slightly worse, which leads further off — the compounding spiral is called covariate shift, and it is the disease behavior cloning dies of. A loss number cannot see it, because the loss is computed on states the DEMONSTRATOR chose. Rollouts from 50 held-out reset seeds can. Note the seed arithmetic: demo episode i consumed reset seed seed + i, so the evaluation seeds 10_000 + … are new poses by construction — we never grade the policy on a start it trained on.

After the rollouts, two lines close the loop the whole book runs on: export_policy writes the ONNX under tensor contract v1, and assert_parity proves torch and onnxruntime produce the same actions before the file is allowed anywhere near the playground. The measured parity delta at the default config is about 2e-06 — comfortably under the 1e-4 gate.

Run it

python curriculum/phase1_imitation/ch1.1_bc/bc.py --seed 0

With 500 scripted demos and default flags, the reference run reaches 31/50 held-out episodes (62% success), mean return -36.6, final train loss 0.010, val loss 0.037. Open the recording:

rerun outputs/ch1.1-bc/bc.rrd

Two timelines. On step: policy/loss/train sawing downward, policy/loss/val flattening out around epoch 300 — after that you are polishing the fit, not learning new behavior. On sim_time: fifty eval episodes laid end to end, eval/pos_err and policy/action per step. The healthy signature is pos_err curves that dive and stay down; the sick ones dive, stall, and saw sideways — go find two or three of those now (about a third of episodes fail at this scale; you have plenty to choose from) and scrub policy/action at the stall. That's what averaged indecision looks like, and diagnosing it from the trace is a skill this book will ask of you thirty more times.

And 62% is worth staring at. The expert that produced the demonstrations succeeds on every one of the fifty seeds measured in the env README (100%). Same observations, same task — we lost a third of the competence in the copying. Some of that is averaging over the expert's hidden phase, some is covariate shift off the demo manifold. More demonstrations buy some of it back (exercise 4 quantifies how much); no amount buys back all of it.

Break it

Two runs, one flag apart (the smaller network and shorter schedule are deliberate — the note at the end of this section explains why, and it's worth the wait):

python curriculum/phase1_imitation/ch1.1_bc/bc.py --seed 0 --epochs 300 --hidden_dim 256
python curriculum/phase1_imitation/ch1.1_bc/bc.py --seed 0 --epochs 300 --hidden_dim 256 --normalize narrow

narrow computes the normalization stats from only the ~20% of training episodes whose block starts closest to the target — and changes nothing else. Same 500 episodes, same network, same epochs. This is not an exotic sabotage; it's the most ordinary bug in robot learning: you computed your stats the week you were testing the rig with the block placed gently near the goal, and never recomputed after you started collecting for real.

Now look at what training reports:

Read the loss rows first. Nothing flags the narrow run — train loss is marginally LOWER, val loss is a wash. Both splits are normalized with the same skewed stats, so both fit and both agree with each other. Every curve in rerun looks like the healthy screenshot from Run It. If you shipped policies on loss curves, you would ship this one.

The rollouts tell a different story, and WHERE they tell it is the lesson. Split the 50 eval episodes by the block's starting distance from the target: for starts inside the region the stats covered (under 0.15 m), the narrow policy essentially matches the honest one — 8/19 vs 9/19, a single-episode difference that is noise at this sample size. For far starts it gives up a quarter of its competence: 15/31 against the honest run's 20/31. Five of the six episodes the narrow policy loses are far starts — the damage lands exactly on the states the stats never described, and no metric computed ON THE DATASET can see it, because the dataset is precisely the thing that got misdescribed.

The mechanism is that clamp from the model region, awake for the first time. Min-max stats from near-goal episodes declare that tee_x lives in roughly [-0.13, +0.13]. A block starting at 0.20 m normalizes to 1.5 — and clamps to 1.0. The policy doesn't see a far block; it sees a block pinned to the edge of a world that ends where the easy demos ended. Open the two recordings side by side and scrub a far-start episode: in the narrow run, policy/action saturates toward a block-edge that isn't where the block is, the pusher arrives short, strokes empty table, re-approaches, strokes again. It isn't confused. It is certain, and it's certain about a world 0.13 meters wide.

The transferable lesson, and it's worth saying plainly: loss curves measure the dataset you gave them, not the world. Normalization stats are part of the model. Version them, recompute them when the data changes, and when a policy trains clean but acts wrong, check what its inputs look like AFTER normalization — rr.log one histogram and this entire class of bug confesses in seconds.

One more measured fact, because this book doesn't hide inconvenient ones: train the narrow run at the chapter's full default scale — 600 epochs, hidden_dim 512 — and the failure heals. The bigger network, given twice the schedule, learns a working strategy for blocks pinned to the edge of its clamped world ("push toward the edge until the block walks into view") and finishes at 37/50 — above the honest default run's 31/50. That should unsettle you more than the failure did: the bug didn't go away, the policy papered over it, and nothing in your metrics distinguishes "fixed" from "compensated". Chapter 1.6 is an entire chapter about why evaluation this shallow will eventually lie to you.

Exercises

Four, in exercises/: a bug-hunt where every training metric is healthy and only rollouts complain, the rollout loop with the middle missing, and two where you commit to a prediction before the run is allowed to tell you the answer.

What's next

Your BC policy averages. On PushT the averaging is mostly survivable — scalar velocities, one expert, its self-disagreements small enough to blur rather than break. But you saw the residue even here: a third of the competence gone, stalls where the expert's hidden phase made one observation carry two labels. Human demonstrations are worse — you never push the block the same way twice, and when demonstrations genuinely disagree, the average of two good trajectories is a trajectory through the space between them, which is to say: through failure. Next chapter the policy stops predicting the next instant's average and starts committing to plans — chunks of future actions predicted together — and the stalls you found in the rerun traces tonight are the first thing that disappears.