The whole course, run once, on a real robot's body
Every method in this book ran against an environment we wrote. PushT, ALOHA-cube, the cartpole — small, clean, ours. That was the right call for learning a mechanism: when the sim is a hundred lines you can read, you always know exactly what the algorithm saw. But it left one honest question open, and it is the only question Phase 5 exists to answer: does any of this survive contact with a real robot?
This chapter runs the entire LeRobot loop — drive, record, train, deploy, evaluate — on
the SO-101's real morphology: the same $150 arm the graduation bridge (G1) sends you to
buy, loaded straight from google-deepmind/mujoco_menagerie. Six sts3215 position-servo
joints, the manufacturer's link lengths, the manufacturer's joint limits, a bundled wrist
camera. Not a simplified stand-in — the real body, in MuJoCo, free-tier, on your laptop. And
the format and the control loop are byte-for-byte what you would run on the metal.
Open real_loop.py. Nine regions: setup, fetch, env, expert, record,
model, train, deploy, report. The middle seven are the loop.
First, fetch the real robot — without committing a single mesh
# Reproducibly fetch the SO-101 model — Apache-2.0, from google-deepmind/mujoco_menagerie# (`robotstudio_so101`). We NEVER commit meshes (invariant #5): the MJCF + 18 STL assets download# once into a gitignored cache, integrity-checked, and skip on every run after. We pull through the# jsdelivr CDN (a GitHub mirror keyed by the same ref) rather than raw.githubusercontent — same# bytes, but built to serve many small files without the burst rate-limiting a from-scratch fetch# trips. The two XMLs we parse are sha-pinned (they define the robot); the binary meshes are# size-checked (a rate-limited HTML error page is a few hundred bytes — a real STL is tens of KB),# and the final proof of a good fetch is that MuJoCo compiles the model. The commit is UN-PINNED on# purpose (G1's doctrine: the upstream moves), so an author bumps --menagerie_ref without this file._RAW = "https://cdn.jsdelivr.net/gh/google-deepmind/mujoco_menagerie@{ref}/robotstudio_so101/{path}"_XML_SHA = { # sha256 of the two MJCFs we depend on; a changed robot definition trips this "so101.xml": "5ad49f2b45c083baac9ffe5d4d3213a5da7eac8039095bb2df177a697aae8308", "scene_box.xml": "14c3826f23889587e63b88ef9a0e7d72318ffe13078fad8287c56dbdb9ba9ad8",}_ASSETS = [ # the 18 STL meshes so101.xml references (kept as a literal list — no directory crawl) "waveshare_mounting_plate_so101_v2.stl", "sts3215_03a_v1.stl", "motor_holder_so101_base_v1.stl", "wrist_roll_follower_so101_v1.stl", "moving_jaw_so101_v1.stl", "base_motor_holder_so101_v1.stl", "upper_arm_so101_v1.stl", "wrist_roll_pitch_so101_v2.stl", "under_arm_so101_v1.stl", "rotation_pitch_so101_v1.stl", "motor_holder_so101_wrist_v1.stl", "sts3215_03a_no_horn_v1.stl", "base_so101_v2.stl", "moving_jaw_so101_gripper_v1.stl", "wrist_roll_follower_so101_camera_mount.stl", "wrist_roll_follower_so101_gripper_part0_v1.stl", "moving_jaw_so101_gripper_part0_v1.stl", "moving_jaw_so101_gripper_part1_v1.stl",] def _download(path: str, dest: Path, ref: str) -> None: """Fetch one file with a few retries (raw.githubusercontent rate-limits bursts).""" url = _RAW.format(ref=ref, path=path) for attempt in range(5): try: with urllib.request.urlopen(url, timeout=30) as resp: dest.write_bytes(resp.read()) return except Exception as err: # noqa: BLE001 — network flake / rate-limit: back off, retry, then surface if attempt == 4: sys.exit(f"failed to fetch {url}: {err}") time.sleep(1.5 * (attempt + 1)) # raw.githubusercontent throttles bursts; ease off def fetch_so101(cache: Path, ref: str) -> Path: """Download the SO-101 MJCF + meshes into `cache` (skip-if-present), then compile to prove it.""" scene = cache / "scene_box.xml" assets = cache / "assets" have_all = scene.is_file() and all((assets / a).is_file() for a in _ASSETS) if not have_all: assets.mkdir(parents=True, exist_ok=True) for name, want in _XML_SHA.items(): _download(name, cache / name, ref) got = hashlib.sha256((cache / name).read_bytes()).hexdigest() if got != want: sys.exit(f"{name} sha256 {got} != expected {want} (menagerie_ref={ref!r} moved the robot def)") for a in _ASSETS: _download(f"assets/{a}", assets / a, ref) if (assets / a).stat().st_size < 10_000: # an HTML error page, not a mesh sys.exit(f"asset {a} is {(assets / a).stat().st_size} bytes — a corrupt/rate-limited download; re-run") model = mujoco.MjModel.from_xml_path(str(scene)) # the real integrity check: does it compile? assert (model.nu, model.nq) == (6, 13), f"unexpected SO-101 model: nu={model.nu} nq={model.nq}" (cache / "manifest.json").write_text(json.dumps( # a pointer, never the meshes themselves {"ref": ref, "source": "google-deepmind/mujoco_menagerie/robotstudio_so101", "xml_sha256": _XML_SHA, "assets": sorted(_ASSETS)}, indent=2, sort_keys=True) + "\n") return scene scene_path = fetch_so101(args.cache, args.menagerie_ref)print(f"SO-101 model: {scene_path} (cached; ref={args.menagerie_ref})")The SO-101 model is an MJCF plus eighteen STL meshes, about 17 MB. Invariant #5 is absolute:
no binaries in git. So we fetch it — once — into a gitignored cache, and skip the download
on every run after. Reproducibility without a committed blob comes from three checks, cheapest
first: the two XMLs we actually parse are sha-pinned (a changed robot definition trips an
assertion, not a silent surprise); the binary meshes are size-checked (a rate-limited HTML
error page is a few hundred bytes, a real mesh is tens of KB); and the final proof of a good
fetch is that MuJoCo compiles the model — nu=6, nq=13 or we stop.
We pull through a CDN mirror keyed by a git ref, and that ref is un-pinned on purpose.
This is the same doctrine G1 states for the LeRobot CLI: the upstream moves weekly, so we hold
a template, not a frozen commit. An author bumps --menagerie_ref and the sha-checks tell
them immediately whether the robot definition moved under them.
The env is the real arm — we do not simplify it
# The SO-101 reach env, wrapping the Menagerie model. This is the "real arm's body": we do NOT# simplify the kinematics — the six sts3215 position servos, link lengths, and joint limits are# the manufacturer's. reset() respawns the box on the floor in a reachable patch (via its# freejoint, exactly as ch1.3's env writes cube qpos), and obs/step/dist mirror the PushT/ALOHA# env contract so the BC loop below is unchanged from ch1.1. Bitwise-deterministic on one CPU.HOME = np.array([0.0, -1.57, 1.57, 1.0, 0.0, 0.0]) # a raised, out-of-the-way rest poseFRAME_SKIP = 10 # 100 Hz physics (timestep 0.005) / 10 Hz control class SO101ReachEnv: def __init__(self, scene_xml: Path): self.model = mujoco.MjModel.from_xml_path(str(scene_xml)) self.data = mujoco.MjData(self.model) self._box_qadr = self.model.jnt_qposadr[self.model.body("box").jntadr[0]] # box freejoint qpos start (=6) self._tip = self.model.site("gripperframe").id self._box = self.model.body("box").id self._lo = self.model.actuator_ctrlrange[:, 0] self._hi = self.model.actuator_ctrlrange[:, 1] def reset(self, seed: int) -> np.ndarray: rng = np.random.Generator(np.random.PCG64(seed)) mujoco.mj_resetData(self.model, self.data) self.data.qpos[:6] = HOME bx, by = rng.uniform(0.26, 0.31), rng.uniform(-0.14, 0.14) # a reachable floor patch in front self.data.qpos[self._box_qadr:self._box_qadr + 3] = [bx, by, 0.03] self.data.qpos[self._box_qadr + 3:self._box_qadr + 7] = [1, 0, 0, 0] self.data.ctrl[:] = HOME mujoco.mj_forward(self.model, self.data) return self._obs() def _obs(self) -> np.ndarray: return np.concatenate([self.data.qpos[:6], self.data.body(self._box).xpos]).astype(np.float32) def step(self, ctrl: np.ndarray) -> np.ndarray: self.data.ctrl[:] = np.clip(ctrl, self._lo, self._hi) # position servos track the target for _ in range(FRAME_SKIP): mujoco.mj_step(self.model, self.data) return self._obs() def gripper_to_box(self) -> float: return float(np.linalg.norm(np.array(self.data.site(self._tip).xpos) - np.array(self.data.body(self._box).xpos))) env = SO101ReachEnv(scene_path)SO101ReachEnv is a thin wrapper, and thin is the point. Unlike the Phase-1 envs we
authored line by line, here the physics, the kinematics, and the actuators are the
manufacturer's. We add exactly two things: a box, respawned on the floor in a reachable patch
each reset (via its free joint — the same trick ch1.3's env used to place its cube), and an
obs/step contract that matches PushT and ALOHA so the behavior-cloning loop below is
unchanged from ch1.1. The observation is nine numbers — six joint angles and the box's
(x, y, z). The action is six joint-position targets: the SO-101's servos track them,
exactly as a real STS3215 does.
Driving is teleoperation, scripted so CI can diff it
# The scripted expert = your teleoperation, scripted so CI can diff runs. A leader arm hands you# joint targets; here a tiny controller computes them. The reach is mostly a fixed "lower-to-the-# table" pose (shoulder_lift/elbow/wrist), STEERED by shoulder_pan toward the box's azimuth — the# one joint that must depend on where the box is. It commands the TARGET pose every step and lets# the position servos interpolate (a stable feedback law, not an open-loop trajectory — that is why# the clone learns it without drifting). PAN_GAIN is a one-number fit of pan -> tip azimuth.REACH_POSE = np.array([0.2, 0.3, 0.7]) # shoulder_lift, elbow_flex, wrist_flex — lowers the gripper to the tablePAN_GAIN = 0.87 def expert_action(box_xyz: np.ndarray) -> np.ndarray: """The 6-D joint-position target that reaches `box_xyz`. Deterministic in the box position.""" pan = -np.arctan2(box_xyz[1], box_xyz[0]) / PAN_GAIN # aim shoulder_pan at the box's azimuth return np.array([pan, REACH_POSE[0], REACH_POSE[1], REACH_POSE[2], 0.0, 0.3], dtype=np.float32)On real hardware you would grab a leader arm and move it; a follower mirrors you, and the
joint angles it reports are your actions (that is G1's lerobot-teleoperate). We can't replay
a human hand deterministically, so a tiny controller stands in. The reach is almost entirely a
fixed lower-to-the-table pose — shoulder-lift, elbow, wrist held at constants — steered by
the one joint that must know where the box is: shoulder_pan, aimed at the box's azimuth.
It commands the target pose every step and lets the position servos interpolate. That detail
matters more than it looks: a stable feedback law (drive toward a fixed goal) is something a
clone can learn from any starting state, where an open-loop trajectory (do X at step 3) drifts
the moment the clone's timing slips. Exercise 3 has you write that shoulder_pan line.
Recording writes the real format — the same one hardware writes
# Record a REAL LeRobotDataset — the identical create -> add_frame -> save_episode -> finalize# sequence ch0.4 and `lerobot-record` use, STATE-only (use_videos=False) so the bytes are# reproducible on CPU. Episode i places the box with env seed (seed + i): a different, repeatable# reach each time. We store the obs we ACTED ON (pre-step) — the off-by-one ch0.4 warned about.def build_features() -> dict: return {"observation.state": {"dtype": "float32", "shape": (OBS_DIM,), "names": STATE_NAMES}, "action": {"dtype": "float32", "shape": (ACT_DIM,), "names": JOINT_NAMES}} def record_demos(dataset_root: Path) -> None: from lerobot.datasets.lerobot_dataset import LeRobotDataset # lazy: pulls in torch/datasets if dataset_root.exists(): # regenerate every run: a stale dir from another --seed is silent wrong data (ch1.1) shutil.rmtree(dataset_root) dataset = LeRobotDataset.create(repo_id="zero2robot/so101_reach", fps=10, features=build_features(), root=dataset_root, robot_type="so101_follower", use_videos=False) for i in range(args.demos): obs = env.reset(args.seed + i) target = expert_action(obs[6:9]) # the reach target for THIS box (constant over the episode) for _ in range(args.steps): dataset.add_frame({"observation.state": obs, "action": target, "task": TASK}) obs = env.step(target) dataset.save_episode() dataset.finalize() # compute stats, write meta/*.parquet — a dataset you could push to the Hub dataset_root = args.out / "dataset"record_demos(dataset_root)This is not a schema we invented and keep in sync. It is LeRobotDataset.create → add_frame → save_episode → finalize — the actual pinned lerobot API, the identical call record.py
made in ch0.4 and lerobot-record makes on hardware. Format parity with everything you have
trained on all course is free by construction. We record state-only (use_videos=False)
so the run is byte-reproducible on one CPU — and, honestly, because the missing camera is part
of the reality gap we are about to name. We store the observation we acted on (pre-step), the
off-by-one ch0.4 warned you about.
Training is ch1.1's behavior cloning, retargeted to six joints
# Load the recorded dataset BACK and train BC on it — the clone never sees the expert, only the# recording (the whole point of the loop). The model is ch1.1's MLP, retargeted 9 -> 6. One change# matters for a robot: the six joints move over VERY different ranges (shoulder_lift swings ~1.7 rad,# shoulder_pan ~0.5), so we compute the MSE in NORMALIZED action space — otherwise the big joints# dominate the loss and the tiny box-dependent pan signal (the only thing that matters) is ignored.from lerobot.datasets.lerobot_dataset import LeRobotDataset # noqa: E402 frames = LeRobotDataset("zero2robot/so101_reach", root=dataset_root).hf_dataset.with_format("numpy")obs_np = np.stack(frames["observation.state"]).astype(np.float32) # (N, 9)act_np = np.stack(frames["action"]).astype(np.float32) # (N, 6)obs_min, obs_max = obs_np.min(0), obs_np.max(0)obs_range = np.where(obs_max - obs_min < 1e-4, np.float32(1.0), obs_max - obs_min)act_min, act_max = act_np.min(0), act_np.max(0)act_range = np.where(act_max - act_min < 1e-4, np.float32(1.0), act_max - act_min) class BCPolicy(nn.Module): """obs float32[9] -> action float32[6]. Same 3-layer MLP as ch1.1; normalization lives inside as buffers so the checkpoint / ONNX carry their own stats and the deploy loop feeds raw obs.""" def __init__(self, hidden_dim: int): super().__init__() self.net = nn.Sequential( nn.Linear(OBS_DIM, hidden_dim), nn.ReLU(), nn.Linear(hidden_dim, hidden_dim), nn.ReLU(), nn.Linear(hidden_dim, ACT_DIM), ) for name, stat in [("obs_min", obs_min), ("obs_range", obs_range), ("act_min", act_min), ("act_range", act_range)]: self.register_buffer(name, torch.from_numpy(np.asarray(stat, np.float32))) def net_norm(self, obs: torch.Tensor) -> torch.Tensor: # raw obs -> NORMALIZED action (train target) normalized = 2.0 * (obs - self.obs_min) / self.obs_range - 1.0 return self.net(normalized.clamp(-1.0, 1.0)) def forward(self, obs: torch.Tensor) -> torch.Tensor: # raw obs -> RAW joint targets (deploy / ONNX) return (self.net_norm(obs) + 1.0) / 2.0 * self.act_range + self.act_min policy = BCPolicy(args.hidden_dim).to(device)The clone never sees the expert. It loads the recorded dataset back off disk and fits
observation → action with plain MSE — the whole point of the loop is that training consumes
your recording, not your recorder. The network is ch1.1's three-layer MLP, retargeted 9 → 6.
One change is load-bearing, and it is a lesson about robots specifically. The six joints move
over wildly different ranges — shoulder-lift swings more than a radian on every reach, while
the box-dependent shoulder_pan correction is a few tenths. Compute the MSE on raw joint
angles and the big joints drown the tiny pan signal — the network learns to sweep down and
ignore where the box is, and the clone reaches the same spot every time. So we take the loss in
normalized action space, where every joint weighs equally and the pan signal survives. (You
will feel the raw-space failure if you look for it; the normalized loss is why the clone works.)
optimizer = torch.optim.Adam(policy.parameters(), lr=args.lr)train_obs = torch.from_numpy(obs_np).to(device)train_act = torch.from_numpy(act_np).to(device)# The MSE target is the NORMALIZED action, so every joint weighs equally (see the model region).target_norm = 2.0 * (train_act - policy.act_min) / policy.act_range - 1.0shuffle = torch.Generator().manual_seed(args.seed)train_loss, global_step = float("nan"), 0for epoch in range(args.epochs): epoch_loss, num_batches = 0.0, 0 for batch in torch.randperm(len(train_obs), generator=shuffle).split(args.batch_size): loss = nn.functional.mse_loss(policy.net_norm(train_obs[batch]), target_norm[batch]) optimizer.zero_grad() loss.backward() optimizer.step() epoch_loss, num_batches = epoch_loss + loss.item(), num_batches + 1 if args.rerun: rr.set_time("step", sequence=global_step) rr.log("policy/loss/train", rr.Scalars([loss.item()])) global_step += 1 train_loss = epoch_loss / num_batches if epoch % 50 == 0 or epoch == args.epochs - 1: print(f"epoch {epoch:3d} train_loss {train_loss:.5f}")torch.save(policy, args.out / "bc_policy.pt")Deploying is four lines, and evaluating is ch1.6
# Deploy the clone into the sim SO-101 (policy -> d.ctrl) and evaluate with ch1.6 rigor: a success# RATE over held-out box placements (seeds 10_000+, never trained on), not one hero rollout. We run# the clone AND two baselines that must FAIL — a no-op that holds the rest pose, and a random-target# flail — so the headline is a DIRECTION (clone >> baselines), the seed-robust claim. --break obs_swap# feeds the clone an observation whose box_x/box_y are transposed vs the recording: the ch0.4 lesson,# now on a real arm — training is spotless, the arm reaches the wrong way.def rollout(kind: str, episode: int) -> tuple[bool, float, np.ndarray]: obs = env.reset(10_000 + episode) # held-out placements, disjoint from the seed+i training set rng = np.random.Generator(np.random.PCG64(args.seed * 1000 + episode)) tips = [] for _ in range(args.steps): model_in = obs.copy() if kind == "clone" and args.break_mode == "obs_swap": model_in[6], model_in[7] = model_in[7], model_in[6] # deploy obs != record obs if kind == "noop": ctrl = HOME.astype(np.float32) elif kind == "random": ctrl = rng.uniform(env._lo, env._hi).astype(np.float32) else: with torch.no_grad(): ctrl = policy(torch.from_numpy(model_in).to(device).unsqueeze(0))[0].cpu().numpy() obs = env.step(ctrl) tips.append(np.array(env.data.site(env._tip).xpos)) return env.gripper_to_box() < args.success_tol, env.gripper_to_box(), np.array(tips) def success_rate(kind: str) -> float: return sum(rollout(kind, e)[0] for e in range(args.eval_episodes)) / args.eval_episodes clone_rate = success_rate("clone")noop_rate = success_rate("noop")random_rate = success_rate("random")print(f"eval: clone {clone_rate:.2f} | no-op {noop_rate:.2f} | random {random_rate:.2f} " f"(success = gripper within {args.success_tol} m of the box, break={args.break_mode or 'none'})") # Capture two gripper-tip paths for the toy NOW, while the policy is still on `device` (the ONNX# export below moves it to CPU): the clone reaching the box, and the same box mis-wired (obs_swap)._, _, clone_tips = rollout("clone", 0)saved_break, args.break_mode = args.break_mode, "obs_swap"_, _, swap_tips = rollout("clone", 0)args.break_mode = saved_breakbox0 = env.reset(10_000)[6:9]obs → policy → d.ctrl → step. That is the entire deployment surface — the sim mirror of the
get_observation → select_action → send_action loop G1 shows you on real hardware. Everything
hard you built in Phase 1 exists to make one of those lines good; the rest is the robot.
And we evaluate it the ch1.6 way: a success rate over held-out box placements (seeds the clone never trained on), against two baselines that must fail — a no-op that holds the rest pose, and a random flail. The headline is a direction, not a number: the recorded-then-cloned policy reproduces the reach clearly above both baselines, on every seed. That is the whole claim — the loop closes end-to-end on the real arm's body. It is a claim about the loop, not about manipulation being solved, and we report the order because the exact rate shifts with the platform's contact and servo settling (ch1.6). The clone matching the scripted expert is the loop working, not a performance flex — a scripted reach is an easy, low-variance target, which is exactly the right bar for a mechanism demo.
What the twin gives you — and what it cannot
Here is the honest core, and it is the reason this chapter is the second-to-last thing you do, not the last.
What you just did transfers to hardware unchanged. You drove a real arm's morphology, recorded the real LeRobot dataset format, trained a policy on it, and redeployed it through the real control loop. Buy the SO-101 and every one of those skills is already yours — that is the entire thesis of G1.
What the twin cannot give you is the reality gap, and you must not let a clean 100% here paper over it. Servo backlash. Friction that isn't in the model. Latency between reading an observation and the motor moving. Camera noise and lighting and a background that shifts at 6pm. Calibration drift between two "identical" arms. The Menagerie model even tells you this itself: its servo gains are a calculated approximation, not the real STS3215 gains — the MJCF says so in a comment. This chapter is morphology plus the loop, not torque-level fidelity, and it reaches for a box rather than pinching and lifting it precisely because a frictional grasp is where the gap bites hardest. Those gaps are not a footnote. They are the whole reason real hardware exists, and closing them is what your weeks on a real arm will be spent on. They stay reading — that is G1.
Break it the way the metal breaks
python real_loop.py --seed 0 --break obs_swap
The most common way a working policy fails on a real robot is not a bad model — it is a bad
wiring between your recorder and your deploy script. --break obs_swap simulates exactly
that: the deploy code reads box_x and box_y in the opposite order than the recording
wrote them. Training is untouched — same dataset, same clean loss near 1e-5. And the arm
confidently reaches the wrong way, its success collapsing from ~1.00 to ~0.20. No loss curve
could have warned you, because training never saw the swap; the bug lives entirely at the
seam where your policy meets the world. This is the ch0.4 lesson — record obs must equal deploy
obs — now with a physical consequence, and it is why G1 insists your calibration id matches
across record, train, and rollout. Exercise 2 makes you generate this failure and explain why
every training metric you watched stayed green.
Where you are
You have run the whole loop, from a blank robot model to a deployed policy, on a real arm's body. The last gap — the reality gap — is the one thing sim could not hand you, and it is the one thing worth crossing on real hardware. Two bridges remain: G1, where you buy the arm and run this exact loop on the metal, and G2/G3, where you decide what your robot should do. You did not need to wait for a robot to learn robotics. You needed the robot to find out you already had.