LocomotionThe Quadruped Walks

Nobody writes the gait

Back in chapter 2.4 you had a walking quadruped, and you wrote its gait by hand. trot_action is a sinusoid — hips sweeping fore-and-aft, knees flexing on the swing — with amplitudes and a frequency you tuned until the legs happened to carry the body forward. It works, and it is a lie of sorts: you are the controller. Change the robot, change the floor, change the speed you want, and you are back at the whiteboard re-deriving sinusoids.

This chapter takes you out of the loop. You hand SAC — the off-policy learner you built from scratch in chapter 2.2 — the same quadruped and the same five-term reward (go forward, stay upright, hold your ride height, don't fall, don't thrash), and let it act. Out of thousands of episodes of trial and error, a repeating stride emerges. Nobody specified a gait; the reward and the physics did. That emergence is the whole chapter.

The env is unchanged — that is the point

We do not touch common/envs/quadruped. Same obs[23], same action[8], same reward weights chapter 2.4 shipped. If a gait is going to appear, it has to appear from the reward as given, on the body as built. The only thing that changes between 2.4 and 2.5 is who chooses the joint targets each step: a hand-written sinusoid, or a neural network trained to maximize return.

What the policy sees, and why (observation design)

A locomotion policy is only as good as its senses. The env's 23-number observation is a deliberate menu, and reading it is half the lesson:

• Joint angles + joint velocities (0..15) — proprioception. Where each of the eight leg joints is, and how fast it is moving. This is what a real robot reads off its motor encoders; it is the policy's sense of its own body.
• Torso height (16), up-vector (17..19) — posture. How tall the body rides and which way "up" points. The reward's height and upright terms are computed from exactly these, so the policy can see the very quantities it is graded on.
• Torso linear velocity (20..22) — how fast, which way. Entry 20 is the forward speed vx the reward pays for. Without it the policy is walking with its eyes shut to its own momentum.

Two design choices are worth pausing on. First, there is no clock and no foot-contact flag in the observation. A gait is periodic, but the policy is not handed the phase of the cycle — it has to infer where it is in its stride from the joint and body velocities. Second, orientation is an up-vector, not a quaternion: a Gaussian policy trips over the sign/wrap seam of a quaternion, while the up-vector's z-component is directly the number the reward reads. Good observation design is choosing coordinates the network can actually learn from.

The --blind_velocity flag zeroes obs 20..22 so you can measure what that forward-speed sense is worth — that is exercise 2.

What the policy commands, and why (action design)

The eight actions are residual position targets, not torques. The env commands DEFAULT_POSE + ACTION_SCALE * action to PD position servos, so:

• action 0 = stand. The servos already hold the crouch, so the do-nothing action doesn't fall over. The policy starts from a stable anchor and only has to learn the small offsets that turn standing into striding.
• the search is easier. Learning residuals around a good default is a far gentler optimization than learning raw joint torques from scratch, where do-nothing means collapse. This is the standard legged-RL setup for a reason, and it is why a from-scratch SAC can find a gait on a laptop at all.

The algorithm is 2.2's SAC, unchanged

There is almost nothing new in the learner — and that reuse is deliberate. The squashed-Gaussian actor, twin Q critics with target networks, the auto-tuned entropy temperature, one gradient step per environment step: all of it is chapter 2.2, pointed at a new env. Locomotion changes the data, not the algorithm.

LOG_STD_MIN, LOG_STD_MAX = -5.0, 2.0  class Actor(nn.Module):    """Squashed-Gaussian policy (ch2.2, unchanged): a state-dependent mean +    log-std, sampled and squashed through tanh into [-1, 1] — exactly the    residual-target range the env expects. The tanh log-prob correction keeps the    entropy term honest; drop it and SAC's exploration collapses."""     def __init__(self, obs_dim: int, act_dim: int, hidden_dim: int):        super().__init__()        self.trunk = nn.Sequential(            nn.Linear(obs_dim, hidden_dim), nn.ReLU(),            nn.Linear(hidden_dim, hidden_dim), nn.ReLU(),        )        self.mean = nn.Linear(hidden_dim, act_dim)        self.log_std = nn.Linear(hidden_dim, act_dim)     def forward(self, obs: torch.Tensor):        h = self.trunk(obs)        return self.mean(h), torch.clamp(self.log_std(h), LOG_STD_MIN, LOG_STD_MAX)     def sample(self, obs: torch.Tensor):        # (action, log_prob, deterministic_action); rsample = reparameterization        mean, log_std = self(obs)        normal = torch.distributions.Normal(mean, log_std.exp())        x = normal.rsample()        action = torch.tanh(x)        log_prob = normal.log_prob(x) - torch.log(1.0 - action.pow(2) + 1e-6)        return action, log_prob.sum(1, keepdim=True), torch.tanh(mean)  class Critic(nn.Module):    """A single Q(obs, action). We build TWO (twin critics) and bootstrap from    the MIN — clipped double-Q, the brace that stops Q chasing its own overshoot."""     def __init__(self, obs_dim: int, act_dim: int, hidden_dim: int):        super().__init__()        self.net = nn.Sequential(            nn.Linear(obs_dim + act_dim, hidden_dim), nn.ReLU(),            nn.Linear(hidden_dim, hidden_dim), nn.ReLU(),            nn.Linear(hidden_dim, 1),        )     def forward(self, obs: torch.Tensor, action: torch.Tensor) -> torch.Tensor:        return self.net(torch.cat([obs, action], dim=1))  obs_dim, act_dim = QuadrupedEnv.OBS_DIM, QuadrupedEnv.ACT_DIMactor = Actor(obs_dim, act_dim, args.hidden_dim).to(device)q1 = Critic(obs_dim, act_dim, args.hidden_dim).to(device)q2 = Critic(obs_dim, act_dim, args.hidden_dim).to(device)q1_targ = copy.deepcopy(q1).requires_grad_(False)q2_targ = copy.deepcopy(q2).requires_grad_(False)actor_opt = torch.optim.Adam(actor.parameters(), lr=args.lr)critic_opt = torch.optim.Adam(list(q1.parameters()) + list(q2.parameters()), lr=args.lr)log_alpha = torch.tensor(float(np.log(args.alpha)), dtype=torch.float32, device=device, requires_grad=args.autotune)alpha_opt = torch.optim.Adam([log_alpha], lr=args.lr) if args.autotune else None

The replay buffer stores terminated, not done — the same truncation lesson from 2.1/2.2, and it matters more here. A fall is a true terminal (the future really is zero), but running out the 500-step clock while still walking is a time limit you must bootstrap through. Conflate them and the policy learns that walking all the way to the horizon is worthless.

class ReplayBuffer:    """Fixed-size circular buffer of transitions — the off-policy bargain in one    data structure. We store `terminated` (a real fall), NOT `done`: a time-limit    truncation must still bootstrap the future value from next_obs, so only a true    fall masks it — the one thing that zeroes the locomotion return's future."""     def __init__(self, capacity: int, obs_dim: int, act_dim: int):        self.capacity = capacity        self.obs = np.zeros((capacity, obs_dim), dtype=np.float32)        self.actions = np.zeros((capacity, act_dim), dtype=np.float32)        self.rewards = np.zeros((capacity, 1), dtype=np.float32)        self.next_obs = np.zeros((capacity, obs_dim), dtype=np.float32)        self.terminated = np.zeros((capacity, 1), dtype=np.float32)        self.ptr, self.size = 0, 0     def add(self, obs, action, reward, next_obs, terminated):        i = self.ptr        self.obs[i], self.actions[i], self.rewards[i, 0] = obs, action, reward        self.next_obs[i], self.terminated[i, 0] = next_obs, float(terminated)        self.ptr = (self.ptr + 1) % self.capacity        self.size = min(self.size + 1, self.capacity)     def sample(self, batch_size: int):        idx = torch.randint(0, self.size, (batch_size,)).numpy()  # seeded torch RNG        t = lambda a: torch.as_tensor(a[idx], device=device)  # noqa: E731        return t(self.obs), t(self.actions), t(self.rewards), t(self.next_obs), t(self.terminated)  buffer = ReplayBuffer(args.buffer_size, obs_dim, act_dim)

The update is 2.2 verbatim:

def sac_update() -> dict:    """One SAC gradient step (ch2.2, verbatim): critic update, then actor +    temperature, then the soft target nudge. Locomotion changes the data, not the    algorithm — that reuse is the point."""    obs, action, reward, next_obs, terminated = buffer.sample(args.batch_size)    alpha = log_alpha.exp().detach()     with torch.no_grad():        next_action, next_logp, _ = actor.sample(next_obs)        min_next_q = torch.min(q1_targ(next_obs, next_action), q2_targ(next_obs, next_action)) - alpha * next_logp        target_q = reward + args.gamma * (1.0 - terminated) * min_next_q    q1_loss = F.mse_loss(q1(obs, action), target_q)    q2_loss = F.mse_loss(q2(obs, action), target_q)    critic_opt.zero_grad()    (q1_loss + q2_loss).backward()    critic_opt.step()     new_action, logp, _ = actor.sample(obs)    min_q = torch.min(q1(obs, new_action), q2(obs, new_action))    actor_loss = (alpha * logp - min_q).mean()  # ascend min_q, keep entropy high    actor_opt.zero_grad()    actor_loss.backward()    actor_opt.step()     if args.autotune:        alpha_loss = -(log_alpha.exp() * (logp.detach() + target_entropy)).mean()        alpha_opt.zero_grad()        alpha_loss.backward()        alpha_opt.step()     with torch.no_grad():  # Polyak-average the targets a hair toward online        for online, targ in ((q1, q1_targ), (q2, q2_targ)):            for p, tp in zip(online.parameters(), targ.parameters()):                tp.mul_(1.0 - args.tau).add_(args.tau * p)    return {"q_loss": (q1_loss + q2_loss).item() / 2.0, "actor_loss": actor_loss.item(),            "alpha": float(log_alpha.exp().item()), "q_value": min_q.mean().item()}

A first taste of domain randomization

There is one genuinely new line, and it lives at reset. Each training episode, we scale the torso mass by a small random factor (default ±15%):

# One env, one gradient step per env step (ch2.2's SAC shape). The quadruped is# the ch2.4 reward env verbatim — we change NOTHING about it; the gait must emerge# from the reward as shipped. DOMAIN RANDOMIZATION lives here: each training reset# scales the torso mass by a small seeded factor so the policy meets a slightly# different body every episode. Eval always uses the NOMINAL body (comparable).env = QuadrupedEnv()torso_id = env.model.body("torso").iddefault_torso_mass = float(env.model.body_mass[torso_id])episode_count = 0  def _blind(obs: np.ndarray) -> np.ndarray:    """Obs-design ablation: optionally hide torso linear velocity (obs 20..22) —    the policy loses its direct forward-speed sense; watch the walk suffer."""    if args.blind_velocity:        obs = obs.copy()        obs[20:23] = 0.0    return obs  def env_reset(randomize: bool) -> np.ndarray:    """Reset with a fresh deterministic seed per episode. When `randomize`, jitter    the torso mass (the DR preview); otherwise restore the nominal body."""    global episode_count    if randomize and args.domain_rand:        factor = float(dr_rng.uniform(1.0 - args.dr_mass_frac, 1.0 + args.dr_mass_frac))        env.model.body_mass[torso_id] = default_torso_mass * factor    else:        env.model.body_mass[torso_id] = default_torso_mass    obs = env.reset(seed=args.seed + episode_count)    episode_count += 1    return _blind(obs)

Now the policy never meets the exact same body twice, so it cannot memorize one body's precise dynamics — it has to find a gait that works across a little variation, which is a more robust gait. This is a preview, one randomized number, of chapter 2.7, where randomizing friction, mass, latency, and external shoves becomes the entire story of crossing the reality gap. The --no-domain-rand flag turns it off so you can compare. Note that evaluation always uses the nominal body, so the gait-emergence curve is measured on one fixed robot and stays comparable across the run.

Run it — and watch the gait emerge

python curriculum/phase2_reinforcement/ch2.5_walk/walk.py --seed 0 --device cpu

On a CPU laptop the default 60k-step config takes about 8 minutes. Watch the forward distance climb off the standing baseline as a stride settles in:

step   5000/60000  eval_return 179.1  fwd_dist +1.596m  fwd_vel +0.327m/s  len 255
step  15000/60000  eval_return 147.1  fwd_dist +1.247m  fwd_vel +0.358m/s  len 219
step  20000/60000  eval_return 307.6  fwd_dist +3.370m  fwd_vel +0.475m/s  len 357
step  30000/60000  eval_return 330.9  fwd_dist +3.732m  fwd_vel +0.517m/s  len 369
step  45000/60000  eval_return 374.6  fwd_dist +4.467m  fwd_vel +0.584m/s  len 386
step  60000/60000  eval_return 247.2  fwd_dist +3.025m  fwd_vel +0.581m/s  len 249
eval: forward +3.025 m  vel +0.581 m/s  return 247.2  len 249
      bar: scripted trot +2.154 m / return 306.6  (random ~-0.30 m, stand ~-0.01 m)
gait emergence: walks (fwd>0.5m) at 5000 env steps

The honest read is richer than "did it work". A gait emerges, and it is fast. By 20k steps the policy is walking, and it walks further than the hand-scripted trot — +3.0 m of forward travel versus the trot's +2.15 m, at nearly triple the forward speed. Nobody wrote that gait; SAC found it by maximizing the same five-term reward the trot was tuned against.

But read the length column. The trot rides out the full 500-step horizon; the emergent gait covers more ground in about 249 steps and then falls. So on total return the trot still wins, 307 to 247: the learned gait is faster but less stable — it has learned to sprint, not yet to sprint and stay up for the full ten seconds. That asterisk is the lesson. Emergent does not mean robust. A reward that pays for forward velocity gets forward velocity; keeping the body up for the whole episode is a harder, slower-to-learn skill on top. (Notice too that the return peaks around 45k and dips by 60k — RL curves are non-monotonic, and the "best" policy is not always the last one.)

The numbers above are the measured seed-0 run. RL is noisy, so the exercises read the signal across several seeds and never off a single run.

Emergent is not robust

Here is the trap the emergence sets. A gait appeared, on its own, and it covers more ground than the gait you hand-tuned in 2.4 — +3.0 m against the scripted trot's +2.15 m, at nearly triple the forward speed. The tempting read is that SAC beat you. Before you accept it, predict one number: over the full ten-second episode, which gait earns more return — the learned sprint or the hand-written trot?

The measurement says the trot, 307 to 247, and the reason is one column over. Read the len field in the run above: the trot rides out all 500 steps; the emergent gait covers its distance in about 249 steps and then falls. It learned to sprint, not to sprint and stay up. A reward that pays for forward velocity gets forward velocity; standing for the whole horizon is a separate, harder skill it has not bought yet. Emergent does not mean robust — the gait is real, and it is fragile, and no single training curve told you which; only rolling it out to the horizon did. (That fragility is exactly what the Scale Lab's extra samples, and chapter 2.7's randomization, exist to buy back.)

The Scale Lab

The free-tier run already shows the gait emerge and outrun the scripted trot on distance. What it has not bought is stability — the fall at ~249 steps. The Scale Lab runs the identical file with a bigger network and far more environment steps on a GPU (--hidden_dim 512 --total_steps 500000), where the open question is whether those extra samples turn the fast-but-fragile sprint into a gait that also rides the full 500-step horizon — so that return, not just distance, clears the trot. It is optional, and its numbers are a Scale-Lab measurement on paid hardware, never a free-tier promise. The lesson of this chapter — that a gait emerges at all, from the reward alone, on a laptop — stands on its own.

Read the real thing

Our gait is real, and it is fragile. walk.py's train loop hands SAC the five-term reward and one quadruped body, takes one update step per env step, and out of it a stride emerges — then that same policy outruns the scripted trot on distance and falls at ~249 steps. Nobody scripted the gait; nobody made it robust either. Closing that gap — emergent but fragile → production-robust — is the entire job of a real locomotion loop, and you can read exactly how it is done.

Clone leggedrobotics/legged_gym at the pinned commit 8fa29ac and open legged_gym/envs/base/legged_robot.py. Scroll to the reward block (roughly lines 846–933). Where we shipped five terms, the base robot defines about nineteen _reward_* methods. _reward_tracking_lin_vel and _reward_tracking_ang_vel (~lines 903–908) reward following a commanded velocity — ours just pays for raw forward speed. _reward_feet_air_time (~line 913) pays for long swing phases: it literally shapes the stride rather than hoping one appears. And a wall of penalties — _reward_action_rate, _reward_torques, _reward_dof_acc, _reward_orientation, _reward_collision (~lines 854–879) — taxes jerky, expensive, or unsafe motion, which is most of what makes a gait smooth enough to survive real hardware. The weights that fuse nineteen terms into one scalar live in legged_gym/envs/base/legged_robot_config.py, in LeggedRobotCfg.rewards.scales (~line 165).

Two more machines in the same two files are what make the gait transfer. Domain randomization: _process_rigid_shape_props (~line 546) jitters ground friction, _process_rigid_body_props (~line 564) jitters base mass, and _push_robots (~line 698) shoves the torso mid-episode — all switched on in the config's LeggedRobotCfg.domain_rand block (~line 156). That is the full version of the one jittered torso-mass line you previewed at reset. And curriculum: _update_terrain_curriculum (~line 722) ramps terrain difficulty as the robot proves it can cover ground, while update_command_curriculum (~line 741) widens the commanded-velocity range only once tracking is good enough. The policy earns harder problems instead of being handed them cold — the reason a production gait learns to stay up, not just to sprint.

Two things you already built transfer unchanged, and spotting them is the payoff. The action is a residual position target around a default stance — in LeggedRobotCfg.control (~lines 120–131), control_type = 'P' and action_scale = 0.5, so action zero stands, exactly our anchor. And the episode-boundary bootstrap that separates a real fall from a time-limit truncation — our terminated-not-done lesson — is what the sibling rsl_rl PPO loop consumes on the training side.

Read next: legged_gym/envs/base/legged_robot.py, the _reward_* block (lines ~846–933), then the scales, domain_rand, and curriculum flags in legged_robot_config.py. Pin the commit above; do not read against the moving default branch. That is the difference between a gait that emerges and a gait that ships.