Plan Through Your EngineSampling-Based MPC (CEM / MPPI)

Every controller you built until now learned. This one doesn't.

Behavior cloning copied a demonstrator. PPO and SAC learned a policy the hard way — thousands of episodes of acting, failing, and nudging weights until a network mapped states to good actions. Different as they are, they share one assumption so deep you probably never questioned it: that controlling a robot means first learning a policy. Pour experience in, read actions out.

This chapter throws the network away.

You spent chapters 3.3 through 3.6 prying open mj_step — integrators, joints as constraints you solve for, contact as a complementarity you project — until the simulator stopped being a black box and became an engine you understand. Step back and notice what an engine actually is: a function that takes a state and an action and hands you the next state. That is the one thing you need to plan. If you can ask "what happens if I do this?" and get an honest answer, you don't have to learn what to do — you can search for it, live, at every control step. That is Model Predictive Control, and it is how a large fraction of real robots are actually driven.

The task: swing a pole up with no demonstrator and no reward to learn from

Cartpole again, but harder, and pointed the other way. In chapter 2.1 the pole started near upright and PPO learned to balance it. Here the pole starts hanging straight down, and the only actuator pushes the cart sideways. You cannot torque the pole up directly — the system is underactuated. The only way up is to pump: rock the cart to build the pole's energy, let it swing higher each time, and catch it at the top. It is the canonical demonstration that planning beats reacting, because acting greedily — always reducing your cost right now — never lets the pole fall the "wrong" way long enough to gain the momentum that brings it up.

There is no training run in this chapter. No demonstrator, no learned reward, no replay buffer. Just a model and a search. The pole swings up anyway.

The idea, in four steps

At every control step, from the state you are actually in:

1. Sample N candidate action sequences, each H steps long — a spray of possible short-term plans.
2. Roll each one forward through the model and score it with a cost you simply write down (upright pole, centered cart, settled motion). No network estimates the value; the engine tells you exactly what each plan leads to.
3. Refine the distribution you're sampling from toward the plans that scored well, and repeat step 1–2 a couple of times.
4. Execute only the first action of the best plan, let the real world advance one tick, and re-plan from where you actually landed.

Step 4 is the "predictive" and the "receding horizon" in Model Predictive Control: you always re-decide from the true current state, throwing away the tail of last step's plan, because no model is perfect and the world drifts. You plan far, commit little, and plan again.

The model is a second copy of the engine

MPC's whole premise is that you have a model — something you can fast-forward and, crucially, rewind, so you can try one plan, snap back to the start, and try the next. We keep the real world and the model as two separate objects on purpose. The planner may only touch sim; the true state only ever advances through env. Saving and restoring the model's state is just reading and writing MuJoCo's qpos/qvel — the full state the engine needs to continue:

# The MODEL you plan through. MPC's whole premise is that you HAVE one: a thing# that maps (state, action) -> next state, that you can fast-forward and rewind at# will. Here it is a second copy of the cartpole simulator — the same mj_step you# took apart in ch3.3-3.6. We keep the "real world" (env) and the "model" (sim)# as SEPARATE objects on purpose: the planner may only touch `sim`; the true state# only ever advances through `env.step`. Here model == world (a PERFECT model);# ch3.6 already measured what happens when they differ — hold that thought for the# ceiling at the end of the chapter.env = CartpoleEnv()   # the real world: we read its state and apply one action per ticksim = CartpoleEnv()   # the imagined model: the planner rolls candidate plans through THISCONTROL_SUBSTEPS = CartpoleEnv.FRAME_SKIP  # each control step = this many mj_step physics ticks (50 Hz control)SLIDER = sim.model.joint("slider").qposadr[0]  # cart position index in qpos/qvelHINGE = sim.model.joint("hinge").qposadr[0]    # pole angle index (0 = upright, pi = hanging down)  def get_state(e: CartpoleEnv) -> tuple[np.ndarray, np.ndarray]:    """Snapshot the full dynamical state (qpos, qvel) — everything mj_step needs to continue."""    return e.data.qpos.copy(), e.data.qvel.copy()  def set_state(e: CartpoleEnv, qpos: np.ndarray, qvel: np.ndarray) -> None:    """Rewind the model to a saved state. mj_forward refreshes derived quantities    (site positions, etc.) so the next mj_step continues from EXACTLY here."""    e.data.qpos[:] = qpos    e.data.qvel[:] = qvel    e.data.ctrl[:] = 0.0    mujoco.mj_forward(e.model, e.data)  def step_model(e: CartpoleEnv, action: float) -> None:    """Advance the model one control step: hold `action` for CONTROL_SUBSTEPS ticks."""    e.data.ctrl[0] = float(np.clip(action, -1.0, 1.0))    for _ in range(CONTROL_SUBSTEPS):        mujoco.mj_step(e.model, e.data)  def state_cost(e: CartpoleEnv, action: float) -> float:    """The TASK, written down as a cost to minimize — there is no learning here, you    just SAY what 'good' means. Swing-up wants the pole UPRIGHT, the cart near center,    and the motion SETTLED so it does not whirl straight through the top. This one    function scores both the imagined rollouts (the planner) and the realized step."""    theta = wrap_angle(float(e.data.qpos[HINGE]))       # angle from upright; 0 up, +-pi down    x = float(e.data.qpos[SLIDER])                       # cart offset from rail center    return ((1.0 - np.cos(theta))                        # 0 upright, 2 hanging down (the dominant term)            + 0.1 * x * x                                # keep the cart near center            + 0.01 * float(e.data.qvel[HINGE]) ** 2      # settle the pole (don't spin through the top)            + 0.005 * float(e.data.qvel[SLIDER]) ** 2    # settle the cart            + 0.001 * float(action) ** 2)                # mild effort penalty  def upright_cos(e: CartpoleEnv) -> float:    """cos(angle-from-upright): +1 exactly up, -1 hanging down. The honest 'is it up?' readout."""    return float(np.cos(wrap_angle(float(e.data.qpos[HINGE]))))

The cost function is the task. There is no learned objective anywhere: you state what "good" means — pole upright, cart centered, motion settled — and the search does the rest. Read state_cost and notice that this single, hand-written function scores both the imagined rollouts and the realized step. Writing the objective down instead of learning it is the whole trade this chapter makes.

Same loop, one idea differs: CEM vs MPPI

Here is the entire planner. Sample, roll, score — then update the sampling mean toward the good plans. The two update rules in the course's "same file, one idea differs" tradition are about fifteen lines apart:

# The one function MPC turns on. Given the current true state and a warm-started# mean plan, it samples N action sequences, rolls each through the MODEL, scores# them, and refines the sampling distribution `iters` times. CEM and MPPI differ# ONLY in how the scores update the mean (the two branches below) — everything# around them is identical. That is the "same file, one idea differs" of the course.  def rollout_cost(q0: np.ndarray, v0: np.ndarray, seq: np.ndarray) -> float:    """Total cost of one candidate action sequence, IMAGINED through the model from    (q0, v0). We REWIND the model to the shared start, then step the whole horizon,    accumulating state_cost — this is the single call MPC makes N x H times a step."""    set_state(sim, q0, v0)    total = 0.0    for action in seq:        step_model(sim, action)        total += state_cost(sim, float(action))    return total  def plan(q0: np.ndarray, v0: np.ndarray, mean: np.ndarray) -> tuple[np.ndarray, np.ndarray]:    """Return (refined mean plan, the sampled first-actions) for ONE control step.    `mean` is warm-started from last step's plan, shifted one tick — most of the    thinking carries over, so a couple of iterations is enough."""    mean = mean.copy()    std = np.full(args.horizon, args.init_std)    first_samples = None    for _ in range(args.iters):        # sample N sequences around the current mean, clipped to the actuator range        noise = rng.standard_normal((args.samples, args.horizon))        samples = np.clip(mean[None, :] + std[None, :] * noise, -1.0, 1.0)  # (N, H)        costs = np.array([rollout_cost(q0, v0, seq) for seq in samples])    # (N,) score each plan        if first_samples is None:            first_samples = samples[:, 0].copy()  # kept only to visualize the fan-out        if args.method == "cem":            # CEM: keep the ELITE fraction (the lowest-cost plans) and refit the            # Gaussian to them. The mean marches toward the elites; the std shrinks            # as they agree, so the search focuses in on the good region.            n_elite = max(1, int(args.elite_frac * args.samples))            elite = samples[np.argsort(costs)[:n_elite]]            mean = elite.mean(axis=0)            std = elite.std(axis=0) + 0.05  # floor keeps it from collapsing to zero        else:  # mppi            # MPPI: no hard cutoff. Weight EVERY sample by exp(-cost / temperature)            # (a softmax over negative cost) and take the weighted average. Good            # plans dominate smoothly; nothing is thrown away.            weights = np.exp(-(costs - costs.min()) / args.temperature)            weights = weights / weights.sum()            mean = (weights[:, None] * samples).sum(axis=0)    return mean, first_samples

• CEM (the cross-entropy method) draws a hard line: sort by cost, keep the elite fraction — the few lowest-cost plans — and refit the Gaussian to them. The mean marches toward the elites; the std shrinks as they agree, so the search focuses in.
• MPPI (model-predictive path integral) refuses to throw anything away. Every sample gets a weight exp(-cost / temperature) — a softmax over negative cost — and the new mean is the weighted average of all of them. Good plans dominate smoothly; the temperature sets how sharply.

They are two answers to one question — how much should a plan's score move the mean? — and at the limit they meet: CEM with a single elite keeps the one best plan, and MPPI as the temperature goes to zero collapses to that same plan. Exercise 3 makes you fill both updates in and prove they agree there.

Run it

python curriculum/phase3_advanced/ch3.9_mpc/mpc.py --seed 0
python .../mpc.py --seed 0 --method mppi

The reference numbers are from --device cpu, the configuration this book can promise is bitwise-deterministic under --seed: numpy's sampling and CPU mj_step are both deterministic, so the same seed run twice matches byte for byte. Every run also plays a no-plan baseline — uniform-random actions from the same start — so the payoff is right there in the output:

                        mean cost  upright frac
MPPI                        1.210          1.00
random (no plan)            1.988          0.00

upright_frac is the fraction of the settle window (the last quarter of the episode) the pole spends within about 25° of straight up. MPC swings up and holds — 1.00 — while random flails at 0.00, and the smooth cost sits far below the baseline. You solved an underactuated control problem with zero learning: no policy, no training, just a model and a search. And it reproduces — seeds 0, 1, 2 all swing up, for both CEM and MPPI. Open the recording to watch the upright-cos curve climb from −1 to +1:

rerun outputs/ch3.9-mpc/mpc.rrd

Break it: a plan that can't see far enough

The claim is a mechanism claim, not a magic one, so measure its edge rather than trust it. --break horizon drops the look-ahead from 25 steps to 3 — same model, same sampler, same cost, just a myopic plan:

python .../mpc.py --seed 0 --break horizon

The pole never comes up. upright_frac collapses to 0.00, seed after seed. A three-step plan cannot see that letting the pole fall further now is what buys the swing later, so it does the locally sensible thing forever and gets nowhere — exactly the greedy failure swing-up is designed to expose. --break samples (3 samples instead of 64) fails the same way for the other reason: too few tries to stumble on an energy-pumping sequence at all. Look-ahead and search width are not free knobs you can shrink; they are what make planning work.

The honest ceiling — and the trade against learning

MPC looks like a free lunch here, and it is not. It cost you two things.

It needs a model. In this chapter the model is the world — a perfect copy of the simulator — which is the best case that exists, and the reason the swing comes up so cleanly. Chapter 3.6 already showed you the other side: run a policy in an engine that only approximates the true dynamics and it degrades, because it is optimizing an imagined trajectory the real world will not follow. MPC has exactly that exposure. A plan is only as good as the model it is dreamed through; a wrong model plans confidently toward the wrong place.

It needs compute at every step. MPC does no work up front and then re-solves an optimization every single tick — N × H model rollouts, times a few refinement iterations, before it can even move. That is the mirror image of a learned policy, which pays a huge one-time training cost and then acts in microseconds with no model at all. This is the real axis this chapter adds to the course: learn once and react fast, or carry a model and plan every step. Neither dominates. The frontier systems you'll meet later blend them — a learned model to plan through, a learned policy to warm-start the search — but you can't understand those hybrids until you've felt both pure ends, and this is the end the physics-engine arc was quietly building toward all along.

Read the real thing

google-deepmind/mujoco_mpc (MJPC) is this file grown up. It is an interactive controller that plans through mj_step — the same engine you plan through here — fast enough to drive humanoids and manipulators in real time. Open its mjpc/planners/: the cross_entropy planner is our CEM, the sampling (predictive sampling) and MPPI-style planners are our weighted-average update, and each task's cost is defined exactly the way state_cost is — a residual you write down, not a reward you learn. The one thing MJPC has that we don't is throughput: it rolls its candidate plans through the model across many CPU threads at once, which is the honest answer to this chapter's ceiling — MPC's price is compute-per-step, and you pay it with cores. Read our whole planner first; then read theirs and see the same idea running a hundred times faster.

Exercises

Three, in exercises/. The first (ex1_predict_planning) is the misconception, head on: predict whether a search through a model can control the cartpole with no learning at all, then run it and watch it swing up. The second (ex2_predict_break) hands you the --break horizon failure to generate yourself — predict what a myopic plan does, then measure the pole refusing to rise. The third (ex3_completion_update) blanks out both the CEM elite-refit and the MPPI weighted-average and asks you to fill them in and prove they agree at the limit — the fifteen-line diff that is the entire difference between the two methods.

What's next

You now hold both pure ends of control: a policy that learns once and reacts, and a planner that carries a model and searches every step. The physics-engine arc closes here — you opened mj_step, then used the thing you understood to plan through it. What's missing is a model you didn't get for free: real robots don't ship with a perfect simulator of themselves. The world-model chapters (3.1–3.2) learned dynamics from data and then acted in imagination; put that together with what you just built and you get the modern recipe — learn a model, then plan through it — which is where the frontier of this field actually lives.