What a VLA eats, and why this chapter is only the plate
A vision-language-action policy takes three things at once — an instruction ("push the T onto the target"), an image (what the camera sees now), and a state (the numbers the robot reports) — and predicts an action. Chapter 1.8 builds and trains that policy. This chapter builds the data it trains on, and nothing more.
That boundary is deliberate. The machinery that turns raw demos into
(instruction_tokens, image_features, state) -> action examples is where most of the
real-world VLA effort — and most of the silent failures — actually live. Get the data
wrong and no policy can recover; get it right and the policy in 1.8 is almost boring.
So this file has no training loop anywhere. It runs top to bottom and writes a dataset.
Open vla_data.py. Five regions: setup, tasks (assemble two tasks into one
pile), language (templates plus a from-scratch tokenizer), vision (a frozen
tiny CNN), and pipeline (stitch it all into examples, then probe it for leakage).
Multi-task: two embodiments, one pile
# Two tasks, two embodiments, ONE pile — that is the multi-task point. A real VLA# (RT-X, OpenVLA) mixes dozens of datasets; the idea is already visible with two.# Each task brings its OWN env, its OWN scripted expert (the SAME experts# gen_demos.py writes the LeRobot v3 datasets with — we replay them in-process here# so we can render frames without a video-decode dependency), its action# dimensionality, and a set of paraphrase instructions.TASKS = [ { "name": "pusht", "env": PushTEnv, "expert": PushtExpert, "act_dim": PushTEnv.ACT_DIM, "templates": [ "push the t block onto the target", "slide the tee onto the goal", "move the t shape to the target pose", "push the block until it covers the target", ], }, { "name": "aloha", "env": AlohaCubeEnv, "expert": AlohaExpert, "act_dim": AlohaCubeEnv.ACT_DIM, "templates": [ "transfer the cube to the other arm", "hand the cube from the right arm to the left arm", "pick up the cube and pass it to the left gripper", "carry the cube across and place it on the target", ], },] def collect_task(task: dict, episodes: int, seed: int, stride: int): """Replay the scripted expert for `episodes` episodes; return per-frame states, padded actions, 96x96 frames, and the episode index of each frame. Deterministic given seed: episode e uses env+expert seed (seed + e), exactly the reproducibility contract gen_demos.py relies on. Actions live in the task's native dimensionality (2 for pusht, 6 for aloha); we zero-pad to ACT_DIM_MAX so both tasks share one action tensor — the honest cost of mixing embodiments. """ env = task["env"]() states, actions, frames, ep_index = [], [], [], [] for e in range(episodes): obs = env.reset(seed + e) expert = task["expert"](noise=0.0, seed=seed + e) step, done = 0, False while not done: action = expert.action(env) if step % stride == 0: # subsample: consecutive frames are near-duplicates padded = np.zeros(ACT_DIM_MAX, dtype=np.float32) padded[: task["act_dim"]] = action[: task["act_dim"]] states.append(obs.astype(np.float32)) actions.append(padded) frames.append(env.render_frame(IMG_HW, IMG_HW)) ep_index.append(e) obs, _, done, _ = env.step(action) step += 1 return (np.asarray(states, np.float32), np.asarray(actions, np.float32), np.asarray(frames, np.uint8), np.asarray(ep_index, np.int64))We use both environments you already have: the PushT pusher (a 2-D velocity action)
and the ALOHA cube handoff (a 6-D bimanual action). Each brings its own scripted
expert — the same experts gen_demos.py uses to write the LeRobot datasets — and we
replay them in-process so we can render each frame without a video-decode dependency.
Two honest frictions show up the moment you mix tasks. First, the action spaces
differ: 2 dims versus 6. We zero-pad PushT's action up to a shared width of 6 and
carry an action_mask marking which dims are real — the same move a real
multi-embodiment dataset makes. Second, the episode lengths differ: PushT ends
when the block lands on the target; the handoff is a longer pick → carry → handoff →
place. So at 12 episodes each you do not get equal frame counts — 309 PushT
frames and 170 ALOHA, 479 total. Every example carries a task_id, so the two
tasks share a tensor but are never confused.
Language: templates and a tokenizer you can read
# A VLA reads words, and words must become integers. We build the WHOLE tokenizer# from scratch: a fixed WORD-LEVEL vocab (no BPE, no HF tokenizers), four special# ids, pad/truncate to MAX_TOKENS. It is tiny (~40 words) on purpose — a real VLA# uses a 30k+ subword vocab from a pretrained language model, which is one of the# things chapter 1.8 reaches for. Here the point is the MECHANISM, laid bare._SPECIALS = ["<pad>", "<unk>", "<bos>", "<eos>"] # ids 0..3, by construction# --break leak appends one of these compass words per FRAME. They must be IN vocab:# an out-of-vocab word would collapse to <unk> and leak nothing._DIRECTIONS = ["east", "northeast", "north", "northwest", "west", "southwest", "south", "southeast"] class Tokenizer: """Word-level tokenizer over a FIXED vocab. Deterministic: the vocab is the sorted set of corpus words, so the same corpus always yields the same ids.""" def __init__(self, corpus: list[str]) -> None: words = sorted({w for text in corpus for w in text.split()}) self.itos = _SPECIALS + words # index -> string self.stoi = {w: i for i, w in enumerate(self.itos)} @property def vocab_size(self) -> int: return len(self.itos) def encode(self, text: str) -> np.ndarray: # [BOS] word ids... [EOS], unknown words -> <unk>, then pad/truncate. ids = [self.stoi["<bos>"]] ids += [self.stoi.get(w, self.stoi["<unk>"]) for w in text.split()] ids.append(self.stoi["<eos>"]) ids = ids[:MAX_TOKENS] + [self.stoi["<pad>"]] * (MAX_TOKENS - len(ids)) return np.asarray(ids[:MAX_TOKENS], dtype=np.int64) def direction_word(action_xy: np.ndarray) -> str: """8-way compass word for a 2D action vector; the leak template appends it.""" sector = int(np.round(np.arctan2(action_xy[1], action_xy[0]) / (np.pi / 4))) % 8 return _DIRECTIONS[sector]Each task gets a small set of paraphrase templates ("push the t block onto the target", "slide the tee onto the goal", …). We pick one per episode, deterministically from the seed, and reuse its wording across that episode's frames — real demos are annotated once, not per timestep. The paraphrases earn their place: they teach a policy that the task is the invariant, not one exact string, so it does not overfit to a single phrasing.
Words have to become integers, so we build the whole tokenizer from scratch — no
transformers, no BPE. It is word-level: the vocab is the sorted set of every word
that can appear (template words plus a few extras), prefixed with four special ids
(<pad>, <unk>, <bos>, <eos>). encode wraps a sentence as
[BOS] words… [EOS], maps any unknown word to <unk>, and pads or truncates to a
fixed 16 tokens. The resulting vocab is 46 ids — four specials and 42 words — and
every content token is in-vocab (oov_rate 0.0) because the vocab is closed by
construction.
Be clear on the scale: a real VLA carries a 30,000+ subword tokenizer inherited from a pretrained language model. Ours is 46 ids. What is identical is the mechanism — string in, fixed-length id array out, specials and OOV handled — and that mechanism is all chapter 1.8 needs from this side.
Vision: a frozen tiny encoder (and what it is not)
class FrozenVisionEncoder(nn.Module): """A from-scratch conv stack: (B, 96, 96, 3) uint8 -> (B, feature_dim). It is RANDOM-INIT and FROZEN — never trained here or in 1.8. Deliberately bare: Conv/ReLU and a global average pool, no BatchNorm (its batch-dependent stats would break determinism) and no attention. The honest claim: a frozen random CNN is a fixed nonlinear projection of the pixels. It still preserves coarse spatial layout — roughly WHERE the block, cube, and arms are — so its output is a usable visual feature, and already more compact and policy-friendly than raw pixels. What it does NOT give you is what a PRETRAINED backbone (DINOv2 / SigLIP, as in OpenVLA and SmolVLA) does: features aligned to objects and to language, transferable across scenes. Closing that gap is the reason chapter 1.8 bolts on a real backbone. """ def __init__(self, width: int, feature_dim: int) -> None: super().__init__() self.stem = nn.Sequential( nn.Conv2d(3, width, 3, stride=2, padding=1), nn.ReLU(), # 96 -> 48 nn.Conv2d(width, 2 * width, 3, stride=2, padding=1), nn.ReLU(), # 48 -> 24 nn.Conv2d(2 * width, 4 * width, 3, stride=2, padding=1), nn.ReLU(), # 24 -> 12 nn.AdaptiveAvgPool2d(1), # -> (4*width, 1, 1) ) self.head = nn.Linear(4 * width, feature_dim) for p in self.parameters(): p.requires_grad_(False) # FROZEN: the perception is fixed, not learned self.eval() @torch.no_grad() def forward(self, images_uint8: torch.Tensor) -> torch.Tensor: # (B, 96, 96, 3) uint8 -> (B, 3, 96, 96) float in [-1, 1] -> (B, feature_dim) x = images_uint8.to(torch.float32).permute(0, 3, 1, 2) / 127.5 - 1.0 return self.head(self.stem(x).flatten(1)) def encode_frames(encoder: FrozenVisionEncoder, frames: np.ndarray, device: torch.device, batch: int = 256) -> np.ndarray: feats = [] for i in range(0, len(frames), batch): chunk = torch.from_numpy(frames[i:i + batch]).to(device) feats.append(encoder(chunk).cpu().numpy()) return np.concatenate(feats).astype(np.float32)A policy cannot condition on a raw 96×96×3 image cheaply, so we hand it a compact
feature vector per frame. FrozenVisionEncoder is a from-scratch conv stack —
three Conv2d/ReLU stages that halve the resolution each time, a global average
pool, and a linear head to feature_dim (64 by default). It is random-init and
frozen: its weights are never trained, here or in 1.8.
This is the chapter's most important honesty. A frozen random CNN is not perception in any meaningful sense — it is a fixed nonlinear projection of the pixels. But it is not useless either: convolution and pooling preserve coarse spatial layout, so the 64 numbers still encode roughly where the block, cube, and arms are, which is already a more policy-friendly signal than the raw pixel grid. That is the entire pedagogical point — you can build a working VLA data pipeline whose "perception" is a random projection, and watch it flow through to features.
What a frozen random encoder does not give you is what a pretrained backbone (DINOv2, SigLIP — the backbones OpenVLA and SmolVLA actually use) does: features aligned to objects and to language, transferable across scenes and tasks. Closing that gap — swapping this stand-in for a real, trained backbone — is a large part of why chapter 1.8 exists. When you read this chapter's features, read them as "the signal a policy will condition on," never as "good vision."
The pipeline, and the leak
# 1) MULTI-TASK: gather both piles and stack them into one dataset.per_task = [collect_task(t, args.episodes_per_task, args.seed, args.frame_stride) for t in TASKS]states = np.concatenate([p[0] for p in per_task])actions = np.concatenate([p[1] for p in per_task])frames = np.concatenate([p[2] for p in per_task])task_id = np.concatenate([np.full(len(p[0]), i, np.int64) for i, p in enumerate(per_task)])ep_id = np.concatenate([p[3] for p in per_task])num_examples = len(states) # 2) INSTRUCTION TEMPLATING + TOKENIZING. The vocab is fixed and known up front —# every template word, the compass words, and "moving" (used by the leak template).corpus = [t for task in TASKS for t in task["templates"]] + _DIRECTIONS + ["moving"]tokenizer = Tokenizer(corpus)# Pick one paraphrase per EPISODE, deterministically from the seed, and reuse its# wording for every frame of that episode — real demos are annotated once per# episode. --break leak overrides this per FRAME (it names the move; see below).instructions = []for i in range(num_examples): task = TASKS[task_id[i]] variant = (args.seed + task_id[i] * 7919 + ep_id[i]) % len(task["templates"]) text = task["templates"][variant] if args.break_mode == "leak": # THE MISCONCEPTION: the annotator "helpfully" writes which way to move into # the instruction. Now the WORDS carry the action, so a policy can read the # answer off language and never look at the pixels. Measured by the probe below. text = f"{text} moving {direction_word(actions[i, :2])}" instructions.append(text)tokens = np.stack([tokenizer.encode(t) for t in instructions])specials = {tokenizer.stoi[s] for s in ("<pad>", "<bos>", "<eos>")}content = tokens[~np.isin(tokens, list(specials))]oov_rate = float((content == tokenizer.stoi["<unk>"]).mean()) if content.size else 0.0 # 3) FROZEN VISION ENCODER: pixels -> features the policy will condition on.encoder = FrozenVisionEncoder(args.conv_width, args.feature_dim).to(device)image_features = encode_frames(encoder, frames, device) # 4) LEAKAGE PROBE — the Break-It's measured signature. How much of the action can a# LINEAR model read out of the instruction tokens ALONE, per task? Build a# bag-of-words instruction feature, least-squares onto the action, report R^2 of the# fit. Normal templates name the TASK, not the move, so within a task the words# barely vary and the probe explains ~none of the per-frame action (R^2 ~ 0). Leak# templates name the direction every frame, so R^2 jumps: the answer is decodable# from language, and a policy trained on it would learn to ignore vision entirely.def language_action_r2(rows: np.ndarray, act_dim: int) -> float: bag = np.zeros((len(rows), tokenizer.vocab_size), np.float64) for r, i in enumerate(rows): for tok in tokens[i]: if tok != tokenizer.stoi["<pad>"]: bag[r, tok] += 1.0 X = np.concatenate([bag, np.ones((len(rows), 1))], axis=1) # + intercept column Y = actions[rows, :act_dim].astype(np.float64) W, *_ = np.linalg.lstsq(X, Y, rcond=None) ss_res = float(((Y - X @ W) ** 2).sum()) ss_tot = float(((Y - Y.mean(0)) ** 2).sum()) return 1.0 - ss_res / ss_tot if ss_tot > 1e-9 else 0.0 r2_by_task = [language_action_r2(np.where(task_id == i)[0], t["act_dim"]) for i, t in enumerate(TASKS)]action_from_language_r2 = float(np.mean(r2_by_task)) # 5) EMIT the unified dataset. A documented .npz (a real stack streams LeRobot v3# from the Hub); the action_mask marks which action dims are valid per embodiment.action_mask = np.zeros((num_examples, ACT_DIM_MAX), np.float32)for i, t in enumerate(TASKS): action_mask[task_id == i, : t["act_dim"]] = 1.0np.savez(args.out / "vla_dataset.npz", instruction_tokens=tokens, image_features=image_features, state=states, action=actions, action_mask=action_mask, task_id=task_id)manifest = { "schema": { "instruction_tokens": ["num_examples", MAX_TOKENS, "int64 token ids"], "image_features": ["num_examples", args.feature_dim, "float32 (frozen CNN)"], "state": ["num_examples", STATE_DIM, "float32"], "action": ["num_examples", ACT_DIM_MAX, "float32 (zero-padded)"], "action_mask": ["num_examples", ACT_DIM_MAX, "1.0 where the action dim is real"], "task_id": ["num_examples", "int64 index into tasks[]"], }, "tasks": [{"id": i, "name": t["name"], "act_dim": t["act_dim"], "templates": t["templates"], "count": int((task_id == i).sum())} for i, t in enumerate(TASKS)], "vocab": tokenizer.itos, "max_tokens": MAX_TOKENS, "feature_dim": args.feature_dim, "frozen_vision_encoder": "from-scratch conv stack, random-init, never trained", "break_mode": args.break_mode or "none",}(args.out / "manifest.json").write_text(json.dumps(manifest, indent=2) + "\n") # rerun: the multi-task MIX (how many examples per task) and a stream of sampled# examples — the frame, its frozen features, state, action, and task id.if args.rerun: for i, t in enumerate(TASKS): rr.log(f"data/task_mix/{t['name']}", rr.Scalars([float((task_id == i).sum())]), static=True) sample = np.unique(np.linspace(0, num_examples - 1, min(args.log_examples, num_examples)).astype(int)) for out_step, i in enumerate(sample): rr.set_time("example", sequence=out_step) rr.log("data/image", rr.Image(frames[i])) rr.log("data/image_features", rr.BarChart(image_features[i].astype(np.float64))) rr.log("data/state", rr.Scalars(states[i].astype(np.float64))) rr.log("data/action", rr.Scalars(actions[i].astype(np.float64))) rr.log("data/task_id", rr.Scalars([float(task_id[i])])) metrics = { "action_from_language_r2": round(action_from_language_r2, 6), # the leak signature "break_mode": args.break_mode or "none", "feature_dim": int(args.feature_dim), "image_feature_mean": round(float(image_features.mean()), 6), # frozen-encoder fingerprint "image_feature_std": round(float(image_features.std()), 6), "max_tokens": int(MAX_TOKENS), "num_examples": int(num_examples), "num_examples_aloha": int((task_id == 1).sum()), "num_examples_pusht": int((task_id == 0).sum()), "oov_rate": round(oov_rate, 6), "r2_aloha": round(r2_by_task[1], 6), "r2_pusht": round(r2_by_task[0], 6), "seed": args.seed, "smoke": bool(args.smoke), "vocab_size": int(tokenizer.vocab_size),}(args.out / "metrics.json").write_text(json.dumps(metrics, indent=2, sort_keys=True) + "\n")print(f"dataset: {num_examples} examples " f"({metrics['num_examples_pusht']} pusht / {metrics['num_examples_aloha']} aloha), " f"vocab {tokenizer.vocab_size}, feature_dim {args.feature_dim}")print(f"language->action R^2: {action_from_language_r2:.3f} " f"(pusht {r2_by_task[0]:.3f}, aloha {r2_by_task[1]:.3f}), break={args.break_mode or 'none'}")print(f"wrote {args.out / 'vla_dataset.npz'} + manifest.json + metrics.json")if args.rerun: print(f"recording: {args.out / 'vla_data.rrd'} — open it with: rerun {args.out / 'vla_data.rrd'}")The pipeline stacks both tasks, tokenizes every instruction, encodes every frame
through the frozen CNN, and writes one dataset: a documented .npz (a real stack would
stream LeRobot v3 from the Hub) plus a manifest.json recording the schema, the tasks,
and the full vocab. The result is 479 examples, feature_dim 64, vocab_size
46.
Then it runs one diagnostic that is the heart of the chapter — a leakage probe. The question: how much of the action can a linear model read out of the instruction words alone? We build a bag-of-words vector from each instruction's tokens, least-squares fit it onto the action per task, and report the R² of that fit. If words tell you nothing about the moment-to-moment action, R² ≈ 0; if the words encode the action, R² approaches 1.
With the clean task-level templates, R² = 0.006 — the instruction names the task, and within a task the words barely move, so language explains essentially none of the per-frame action. That is what you want: the action has to come from the image and state, exactly where 1.8's policy will be forced to look.
Break it
--break leak — "a more descriptive instruction can't hurt." This appends the
current move direction to every frame's instruction ("… moving northeast"). It reads
like a better annotation. It is a trap. The probe R² jumps from 0.006 to
0.71 — the action is now linearly decodable from the words alone. A policy trained
on this data would minimize its loss by reading the answer off the instruction and
ignoring its camera entirely; drop the image at test time and it would look fine on
this distribution and fail the instant the instruction stops narrating the move.
The gap is seed-robust — the leak probe reads 0.71 across seeds 0, 1, and 2 — and the lesson is general: anything an annotator can see that correlates with the action can leak into the instruction and hollow out the vision pathway. The loss curve in 1.8 would never warn you; the model would look like it is learning. The only place to catch it is here, in the data.
What we cut
This is a real VLA data pipeline in shape, but every heavy part is a stand-in:
- The vision encoder is frozen and random, not a pretrained DINOv2/SigLIP. No object or language alignment — that is 1.8's upgrade and the single biggest gap.
- The tokenizer is a 46-id fixed vocab, not a 30k+ subword tokenizer from a pretrained LM.
- Two tasks, not dozens. RT-X / OpenVLA mix many datasets across many robots; the multi-task mechanism (pad + mask + task_id) is already the real one.
- We emit inputs, we don't condition a policy on them. Wiring
(tokens, features, state)into an action head is chapter 1.8.
None of these silently fakes a number — each is a whole capability deferred so the data pipeline stays readable end to end. The "read the real thing" segment walks a production VLA data stack so you can see exactly what these paragraphs left out.
Run it
python curriculum/phase1_imitation/ch1.7_vla_data/vla_data.py --seed 0 --device cpu
| cpu-laptop | expected wall-clock on cpu-laptop: ~0.03 min (measured) | measured |
| mps | wall-clock on mps: not yet measured | pending |
| t4 | expected wall-clock on t4: ~0.15 min (measured) | measured |
| 4090 | wall-clock on 4090: not yet measured | pending |
| value | |
|---|---|
| examples (pusht / aloha) | 479 (309 / 170) |
| tokenizer vocab | 46 ids |
| frozen feature dim | 64 |
| action_from_language_r2 (clean) | 0.006 |
action_from_language_r2 (--break leak) |
0.71 |
rerun outputs/ch1.7-vla-data/vla_data.rrd
Read the real thing
The paired reading is huggingface/lerobot, pinned here at tag v0.4.4 — the
library whose data format our .npz is a stripped-down stand-in for, and whose
SmolVLA policy is a real version of the pipeline you just built. (At this tag the
package moved under src/, so every path below starts src/lerobot/.) Read it in four
passes, one per stand-in.
The multi-task mix and the language annotation. Our tasks region gathers two
TASKS in-process, and our language region picks one paraphrase per episode from a
hand-written templates list. Production does not template — the instruction is
data. In src/lerobot/datasets/lerobot_dataset.py, MultiLeRobotDataset
concatenates many LeRobotDatasets into one pile (our two-task np.concatenate,
generalized to dozens of datasets and embodiments), and each frame's instruction is a
real string an annotator wrote — stored per episode and looked up in __getitem__
(item["task"] = self.meta.tasks.iloc[task_idx].name). What they add: the language is
collected, not generated from a fixed vocabulary, so it carries the phrasing
diversity our four paraphrases only gesture at.
The tokenizer. Our Tokenizer is a 46-id closed word vocab. SmolVLA's
TokenizerProcessorStep in src/lerobot/policies/smolvla/processor_smolvla.py loads a
pretrained subword tokenizer by name (tokenizer_name=config.vlm_model_name) and
pads to tokenizer_max_length (48). What they add: a 30k+ subword vocabulary inherited
from the language model, so an unseen word decomposes into known subwords instead of
collapsing to our single <unk>.
The vision backbone. Our FrozenVisionEncoder is a random-init conv stack. SmolVLA
loads a whole pretrained VLM in src/lerobot/policies/smolvla/smolvlm_with_expert.py —
AutoModelForImageTextToText.from_pretrained("HuggingFaceTB/SmolVLM2-500M-Video-Instruct") —
and encodes frames through its vision_model (a SigLIP encoder) via embed_image. What
they add: features aligned to objects and to language, transferable across scenes —
exactly the "good vision" our frozen projection is honest about not being.
What conditions the action head. Here we only emit (tokens, features, state);
nothing consumes them. In src/lerobot/policies/smolvla/modeling_smolvla.py,
embed_prefix fuses image, language, and state embeddings into a prefix; embed_suffix
embeds the noisy action and the flow-matching timestep; and the action expert
(smolvlm_with_expert.py, forward_cross_attn_layer) attends from action tokens to
that prefix. That cross-attention is the wire chapter 1.8 solders — the flow-matching
head from chapter 1.5, conditioned at last on all three inputs.
Read next, in order: MultiLeRobotDataset in lerobot_dataset.py, then
processor_smolvla.py for the tokenizer step, then modeling_smolvla.py's
embed_prefix / embed_suffix — the three lines of your pipeline, grown up.
Exercises
Four, in exercises/. Two ask you to commit to a prediction before the run answers —
the leakage probe (clean vs --break leak), and the multi-task mix (do both tasks
appear, and in what proportion). One is a bug-hunt in the tokenizer (the leading <bos>
is dropped, so every id is shifted a slot). One has you implement the bag-of-words
count the leakage probe is built on.
What's next
You now have the three inputs a VLA conditions on — tokenized instructions, frozen image features, and state — stacked into one multi-task dataset, plus a probe that tells you whether your language is honestly describing the task or secretly dictating the action. Chapter 1.8 wires these inputs into a tiny VLA: it swaps the frozen random CNN for a real backbone, the 46-id vocab for a real tokenizer, and puts the flow-matching action head from chapter 1.5 on top — conditioned, at last, on all three.