Two ways to say an action
By now you have a policy that can emit an action chunk — H timesteps of act_dim numbers,
the coordinated little burst of motion ch1.3 argued for. The question this chapter asks is
narrow and mechanical: how do you get that chunk out of a model?
You have already built one answer. ch1.5's flow head treats the chunk as a point in a
continuous space and integrates it out of noise in a few Euler steps — the whole chunk at
once, no vocabulary anywhere. This chapter builds the other answer, the one pi0-FAST and
OpenVLA use: turn the chunk into a short string of discrete tokens and let a language model
emit them one at a time, so a robot's motion rides the exact same cross-entropy machinery as
text. To do that you need a codec — something that maps a real-valued H × act_dim chunk to
integers and back. That codec is the whole chapter. No policy is trained, no environment is
stepped; like ch1.7, the subject is the data pipeline itself. Open fast.py. Eight regions:
setup, dct, quantize, bpe, data, codec, measure, report. It is
pure numpy — no torch, no model.
The naive tokenizer, and why it hurts
The obvious way to tokenize an action chunk is per-step-per-dim binning (this is what RT-2
does): chop each scalar's range into bins and emit one token per number. A chunk is then
H × act_dim tokens, always — for H = 24 and a 6-DoF arm that is 144 tokens for a single
burst of motion, and the language model has to generate all of them, autoregressively, every
control cycle. Tokens are the currency of an LM's compute; per-step binning spends them
recklessly. FAST's whole claim is that you can say the same motion in far fewer.
Step 1 — the DCT: change the basis
# The DCT-II as a plain matrix — no scipy, no FFT trick, just the definition. Row k of D is# the k-th cosine basis vector (k=0 = DC average, higher k = faster wiggles). ORTHONORMAL# scaling makes D @ D.T = I, so the inverse is just D.T and Parseval preserves L2 energy.def dct_matrix(n: int) -> np.ndarray: """(n, n) orthonormal DCT-II matrix. coeffs = D @ signal; signal = D.T @ coeffs.""" k = np.arange(n)[:, None] # frequency index (rows) t = np.arange(n)[None, :] # time index (cols) d = np.cos(np.pi * (2 * t + 1) * k / (2 * n)) * np.sqrt(2.0 / n) d[0] *= 1.0 / np.sqrt(2.0) # DC row scaled so the basis is orthonormal return d DCT = dct_matrix(H) def dct(chunk: np.ndarray) -> np.ndarray: """(H, act_dim) trajectory -> coefficients, each dim's time series independently. Row 0 is the average pose; rows toward H-1 are the fast wiggles.""" return DCT @ chunk def idct(coeffs: np.ndarray) -> np.ndarray: """Inverse: coefficients -> trajectory. Just the transpose (D is orthonormal) — no scaling to undo, no schedule (contrast ch1.4's DDPM).""" return DCT.T @ coeffsA smooth trajectory wastes the time domain. Neighboring samples are nearly equal, so most of
the H numbers per dimension are redundant. The discrete cosine transform re-expresses the
same chunk in a basis of cosine waves — a slow one (the average level), a slightly faster one,
and so on up to the fastest wiggle. We build it as a plain H × H matrix, one cosine per row,
with no scipy and no FFT trick: coeffs = D @ chunk. The one design choice that matters is
the orthonormal scaling, which makes D @ Dᵀ = I. That buys two things we lean on hard:
the inverse transform is just the transpose (idct is one matmul, no schedule to undo —
contrast ch1.4's DDPM reverse loop), and, by Parseval's theorem, the transform preserves L2
energy exactly. Hold onto that second fact.
Step 2 — quantize: where the zeros come from
# Turn real coefficients into small integers. round(x / step) is the whole quantizer.# Two things make the grid mostly ZEROS (what BPE compresses): a smooth trajectory's# high-frequency coeffs are ~0 and round to 0, plus an optional low-pass keeping only the# K lowest frequencies per dim. In --break time_domain we quantize raw samples — no zeros.def quantize(coeffs: np.ndarray, keep: int, q_scale: float, q_max: int) -> np.ndarray: q = np.clip(np.round(coeffs / q_scale), -q_max, q_max).astype(np.int64) if keep < len(q): q[keep:] = 0 # drop the high-frequency rows (a low-pass; only meaningful in the DCT domain) return q def dequantize(q: np.ndarray, q_scale: float) -> np.ndarray: return q.astype(np.float64) * q_scale def flatten_grid(q: np.ndarray) -> list[int]: """(H, act_dim) grid -> 1-D token list, DIM-MAJOR: each dim's H coeffs run low->high frequency and land contiguous, so every dim ends in a long zero-run — what BPE eats.""" return q.T.reshape(-1).tolist()round(x / step) is the entire quantizer. Applied to the DCT coefficients, something useful
happens: a smooth motion has almost no energy in the high-frequency cosines, so those
coefficients are already near zero and round to exactly 0. The coefficient grid comes out
mostly zeros. That is the compressible structure the next step feeds on — and it is created by
changing basis, not by the rounding itself.
Here is the sharp version of the Parseval point. Because D is orthonormal, quantizing the
coefficients injects the same error energy as quantizing the raw samples would. So the DCT
does not buy you a better reconstruction — it buys you the same reconstruction with the error
repackaged into a form you can throw a compressor at.
Step 3 — BPE, on actions instead of text
# Byte-pair encoding, the minbpe move, on ACTION tokens. Repeatedly find the most frequent# adjacent PAIR across the corpus and merge it into one new token id. Runs of zeros collapse# first (0,0)->A, (A,A)->B, ... — a chunk's 20-zero tail becomes one or two tokens. BPE is# LOSSLESS entropy coding on top of the lossy DCT+quantize: it changes the token COUNT, never# the reconstruction. Ties broken by pair value -> deterministic.def merge_seq(seq: list[int], pair: tuple[int, int], new_id: int) -> list[int]: out, i = [], 0 while i < len(seq): if i < len(seq) - 1 and (seq[i], seq[i + 1]) == pair: out.append(new_id) i += 2 else: out.append(seq[i]) i += 1 return out def train_bpe(seqs: list[list[int]], num_merges: int) -> dict[tuple[int, int], int]: """Learn merges. Returns an ordered dict (a,b)->new_id; order IS the apply order (encode replays it). Base ids are already 0..V-1 (see remap below).""" seqs = [list(s) for s in seqs] next_id = max((max(s) for s in seqs if s), default=-1) + 1 merges: dict[tuple[int, int], int] = {} for _ in range(num_merges): counts: dict[tuple[int, int], int] = {} for s in seqs: for pair in zip(s, s[1:]): counts[pair] = counts.get(pair, 0) + 1 if not counts: break best = max(counts, key=lambda p: (counts[p], p)) # most frequent; tie -> largest pair (deterministic) if counts[best] < 2: break # nothing repeats: no merge would shorten the corpus merges[best] = next_id seqs = [merge_seq(s, best, next_id) for s in seqs] next_id += 1 return merges def bpe_encode(seq: list[int], merges: dict[tuple[int, int], int]) -> list[int]: for pair, new_id in merges.items(): # replay merges in learned order seq = merge_seq(seq, pair, new_id) return seq def bpe_decode(seq: list[int], merges: dict[tuple[int, int], int]) -> list[int]: inv = {new_id: pair for pair, new_id in merges.items()} changed = True while changed: # expand merged tokens back to their pairs until only base ids remain changed, out = False, [] for t in seq: if t in inv: out.extend(inv[t]) changed = True else: out.append(t) seq = out return seqThis is Karpathy's minbpe move, re-derived on action tokens. Start from an alphabet of the
distinct integers; repeatedly find the most frequent adjacent pair across the whole corpus
and merge it into one new token id; record the merge. Runs of zeros collapse first — (0,0) → A, then (A,A) → B — so a coefficient block's long zero tail becomes one or two tokens. BPE is
lossless: it is entropy coding stacked on top of the lossy DCT-and-quantize, so it changes
the token count and never the reconstruction. (The artifact asserts the round-trip, because
a codec you cannot invert is not a codec.)
Putting it together, and measuring it
# The whole FAST codec, one chunk at a time. encode: (DCT ->) quantize -> flatten. The# reconstruction comes straight from the quantized integers (BPE is lossless). --break# time_domain skips the DCT and spends the budget in TIME: keep every Nth action and# zero-order-HOLD it, then quantize the staircase — no basis in which energy concentrates.def downsample_hold(chunk_norm: np.ndarray, factor: int) -> np.ndarray: return np.repeat(chunk_norm[::factor], factor, axis=0)[: len(chunk_norm)] # ZOH back to H rows def encode_chunk(chunk_norm: np.ndarray) -> tuple[list[int], np.ndarray]: if USE_DCT: q = quantize(dct(chunk_norm), KEEP, args.q_scale, args.q_max) else: # --break time_domain: quantize the held (downsampled) samples directly q = quantize(downsample_hold(chunk_norm, args.downsample), H, args.q_scale, args.q_max) return flatten_grid(q), q def decode_chunk(q: np.ndarray) -> np.ndarray: grid = dequantize(q, args.q_scale) return idct(grid) if USE_DCT else grid # break path is already a time-domain staircase def error_jerk(recon: np.ndarray, target: np.ndarray) -> float: """Mean squared 2nd difference of the reconstruction ERROR — a smoothness meter for what the codec ADDED. The --break time_domain staircase inflates it ~200x (jerky residual).""" return float((np.diff(recon - target, n=2, axis=0) ** 2).mean()) # Encode every chunk, learn ONE BPE over the pooled action-token corpus, and remap the# raw integers to a dense 0..V-1 alphabet first (so merged-token ids never collide).raw_seqs, quants = zip(*(encode_chunk(c) for c in chunks))alphabet = sorted({v for s in raw_seqs for v in s})to_id = {v: i for i, v in enumerate(alphabet)}base_seqs = [[to_id[v] for v in s] for s in raw_seqs]merges = train_bpe(base_seqs, args.num_merges)encoded = [bpe_encode(s, merges) for s in base_seqs]assert all(bpe_decode(e, merges) == b for e, b in zip(encoded, base_seqs)), "BPE must round-trip (it is lossless)" fast_tokens = sum(len(e) for e in encoded) # DCT->quantize->BPE: the methodnaive_tokens = n_elems # per-step-per-dim binning: one token per action numbercompression_ratio = naive_tokens / fast_tokenscoeff_zero_frac = float(np.mean([(q == 0).mean() for q in quants])) # why BPE wins: the share of quantized coeffs that are 0The data region pulls real robot action chunks — the same PushT and ALOHA scripted demos
ch1.7's dataset is built from, replayed in-process for their actions only (no rendering, so
nothing here is platform-sensitive), sliced into non-overlapping H = 24 chunks. We encode
every chunk, learn one BPE over the pooled corpus, and count. The measured headline, seed 0:
| tokenizer | tokens | reconstruction RMSE |
|---|---|---|
| per-step-per-dim binning | 1968 | 0.012 |
| FAST (DCT → quantize → BPE) | 888 | 0.014 |
About 2.2× fewer tokens at the same reconstruction error — and it holds on every seed (2.2 / 2.3 / 2.5×). Read the two columns together: the RMSE barely moved (Parseval promised that), and yet the token count more than halved. Where did the tokens go? Into the ~17% of coefficients that quantized to zero and the BPE merges that ate their runs. That is the entire trick, and it is honest: no policy, no rollout, no success rate — a codec, measured as a codec.
Scout the data before you trust the method
Textbook FAST goes further: it truncates, keeping only a handful of low-frequency coefficients and discarding the rest, on the theory that "the robot never jerks." That works on smooth human teleop. It does not work here, and the honest reason is worth internalizing. Our scripted experts switch phases — approach, grasp, carry — and each switch is a step-discontinuity in the action, which is exactly the high-frequency content a low-pass throws away. Truncating our chunks to 6 of 24 coefficients drives RMSE to 0.21 — a fifth of the whole action range, unusable. So on this data the compression comes from Parseval + BPE-on-zeros, not from truncation. The toy — a deliberately smooth synthetic chunk — is where truncation is cheap (keep 6 of 24 → RMSE ~0.04) and where the demo lets you watch the mechanism at its best. Same method, different data, honestly different result: check the structure your method assumes before you believe a number.
Break it: is it the DCT, or the BPE?
# Reconstruction error (normalized action units) and error smoothness, per chunk. The naive# baseline quantizes raw samples at the SAME step; by Parseval the orthonormal DCT adds no# error of its own, so FAST's and naive's error energies are comparable. Compression is the win.sq_err_fast = sq_err_naive = 0.0ejerk_fast = ejerk_naive = 0.0for chunk_norm, q in zip(chunks, quants): recon = decode_chunk(q) q_naive = np.clip(np.round(chunk_norm / args.q_scale), -args.q_max, args.q_max) recon_naive = q_naive * args.q_scale # per-element time-domain dequant sq_err_fast += float(((recon - chunk_norm) ** 2).sum()) sq_err_naive += float(((recon_naive - chunk_norm) ** 2).sum()) ejerk_fast += error_jerk(recon, chunk_norm) # smoothness of FAST's residual ejerk_naive += error_jerk(recon_naive, chunk_norm) # smoothness of the white time-domain residual fast_rmse = (sq_err_fast / n_elems) ** 0.5naive_rmse = (sq_err_naive / n_elems) ** 0.5ejerk_fast, ejerk_naive = ejerk_fast / len(chunks), ejerk_naive / len(chunks)# The FORK (ch1.5 flow vs FAST): FAST decodes AUTOREGRESSIVELY, one LM step per token# (~fast_tokens/len(chunks) sequential steps); flow decodes the whole chunk in# FLOW_DECODE_STEPS Euler steps, flat. Tokens buy LM reuse + exact likelihood; flow buys a# short constant decode.fast_ar_steps = fast_tokens / len(chunks)print(f"FAST : {fast_tokens} tokens | naive per-step binning: {naive_tokens} tokens " f"-> {compression_ratio:.2f}x fewer ({coeff_zero_frac:.0%} of quantized coeffs are 0) " f"[break={args.break_mode or 'none'}]")print(f"recon RMSE (normalized): FAST {fast_rmse:.4f} | naive per-step {naive_rmse:.4f} (comparable — Parseval)")print(f"error jerk (residual smoothness): FAST {ejerk_fast:.6f} | naive per-step {ejerk_naive:.6f}")print(f"decode fork: FAST ~{fast_ar_steps:.1f} autoregressive steps/chunk vs " f"flow {FLOW_DECODE_STEPS} Euler steps (constant)")It is tempting to think BPE is doing the real work and the transform is decoration. Settle it by
generating the failure. --break time_domain spends the same kind of budget — keep fewer
numbers, reconstruct — but in the time domain: keep every Nth action and zero-order-hold it,
then quantize the staircase. Predict what that does to the reconstruction, then run it.
| metric | clean FAST | --break time_domain |
|---|---|---|
| reconstruction RMSE | 0.014 | 0.220 (~15×) |
| error jerk | 0.0012 | 0.264 (~200×) |
The motion falls apart. A trajectory's information is spread across every timestep — there is no basis in the time domain where energy concentrates — so dropping timesteps and holding is a jerky staircase with fifteen times the error. Keep the same number of coefficients and the smooth motion comes back; keep the same number of samples and it does not. The DCT basis is load-bearing. (One honest note the pilot's playbook forces: the naive comparison in the contract — plain per-step binning giving "an order of magnitude more tokens" — does not survive measurement, because BPE happily compresses a slow time-domain signal too. The claim that reproduces, and the break that actually degrades, are the ones above.)
The fork: FAST vs flow
You now hold both decoders, so name the tradeoff plainly. FAST decodes autoregressively — one LM step per token, ~35 sequential steps for one of these chunks — but every step is a plain cross-entropy over a vocabulary, so it reuses a language model's whole stack and gives you exact per-token likelihoods for free. Flow (ch1.5) decodes the entire chunk continuously in a few Euler steps flat, no vocabulary and no autoregression, but it needs a bespoke sampler and gives no discrete likelihood. Tokens buy language-model reuse; flow buys a short, constant decode. pi0 famously ships both — flow for real-time control, FAST for pretraining — and now you know why neither is simply "better."
Read the real thing
Everything here has a production form in openpi's pi0_fast.py and its FAST processor: the
cosine transform, the scale-and-round quantizer, the byte-pair merges, and the autoregressive
detokenize that runs your bpe_decode → dequantize → idct in reverse. You will recognize every
line. What the real processor adds is scale (a real action vocabulary, real teleop) and the wiring
into an LM cross-entropy head — the fork this chapter only sketched.
What's next
You turned continuous motion into a short string of integers and back, and measured — honestly, on real robot chunks — exactly what the DCT, the quantizer, and the BPE each contribute. That is the last piece of the modern VLA data stack: a policy can now speak actions the way it speaks words, and the same transformer that reads an instruction can write the motion that answers it.