The wrong mental model
You have a trained policy. Someone says "ship it in int8, it'll be four times smaller and
faster." The obvious thing to try is a cast: take each weight, round it to the nearest whole
number, store it as a signed byte. Try it on the ch1.1 behavior-cloning MLP, whose weights all
live in (-0.5, 0.5), and the policy evaporates — round(0.31) = 0, round(-0.12) = 0,
about 95% of the weights round to zero, and the robot goes limp.
That failure is the whole lesson in miniature. INT8 quantization is not a rounding. It is a
scale. The int8 grid is a fixed ruler with 255 evenly spaced ticks from -127 to +127;
the same ruler for every tensor in every model. Quantizing means choosing a real number s —
the spacing between ticks in your tensor's units — and storing q = round(w / s). You recover
w ≈ s · q. Rounding threw away the scale and measured a policy in the wrong units. Open
quantize.py. Six regions: setup, data, train, quantize, deploy, report.
Everything in this chapter runs on the CPU in numpy, and that is a feature: int8 and int32 integer arithmetic is exact and bitwise-reproducible. Same seed, same bytes — no cuDNN nondeterminism to caveat. This is the one chapter where "deterministic" needs no asterisk.
A scale, per tensor and per channel
# The core. A symmetric int8 scale maps a real range [-r, r] onto the integer grid# [-127, 127]: s = r / 127, and q = round(w / s) clamped to the grid. Dequantize is one# multiply, w_hat = s * q. The ONLY choice is what "r" ranges over:# per-tensor : r = max|W| over the WHOLE matrix -> one scale. One fat channel sets the# step size for every channel, so small-magnitude rows lose all resolution.# per-channel : r = max|W[o]| per OUTPUT ROW -> one scale each. Every row gets a step# size matched to its own range. This is the recovery.def quantize_weight(W: np.ndarray, per_channel: bool): r = np.abs(W).max(axis=1, keepdims=True) if per_channel else np.abs(W).max() # (out,1) or scalar scale = np.maximum(np.asarray(r, np.float32) / QMAX, 1e-8) # never divide by zero q = np.clip(np.round(W / scale), -QMAX, QMAX).astype(np.int8) return q, scale # scale broadcasts back over `in`: w_hat = q * scale def weight_roundtrip_err(per_channel: bool) -> float: """Mean |W - dequant(quant(W))| across the three layers. Per-channel is guaranteed <= per-tensor: a per-row scale is a strict refinement of a single one.""" errs = [] for W, _ in LAYERS: q, scale = quantize_weight(W, per_channel) errs.append(np.abs(W - q.astype(np.float32) * scale).mean()) return float(np.mean(errs)) # THE MISCONCEPTION killed in one line: rounding weights to int8 with NO scale collapses# them — everything in (-0.5, 0.5) rounds to 0. The scale is what saves the signal.NAIVE_ROUND_ZERO_FRAC = float(np.mean([(np.round(W) == 0).mean() for W, _ in LAYERS])) # Activations we cannot quantize offline — we don't know their range until data flows. So run# the calibration states through the fp32 net, record each layer's INPUT range, and set the# scale r/127 with r = max|a| (minmax) or the 99.9th percentile of |a| (percentile ignores a# single freak activation that would otherwise blow the scale up).def calibrate(calib_input: np.ndarray, rule: str) -> list[float]: scales, x = [], normalize(calib_input) for i, (W, b) in enumerate(LAYERS): r = np.abs(x).max() if rule == "minmax" else np.percentile(np.abs(x), PERCENTILE) scales.append(float(max(r / QMAX, 1e-8))) # this layer's activation scale, from its INPUT x = x @ W.T + b if i < len(LAYERS) - 1: x = np.maximum(x, 0.0) return scales # The --break: calibrate on a NARROW slice — only the frames where the block already sits# near the target, where the policy barely moves and activations are small. The scales come# out too tight; at real deployment the activations run off the end of the grid and clamp.if args.break_mode == "bad_calib": near_goal = calib_dist <= np.quantile(calib_dist, 0.25) # the 25% closest-to-goal states calib_input, calib_rule = calib_obs[near_goal], "minmax" # narrow AND minmax: no percentile safety net either print(f"[break bad_calib] calibrating on {int(near_goal.sum())}/{len(calib_obs)} near-goal frames " f"(block dist <= {np.quantile(calib_dist, 0.25):.3f}) — a distribution the policy never deploys in")else: calib_input, calib_rule = calib_obs, args.calibACT_SCALES = calibrate(calib_input, calib_rule) def forward_int8(obs: np.ndarray, per_channel: bool, quant_act: bool) -> np.ndarray: """One int8 forward, two modes. quant_act=False is WEIGHT-ONLY (the dominant real-world PTQ mode: int8 weights, dequantized on the fly, fp32 matmul) — the deployment triangle. quant_act=True is FULL INTEGER (activations quantized with the STATIC calibrated scales, int8 @ int8 -> int32 accumulate) — the path the --break attacks. Bias stays fp32.""" x = normalize(obs) for i, (W, b) in enumerate(LAYERS): Wq, s_w = quantize_weight(W, per_channel) if quant_act: s_x = ACT_SCALES[i] x_int = np.clip(np.round(x / s_x), -QMAX, QMAX).astype(np.int32) # quantize activations (may CLAMP) acc = x_int @ Wq.astype(np.int32).T # exact integer matmul x = acc.astype(np.float32) * (s_x * s_w.reshape(-1)) + b # dequant: s_x * s_w per output channel else: x = x @ (Wq.astype(np.float32) * s_w).T + b # dequant weights, fp32 matmul if i < len(LAYERS) - 1: x = np.maximum(x, 0.0) return denormalize(x)quantize_weight is the whole idea in four lines. Take the range r = max|W|, set the scale
s = r / 127, and round W / s onto the grid. Dequantize is a single multiply, s · q. The
only decision is what r ranges over:
- per-tensor: one
rfor the whole matrix. Simple — and wasteful. One fat-tailed output channel with a big weight sets the tick spacing for every channel, so a row whose weights are all tiny gets crushed onto three or four ticks and loses almost all its resolution. - per-channel: one
rper output row. Every channel gets a tick spacing matched to its own range. The fat channel no longer taxes the small ones.
Measure the round-trip error mean|W − s·q| both ways and per-channel is guaranteed smaller
— a per-row scale is a strict refinement of a single scale. On the trained policy it is ~1.7–2.0×
smaller, every seed. That refinement is the recovery you will see propagate all the way to the
policy's actions. And the one-liner right below it, NAIVE_ROUND_ZERO_FRAC, is the misconception
from the opening, measured: round with no scale and ~95% of the weights collapse to zero.
Activations don't hold still
Weights you can quantize offline — they are fixed numbers, you know their range. Activations you cannot: their range depends on the input, and you don't know it until data flows through the net. So you calibrate.
# The core. A symmetric int8 scale maps a real range [-r, r] onto the integer grid# [-127, 127]: s = r / 127, and q = round(w / s) clamped to the grid. Dequantize is one# multiply, w_hat = s * q. The ONLY choice is what "r" ranges over:# per-tensor : r = max|W| over the WHOLE matrix -> one scale. One fat channel sets the# step size for every channel, so small-magnitude rows lose all resolution.# per-channel : r = max|W[o]| per OUTPUT ROW -> one scale each. Every row gets a step# size matched to its own range. This is the recovery.def quantize_weight(W: np.ndarray, per_channel: bool): r = np.abs(W).max(axis=1, keepdims=True) if per_channel else np.abs(W).max() # (out,1) or scalar scale = np.maximum(np.asarray(r, np.float32) / QMAX, 1e-8) # never divide by zero q = np.clip(np.round(W / scale), -QMAX, QMAX).astype(np.int8) return q, scale # scale broadcasts back over `in`: w_hat = q * scale def weight_roundtrip_err(per_channel: bool) -> float: """Mean |W - dequant(quant(W))| across the three layers. Per-channel is guaranteed <= per-tensor: a per-row scale is a strict refinement of a single one.""" errs = [] for W, _ in LAYERS: q, scale = quantize_weight(W, per_channel) errs.append(np.abs(W - q.astype(np.float32) * scale).mean()) return float(np.mean(errs)) # THE MISCONCEPTION killed in one line: rounding weights to int8 with NO scale collapses# them — everything in (-0.5, 0.5) rounds to 0. The scale is what saves the signal.NAIVE_ROUND_ZERO_FRAC = float(np.mean([(np.round(W) == 0).mean() for W, _ in LAYERS])) # Activations we cannot quantize offline — we don't know their range until data flows. So run# the calibration states through the fp32 net, record each layer's INPUT range, and set the# scale r/127 with r = max|a| (minmax) or the 99.9th percentile of |a| (percentile ignores a# single freak activation that would otherwise blow the scale up).def calibrate(calib_input: np.ndarray, rule: str) -> list[float]: scales, x = [], normalize(calib_input) for i, (W, b) in enumerate(LAYERS): r = np.abs(x).max() if rule == "minmax" else np.percentile(np.abs(x), PERCENTILE) scales.append(float(max(r / QMAX, 1e-8))) # this layer's activation scale, from its INPUT x = x @ W.T + b if i < len(LAYERS) - 1: x = np.maximum(x, 0.0) return scales # The --break: calibrate on a NARROW slice — only the frames where the block already sits# near the target, where the policy barely moves and activations are small. The scales come# out too tight; at real deployment the activations run off the end of the grid and clamp.if args.break_mode == "bad_calib": near_goal = calib_dist <= np.quantile(calib_dist, 0.25) # the 25% closest-to-goal states calib_input, calib_rule = calib_obs[near_goal], "minmax" # narrow AND minmax: no percentile safety net either print(f"[break bad_calib] calibrating on {int(near_goal.sum())}/{len(calib_obs)} near-goal frames " f"(block dist <= {np.quantile(calib_dist, 0.25):.3f}) — a distribution the policy never deploys in")else: calib_input, calib_rule = calib_obs, args.calibACT_SCALES = calibrate(calib_input, calib_rule) def forward_int8(obs: np.ndarray, per_channel: bool, quant_act: bool) -> np.ndarray: """One int8 forward, two modes. quant_act=False is WEIGHT-ONLY (the dominant real-world PTQ mode: int8 weights, dequantized on the fly, fp32 matmul) — the deployment triangle. quant_act=True is FULL INTEGER (activations quantized with the STATIC calibrated scales, int8 @ int8 -> int32 accumulate) — the path the --break attacks. Bias stays fp32.""" x = normalize(obs) for i, (W, b) in enumerate(LAYERS): Wq, s_w = quantize_weight(W, per_channel) if quant_act: s_x = ACT_SCALES[i] x_int = np.clip(np.round(x / s_x), -QMAX, QMAX).astype(np.int32) # quantize activations (may CLAMP) acc = x_int @ Wq.astype(np.int32).T # exact integer matmul x = acc.astype(np.float32) * (s_x * s_w.reshape(-1)) + b # dequant: s_x * s_w per output channel else: x = x @ (Wq.astype(np.float32) * s_w).T + b # dequant weights, fp32 matmul if i < len(LAYERS) - 1: x = np.maximum(x, 0.0) return denormalize(x)calibrate runs a calibration set — a handful of held-out states — through the fp32 network
and records the range of each layer's input activations. From that range you get an activation
scale, exactly like a weight scale. Two rules for reading the range:
- min-max:
r = max|a|. Honest, until one freak state produces a giant activation and drags the scale up so far that ordinary activations lose their resolution. - percentile:
r =the 99.9th percentile of|a|. Ignores the top 0.1%, so a single outlier can't blow up the scale. This is the robust default (it is exactly what onnxruntime's percentile calibrator does).
With activation scales in hand, the forward pass becomes all integers: quantize the input to
int8, do an int8 @ int8 → int32 matmul (int32 because a sum of products of bytes overflows a
byte), then dequantize the accumulator back to real units with s_x · s_w and add the fp32 bias.
# The deployment triangle: for each config measure (a) storage size, (b) action error vs# the fp32 policy, (c) per-call CPU latency, (d) task success with a Wilson interval so a# "drop" only counts once it clears fp32's band (ch1.6).Z95 = 1.959963985 # 0.975 standard-normal quantile; the 95% two-sided Wilson interval, no scipy def wilson_ci(k: int, n: int) -> tuple[float, float]: """95% Wilson score interval for k successes in n trials (ch1.6's from-scratch CI).""" if n == 0: return (0.0, 1.0) p, z2 = k / n, Z95 * Z95 denom = 1.0 + z2 / n center = (p + z2 / (2 * n)) / denom half = (Z95 / denom) * math.sqrt(p * (1.0 - p) / n + z2 / (4 * n * n)) return (max(0.0, center - half), min(1.0, center + half)) def model_size_bytes(int8: bool, per_channel: bool) -> int: """Storage the model actually costs. fp32: 4 bytes/param. int8: 1 byte/weight + the fp32 scales (1 per tensor, or 1 per output channel) + fp32 bias (unquantized here).""" total = 0 for W, b in LAYERS: total += (W.size * 4 if not int8 else W.size + (W.shape[0] if per_channel else 1) * 4) + b.size * 4 return total def action_mse_vs_fp32(fn) -> float: return float(np.mean((fn(eval_obs) - forward_fp32(eval_obs)) ** 2)) def latency_ms(fn) -> float: """Median single-observation latency (batch=1, the real deployment shape). Wall clock is never bit-reproducible, so --smoke skips it (0.0) to keep metrics.json byte-stable; the real number comes from a full run + wallclock-bench.""" if args.smoke: return 0.0 one = eval_obs[:1] for _ in range(20): fn(one) # warm up so the first-call cost doesn't skew the median samples = [] for _ in range(200): t0 = time.perf_counter() fn(one) samples.append((time.perf_counter() - t0) * 1e3) return float(np.median(samples)) def rollout_success(fn, n_episodes: int) -> tuple[int, int]: """Roll `fn` out on held-out start seeds; return (successes, n). Bit-reproducible.""" env, k = PushTEnv(), 0 for e in range(n_episodes): obs = env.reset(EVAL_BASE_SEED + 500 + e) # a held-out band, disjoint from training + eval-obs seeds done, info = False, {} while not done: obs, _, done, info = env.step(fn(obs[None])[0]) k += bool(info["success"]) return k, n_episodes def measure(fn, int8: bool, per_channel: bool) -> dict: k, n = rollout_success(fn, args.eval_episodes) lo, hi = wilson_ci(k, n) return {"size_kb": round(model_size_bytes(int8, per_channel) / 1024, 3), "action_mse": round(action_mse_vs_fp32(fn), 8), "latency_ms": round(latency_ms(fn), 4), "success": k, "n": n, "success_rate": round(k / n, 4), "ci_lo": round(lo, 4), "ci_hi": round(hi, 4)} # The triangle is WEIGHT-ONLY int8 (quant_act=False): the clean, seed-robust headline.triangle = { "fp32": measure(lambda o: forward_fp32(o), False, False), "per_tensor_int8": measure(lambda o: forward_int8(o, per_channel=False, quant_act=False), True, False), "per_channel_int8": measure(lambda o: forward_int8(o, per_channel=True, quant_act=False), True, True),}# The all-integer path (part 2): per-channel weights + STATIC calibrated activations. Under a# good calibration it pays a modest extra floor; --break bad_calib makes its error explode.full_int8 = measure(lambda o: forward_int8(o, per_channel=True, quant_act=True), True, True)for name, cfg in list(triangle.items()) + [("full_int8(+act)", full_int8)]: print(f" {name:18s} {cfg['size_kb']:6.2f} KB mse {cfg['action_mse']:.2e} {cfg['latency_ms']:6.3f} ms " f"success {cfg['success']}/{cfg['n']} [{cfg['ci_lo']:.2f}, {cfg['ci_hi']:.2f}]") wt_err_per_tensor = weight_roundtrip_err(per_channel=False)wt_err_per_channel = weight_roundtrip_err(per_channel=True)The deployment triangle
There is no free lunch in deployment; there is a triangle — size, accuracy, latency — and you
have to look at all three corners at once. quantize.py measures them for FP32, per-tensor INT8,
and per-channel INT8 (the triangle uses weight-only int8: int8 weights, dequantized on the
fly, fp32 matmul — the dominant real-world mode, what llama.cpp's Q8 and HuggingFace int8 do).
Size. Guaranteed. An int8 weight is one byte where fp32 was four. The measured ratio is ~3.6×, a hair short of 4× only because the biases and the per-channel scales stay fp32.
Accuracy. This is the headline, and it reproduces on every seed: per-tensor INT8 spikes the action error; per-channel INT8 recovers most of it — 2–7× lower action-MSE-vs-fp32, at the same ~4× size saving. The per-row scale you built in region three is the entire difference. We grade accuracy with ch1.6 rigor: each config is also rolled out for a success rate with a Wilson interval, and at this eval budget those intervals overlap fp32 — so we make no task-success claim. Quantization here is task-indistinguishable from fp32 at N=24. The claim we stand behind is the deterministic action-error direction and the size win, not a success-rate drop that never cleared the band.
Latency. Here is the honesty the marketing skips. The size win is guaranteed; the latency win is not. On a laptop CPU with no fused int8 kernel, quantizing and dequantizing around a plain matmul is overhead, and naive int8 comes out ~6× slower than fp32. That is not a bug in the code — it is the reason production deployment reaches for TensorRT or a fused QLinear kernel, where the integer matmul is actually faster than the float one. We measure it and print whichever wins; on this CPU, int8 loses.
Break it: calibrate on the wrong distribution
python curriculum/phase5_practitioner/ch5.7_quantize/quantize.py --seed 0 --break bad_calib
The fragile corner of static quantization is the calibration set. --break bad_calib calibrates
the activation scales on a narrow slice — only the frames where the block already sits near
the target, where the policy barely moves and activations are small. The scales come out too
tight. At real deployment the activations run right off the end of the int8 grid and saturate
at ±127, and the full-integer action error explodes — 4.5–24× worse than a representative
calibration, every seed.
Two things to notice. First, the weight-only triangle is untouched — the break only attacks the activation path, so a bug in your calibration data hides completely from the weight-quant numbers. Second, switching per-tensor → per-channel weights does not rescue it: when activations clamp, weight granularity is irrelevant. These are two independent failure modes. The fix for saturation is a calibration set that matches deployment (plus a percentile clip for the single-outlier flavor) — not a finer weight scale.
What you built
A complete post-training quantization pipeline, from scratch, in numpy you can read: a symmetric
int8 scale, per-tensor and per-channel weight quantization, static activation calibration with a
percentile clip, a full-integer forward pass, and an honest three-corner measurement of what it
costs. The read-the-real-thing segment opens onnxruntime's quantization/ tools — calibrate.py,
onnx_quantizer.py, quantize.py — and you will recognize every piece, because you just wrote
the toy version of each. The scale lab is the corner this CPU can't show you: the same policy
under a fused int8 kernel, where the latency finally turns in int8's favor.