9Quantisation

Four-bit weights take a decode step from 26.93 GB to 15.01 GB (arithmetic), and on this board bytes are time. This chapter covers the formats the engine reads, the quantiser that wrote the published 4-bit set, the gate that decides whether a weight set ships, and what stays at full precision.

The served weights are the published set, 0xBakeer/TandemLLM-Qwen3.8-27B-NVFP4 on Hugging Face: every projection in NVFP4, quantised once from Qwen’s BF16 release with GPTQ, plus an output head in FP8 (e4m3). The files load over the vendor’s FP8 checkpoint, which supplies every tensor the overlays do not replace. The 27B also runs from six public EXL3 sizes (Section 9.7). The second model, Qwen3.8-Flash-Next, is not quantised here: it runs from turboderp’s published EXL3 packs, read as they are (Section 9.9).

9.1 What the checkpoint stores

Qwen ships the model as block-scaled FP8. Every large projection is a pair of tensors: X.weight holds e4m3 codes of shape $[N, K]$, and X.weight_scale_inv holds one BF16 scale per 128 by 128 block. A weight is

engine/loader.py keeps that pair as an FP8Block and never expands it in memory, because a decode step pays for exactly these bytes. Some tensors stay BF16 in the checkpoint: the output head, the embedding, every norm, the convolution weights and two small projections of the recurrent layers.

9.2 NVFP4

NVFP4 stores a weight in three tensors:

A weight is $\text{e2m1}(\textit{code}) \cdot \textit{weight\_scale}[n, \lfloor k/16 \rfloor] \cdot \textit{weight\_scale\_2}$. The e2m1 grid has eight magnitudes (0, 0.5, 1, 1.5, 2, 3, 4 and 6) and a sign. A weight costs 0.5625 bytes against 1.0 in FP8, so one layer’s MLP drops from 267.4 MB to 150.4 MB (arithmetic).

The group of 16 is what makes 4 bits usable. An FP8 block shares one scale across 16,384 weights; an NVFP4 group shares it across 16, so the grid follows a row’s local range.

The engine’s kernel is W4A16: it unpacks the 4-bit codes to 16-bit numbers in registers and multiplies on the ordinary tensor cores. On this board NVFP4 is a storage format, not a compute format. It saves bytes, and the arithmetic stays 16-bit (Chapter 10).

9.3 The quantiser

tools/quant_nvfp4.py build loads the BF16 model once (about 50 GB) and does everything in that process.

1. It runs a calibration text through the engine. tools/calib_corpus.py writes it from bench/calib.txt, Python standard-library modules, English Wikipedia and German Wikipedia: 149,682 tokens in the build that was published. Any document that shares a run of 12 words with a held-out file is dropped.
2. It records each projection’s mean square input, and the full second moment of each input, $H = X^\top X$. At fp32 the second moments are 101 GB for 64 layers, so the build takes them in passes of about 13 layers (--h-budget-gb 22).
3. It quantises every projection with GPTQ on the NVFP4 grid. GPTQ rounds one input column at a time and pushes each column’s rounding error onto the columns not yet rounded, weighted by the inverse of $H$. At the first column of each group of 16, a clip search picks the group’s scale from the weights as they stand.

The clip search tries five scale multipliers per group (1.00, 0.95, 0.90, 0.85 and 0.80). A smaller multiplier clips the group’s largest weight and gives the rest a finer grid. It keeps the multiplier with the smallest error as the model sees it,

where act_ms$[k]$ is the mean square of input channel $k$ over the calibration text. A channel the model never uses can take a coarse grid at no cost, and a channel it leans on cannot. GPTQ left a lower output error than the clip search alone on the calibration inputs, and a lower loss on held-out text, so the published set uses it. The format and the kernel do not change between the two; only the stored codes do.

The output head is quantised separately. tools/quant_head.py build writes it as e4m3 with one fp32 scale per vocabulary row (1.27 GB instead of 2.54 GB), searching the multipliers 1.0, 0.95 and 0.90 of each row’s largest weight. One scale per row, because rows are tokens, and their sizes differ more than a block scale shared by 128 neighbouring ids can follow. The head decides what the engine writes, so it took the full gate. Its kernel returns fp32 logits at every row count.

9.4 The gate

tools/quality_gate.py reports three things, and a weight set ships only when all three hold.

1. Held-out loss, teacher-forced, as a change against a reference on the same tokens in the same process. With --plan the reference is a model loaded from another checkpoint, which is how the published set is scored against BF16 while it loads over the FP8 release as it does in serving. The gate files are bench/heldout_code.txt, a Python file the calibration does not contain, and bench/heldout_prose.txt, English prose on unrelated topics. The bar is +0.05 nats on the worse of the two.
2. Argmax agreement with the reference, overall and at positions where the reference is confident (its top choice ahead of the second by at least 1.0 in the logits). On a flat distribution over 248,320 tokens the top choice is decided by rounding, so the confident number is the one to watch.
3. A free generation of 900 greedy tokens on five prompts the calibration never saw, with repetition statistics.

Test 3 matters most. Teacher-forced loss never lets an error grow, so it cannot see a model that drifts off its own path. On an earlier engine on this board, loss rated a broken configuration better than a healthy one.

The published set passes on both gate files: +0.0221 nats on prose and +0.0426 on code, inside the +0.05 bar (Figure 9.1). Code costs more than prose. Re-run the gate before quantising anything further, and before trusting a weight set from elsewhere: a weight set passes or fails on its own measurement, not on its format.

9.5 What stays at full precision

The 48 recurrent states stay fp32. They are 151 MB, read and written once a step, about 1.5 ms of traffic. A recurrent state is a sum over the whole sequence, carried across every step, so its rounding error grows with every step; a weight’s does not. The convolution weights, the decay and bias tensors of the recurrent layers, every norm and the drafters’ own layers keep their shipped precision too.

9.6 Profiles and bytes

Three serve.env profiles ship with the NVFP4 set (Table 3.8); the EXL3 weight profiles of Section 9.7 and Flash-Next’s packs are switched with ops/use-profile.sh. The FP8 profile reads nothing but the vendor’s files, so its quality is the checkpoint’s. The balanced profile keeps the recurrent and attention layers on the vendor’s weights and takes the published set’s NVFP4 MLPs and FP8 head; the MLPs carry 70 % of a verify’s projection bytes. Neither profile’s speed is measured on this release.

Bytes and accepted tokens partly substitute for each other. At one token a step every byte saved is time saved. On a round that commits several tokens the weight read is already spread over them, so the gain from fewer bytes shrinks as acceptance grows.

9.7 The 27B in EXL3

Since 0.5.0 the 27B also runs from EXL3 weights at six sizes, one weight profile each (ops/profiles/exl3-<bpw>.env, switched with ops/use-profile.sh, Section 6.3). We quantised them with exllamav3 from Qwen’s BF16 release, and all six are public on Hugging Face as 0xBakeer/TandemLLM-Qwen3.8-27B-EXL3-<bpw>. The drafters for these profiles are in 0xBakeer/TandemLLM-Qwen3.8-27B-EXL3-drafters: both DFlash2 drafters with their projections in EXL3 at 3 bits a weight (the K3 files) and a draft head at 4 bits (K4). The six sizes and the drafters form one collection, TandemLLM Qwen3.8-27B EXL3.

Each card also has a speed table for every size on one DGX Spark (TandemLLM 0.6.2, one greedy request of 512 tokens, speculation on). An EXL3 profile needs four more downloads beside its weights: the K3 drafters and the K4 head (1.9 GB), the BF16 drafters 0xBakeer/TandemLLM-Qwen3.8-27B-DFlash2-b8 and -b16 (7.7 GB, into the repository’s train/), and the base checkpoint Qwen/Qwen3.8-27B-FP8 (30.4 GB, without mtp.safetensors). Since 0.8.1 ops/use-profile.sh fetches all of them, checks each weight file against the hub’s sha256, and needs no token. Each card’s calibration/ folder has the recipe; the calibration repository itself stays private, and no profile reads it.

9.8 Standalone EXL3 packs

Our EXL3 profiles are overlays: their files hold the coded projections, and the vendor’s FP8 checkpoint supplies the rest. Since 0.8.2 a complete EXL3 pack of the 27B family also loads on its own, passed as --model, with no BF16 or FP8 base to download. 0xWhiteMage wrote this path in pull request #1, and the first community pack (Chapter 7) runs on it. The loader recognises quantization_config.quant_method: exl3 in the pack’s configuration; the pack must hold the embedding, the norms and the recurrent layers’ small tensors too.

python server/app.py --model /path/to/exl3-pack --drafter none --vision off

The projections and the output head keep their EXL3 codes, channel scales and mixed bit widths, also when a projection’s parts span several shards. When the pack declares a BF16 source (dtype or torch_dtype), the loader turns its FP16 plain tensors into BF16 only where every stored value survives the round trip exactly. Missing or non-BF16 source metadata, a value that is not finite, or a lossy conversion stops the load with a named error. The pack’s stored values are kept; precision lost while the pack was made cannot come back, and only the original checkpoint can show that a pack equals its source.

The path adds no codebook, no architecture and no MTP format: the mul1 reader and its checks are the ones above. It has three limits. It is text only, even for a pack that carries a vision tower, and the registry refuses a community entry that claims images on it (ENG-271). Quantised MTP layers are not qualified. A stock DFlash2 drafter runs one sequence (--drafter dflash2 --dflash2-ckpt <dir>), and speculative parallel serving still needs the lenrouter configuration with our drafters, which use-profile.sh community/<id> fetches for a pack whose entry names them. Our own profiles, an EXL3 overlay over the base or NVFP4, load exactly as before.

9.9 Flash-Next’s EXL3 packs

turboderp publishes Qwen3.8-Flash-Next in five sizes, one branch each of turboderp/Qwen3.8-Flash-Next-exl3, and the engine reads them as they are: no quantiser of ours runs on them and no format is added. The branch name says what is inside. 3.05bpw_h5_ng5 is 3.05 bits a weight on average, the output head at 5 bits and the n-gram table at 5 bits a value. Table 17.2 lists the five with the window each fits. A pack is lossy against the BF16 release by design, and speculation is lossless against the pack that is served (Section 17.7).

engine/models/flashnext/pack.py reads every shard header and sorts each tensor into one kind (Table 9.3). An EXL3 projection is four tensors: X.trellis holds the codes at $K$ bits a weight, X.suh and X.svh scale and sign the rotated input and output sides, and X.mul1 is the codebook’s multiplier. $K$ is read from each tensor’s own shape. On the 3.05 pack the audit counted 303,121 tensors, every routed expert at $K=3$, and the dense projections and the head at $K=5$ except the sparse-attention indexer’s projection ($K=3$) and the MTP layer’s ($K$ from 3 to 5). By those headers a plain decode token reads about 4.23 GB in all at 1k of context (arithmetic).

tools/fn_exl3_xcheck.py holds our reading against the converter’s own on the 3.05 pack: the trellis decode equal bit for bit on 16 tensors, the dequantised weight within 0.36 % of the largest weight (our BF16 rounding of the fp32 rotations), and 4,096 n-gram rows equal bit for bit (measured).

The mixers are read on every token, so the engine keeps them as int8 with one fp32 scale for each group of 128 weights of a row (QWEN38_FN_MIX, int8; int8rows is one scale a row, fp16 the pack’s values); a weight is off by at most its group’s largest weight over 254. Against the pack’s fp16 mixers on the same kernels, with 8,192 tokens on 3.05, the groups gave a KL of 0.0031 on code and 0.0188 on prose, inside the arithmetic noise between the reference forward and the kernels (0.0038 and 0.0255). One scale a row gave 0.267 on code and 0.047 on prose, because rows whose largest weight is 10 to 420 times their median left the small weights coarse. Int4 gave 0.38 on prose and changed the continuation at its first token, so the mixers stay int8 (measured, tools/fn_mix_quality.py). The groups cost no decode speed and 0.02 GiB. The prefill reads exact fp32 copies of the mixers, the router and the small projections (QWEN38_FN_PREFILL_MM); with fp16 copies it failed its agreement gate on a 3k-token code prompt.

The n-gram table codes each 160-value row as a ring of trellis codes at 4 to 6 bits a value, with one fp16 scale a row. It is 32.6 GB in the 3.05 pack and never sits in device memory: each token reads 16 rows and decodes them on the device. A profile reads it warm, through the page cache, or cold, with O_DIRECT through a 2 GiB row cache (QWEN38_FN_TABLE); Chapter 17 says which pack uses which and why.

Nobody has measured a pack’s quality against the BF16 model. The BF16 release is 360 GB and does not fit the board, so the registry’s quality fields are empty. What is measured is our arithmetic against the reference on the same pack: all 48 layers equal the official transformers layer within BF16 noise (relative error 0.19 to 1.49 %).

9.10 Knobs

The gate files are one held-out file per domain, 2,048 tokens each, and the wider texts add 8k to 12k tokens per domain. No task benchmark suite has run against the quantised engine (Chapter 15, Limits).