12Exactness

Under greedy decoding, TandemLLM writes the text its target model writes one token at a time, with one exception: at a near tie between the two best tokens, a batched verify can round the other way. Drafters, the lookup store and the caches change how fast the text arrives, not what it says. The length router picks the tree’s shape from timings, so two runs of the same request can also part at a near tie (Section 12.3). This chapter states the promise, names the exception, and lists the checks that hold the engine to it.

12.1 The promise

The reference is plain greedy decoding on the same weights: one forward pass per token, take the highest-scoring token, repeat. For every greedy request, the engine’s output tokens equal the reference’s tokens up to the first near tie where the two paths choose differently.

“The same weights” matters. A profile chooses the weights, and the NVFP4 profile is held to NVFP4 plain decoding, not to the BF16 release or the FP8 checkpoint. The difference between weight sets is a quality question, and the quality gate measures it separately (Section 9.4). Speculation never adds to that difference.

The promise, with its exception, covers:

• any drafter, including one that is wrong on purpose on every token;
• chains and trees of any shape the router builds, and both block widths;
• a request resumed from the prefix cache or the resident prefix;
• request options that change the scores (penalties, logit_bias, structured-output masks), because the engine applies them to every verified row exactly as plain decoding applies them to its one row.

12.2 The exception: near ties

Two tokens can score the same up to one rounding step of the arithmetic. Which one wins an argmax then depends on the order of a floating-point sum. A batched verify and a one-token step do not always sum in the same order, so at such a position the two can pick different tokens. After that position the texts differ, and both are correct greedy continuations of their own prefixes.

The engine keeps this rare by design. The head returns fp32 logits, and every matrix-product kernel fixes its reduction order by the weight’s shape and never by the row count (Chapter 10). Two differences remain between a batched verify and a one-token step: the output head uses one kernel for a single row and another for several, and the verify path of the recurrent and attention layers does the one-token arithmetic in a different order. A small fp32 difference that falls on a BF16 rounding boundary changes one value; the recurrent state and the KV cache carry it forward, so over a long output the verify’s logits drift from the one-token decode’s. Where the two best tokens are closer than that drift, the two paths can choose differently.

The agreement gate, tools/verify_spec.py, prints the gap between the top two logits in BF16 units in the last place (ulps) wherever outputs differ. A difference passes only when the gap at that position is within one ulp; anything larger is a bug. The gate decodes a fixed number of tokens and does not stop at the end-of-turn token, so a difference after that token, which no answer served with the default stop tokens contains, is judged separately.

The paper measured how often this happens (Figure 12.1). Of 30 prompts, 17 outputs are identical to plain greedy decoding to the end of the answer: 13 over all 256 tokens, and 4 that differ only after the answer ended (largest gap 1.46 ulp). The other 13 follow plain decoding up to a near tie inside the answer, with a gap of 0.04 to 0.76 ulp, take the other token, and continue from there; in one of them the token taken is the end-of-answer token, so that answer ends early. The one-ulp rule names the condition under which the two decoders can differ. It does not show that the difference is harmless.

12.3 Two runs of the same request

The length router sizes each block from timings: the draft and verify times it measures inside the request and, unless the server runs with --no-learn-cost, the costs it learned from earlier requests. Two runs of the same prompt can therefore build different trees. Identical tree shapes give identical bits, but different shapes can move a near tie, so the text of two runs is the same only while their shapes are. Two runs alone can part this way, and so can a run with side-lane guests and one without.

Table 12.1 has a measurement on the 0.4.0 code (lookup store off, 2026-10-05): one raw greedy turn with a prompt of about 5,000 tokens at temperature 0, sent alone twice and once with 10 side-lane guests. Each parting is a near tie. Where the runs alone parted, the gap between the two best tokens was 0.022 and 0.029 nats at token 12, and 0.052 and 0.021 nats at token 504. Each text after the parting is a correct greedy continuation of its own prefix. --no-learn-cost takes out what earlier requests taught the router, not the timings inside the request.

runs compared	--no-learn-cost	learned costs
alone, twice	token 12	token 504
alone, and with 10 side-lane guests	token 61	token 573

The checks below therefore hold arithmetic to bit equality, and text only where the shapes are fixed. A text check on the served engine (tools/api_text_check.py, tools/lane_check.py) can show a difference with no bug behind it. Such a difference is judged by the near-tie rule, as tools/verify_spec.py judges one.

12.4 Release identity

Every release candidate is also held to the release before it. The served configuration with the new flags off must produce bit-identical logits and state to the build it replaces (tools/flagoff_identity.py). That tool runs one fixed scenario: a prefill, chain verifies of 16 and 8 rows, a branching tree verify and a one-token step. The shapes are fixed, so it compares the arithmetic, not a router’s choices; given the same tree shapes, the candidate writes the bits the release writes. A speed change that alters one bit of that scenario does not ship. The text check sends 13 plain requests (the five benchmark workloads, streamed and not, one with thinking, one with tools, one under penalties, one seeded sampled request) to both builds and compares them byte for byte, pass for pass (tools/api_text_check.py). It runs the served router, so a difference there is read with the near-tie rule, and the same build’s second pass is the control. StairCut is the one documented exception: its verify shapes differ from the release before it, so at near ties its text can differ. It is held to plain greedy decoding instead, under the near-tie rule. Every release since has passed the identity gate against the one before it, on the raw path.

12.5 Sampling

A sampled request (temperature above 0) is exact in distribution. At each verified node the engine draws the target’s own token, and a drafted token survives exactly while the draw lands on it; when the draw lands elsewhere, that draw is the token. For a drafter that proposes one fixed token this is the standard rejection sampler, accepting with probability $\min(1, p/q)$, with no second draw. All sampling filters (temperature, top_p, top_k, min_p) run after the penalties, through one function in engine/sample.py, so drafted rows and plain rows see the same distribution.

A request with a seed draws with noise keyed by its position: the token at index $t$ is the argmax of $\log p_t + G_t$, where $G_t$ is Gumbel noise from a generator seeded by $(\textit{seed}, t)$. That is an exact sample from $p_t$, and it does not depend on how many draws the drafters made before, so the same seed gives the same text whatever the drafters proposed, up to the near-tie rule: another tree shape can move the last bits of $\log p_t$, and where two values of $\log p_t + G_t$ are that close, the draw can go the other way. tests/test_sample.py checks the samplers statistically on the CPU, and tools/sampled_dist.py checks the served engine’s sampled answers against the target sampled alone.

12.6 Caches

A cache restore gives back bytes, not an approximation. The prefix cache, the resident prefix and the disk tier take their snapshots on the prefill chunk grid, and a resumed request forwards the rest of its prompt on the same grid, so it computes the same arithmetic as a cold prefill on that grid. Tests on a small random model check that a resumed request’s logits and state equal a cold prefill’s bit for bit. The disk tier’s files carry a fingerprint of the weights, the engine’s code and the arithmetic flags, and a file of another fingerprint is never read. On the served engine, tools/disk_restore_check.py checked that a second process’s cold prefill equals the first’s bit for bit and that the first process’s disk files restore to it (measured, a build of the 0.4.0 series, a 10,701-token prompt).

Session resumes are one step weaker. A session resume continues a conversation from the state its last turn ended in, which verify rounds wrote rather than prefill chunks. That state produced the previous answer, but it is not the state a cold re-read would compute. It is held to the greedy gate (same tokens), not to bit equality. Resuming at the resident prefix’s end anchor is of that kind, and so is everything a prefill that resumed there writes. The engine tracks the lowest such row (taint): the state store takes no prefix snapshot of such a prompt, the disk tier writes nothing of it, and resuming at a grid anchor below it gives a cold grid prefill’s state again, bit for bit. Chapter 13 has the details.

An opt-in response cache (--response-cache) answers an identical greedy request from memory. It is off by default, so a benchmark can never be pointed at a dictionary.

12.7 What changes output on purpose

Some features change the text because the client or the operator asked for it. They are not exceptions to the promise: plain decoding with the same setting writes the same text.

• Penalties (repetition_penalty, presence_penalty, frequency_penalty) and no_repeat_ngram_size.
• Structured outputs and forced tool calls, which mask the tokens the constraint does not allow.
• A reasoning budget, and the conductor’s hidden closes and notes, which change what the model reads.

Two relaxed accept rules exist (--relax-tau, --relax-rank) that accept a drafted token when it is close to the target’s choice. They break the promise, exist only for measurement, and are off in every profile; the server prints a warning when either is set.

12.8 Flash-Next: no near-tie exception

Qwen3.8-Flash-Next is held to a stricter promise. Its verify kernels give every row the decode step’s bits, whatever the number of rows, so a greedy speculative run equals plain greedy decoding token for token, with no near-tie exception, and a seeded sampled request draws what plain sampled decoding draws. The load-time self-check proves the row property on the device before speculation turns on. Two places are held to less. A request resumed from an end anchor follows the 27B’s greedy gate, because the decode kernels wrote that state and the prefill kernels round differently. A think budget at low effort closes at a block boundary, so a speculative and a plain run close the reasoning a few tokens apart. Section 17.7 has the checks.

12.9 The checks

check	what it holds
tools/verify_spec.py	greedy output with no drafter, an adversarial drafter, a half-right drafter and the served drafters must match, up to the one-ulp rule
tools/flagoff_identity.py	the served configuration with new flags off is bit-identical to the build it replaces, on a fixed scenario of fixed shapes
tools/api_text_check.py	13 plain requests compared byte for byte with the current release, pass for pass; a near tie can part them, as it can two runs of one build
tools/cache_gate.py	warm and cold runs agree: bit-identical on the chunk grid, the same tokens for a session
tests/test_forward_tree.py, test_tree.py	tree verify equals chain verify on a tiny model, on the CPU
tests/test_cache.py, test_resident.py, test_disk_cache.py	restores are bit-exact, the disk tier’s through a reopened tier and after a failed restore
tests/test_end_anchor.py	resuming at the end of the last generation writes the same greedy tokens as one at the prompt’s line
tests/test_side_lane.py	the engine is bit for bit an uninterrupted run’s after a guest
tools/lane_check.py	a long request’s tokens with guests against runs alone: the same up to the first fork, and the fork a near tie (the runs took each other’s runner-up, top-2 gap within 0.125 nats, printed)
tests/test_sample.py, tools/sampled_dist.py	sampled output follows the target’s distribution