11Drafters, StairCut and tree verify

A verify of 16 tree rows costs about the same as a verify of 8. On the published NVFP4 set at 1k tokens of context, a tree verify with its commit takes 75.5 ms at 8 rows, 74.4 ms at 16, 83.1 ms at 24 and 87.8 ms at 32 (measured, ops/stair-tables-nvfp4.json, median of 8). That flat stretch is why this engine speculates: the model’s forward pass is bound by reading weights, and extra rows in the same pass are nearly free. TandemLLM therefore lets cheap drafters guess many tokens and checks them all at once. This chapter covers the drafters, how StairCut decides what to spend each round, how the guesses become one tree, and how the tree is verified through recurrent layers.

11.1 The round

Each round has four steps (Figure 11.1).

1. Draft. The block drafter proposes a block of tokens, and the lookup drafter proposes continuations it has seen before. The router merges them into one draft tree of at most 32 nodes.
2. Verify. One forward pass of the target model runs over the whole tree and reads the model’s own next token at every node.
3. Accept. Walk down from the root. At each node take the model’s token and move to the child that carries it. Stop where no child carries it. The model’s token at the stopping point is kept too, so a round yields at least one token.
4. Commit. Keep the KV rows and the recurrent state of the accepted path, and drop the rest.

On the benchmark row a StairCut round takes 93.4 ms and commits 4.58 tokens (measured, the paper).

11.2 The block drafter

engine/drafters/dflash2.py runs DFlash2, a small drafter that proposes a whole block of tokens at once. It has five decoder layers of its own (hidden size 5,120) and reads five hidden states of the target, the residual stream entering layers 5, 19, 33, 47 and 61, fused into one input. It has no embedding and no head of its own: it borrows the target’s. Inside a block every row attends to every other row, which is what lets it guess 15 tokens in one pass instead of 15 chained steps.

The engine holds two fine-tunes of the vendor’s drafter, trained on the target’s own output and published on Hugging Face: one trained at block 8 (the narrow drafter: it proposes 7 tokens a block) and one at block 16 (the wide drafter, 15 tokens). StairCut, the served behaviour, drafts with the block 16 one only. The served profile loads the block 8 one too and releases it at the first round; it drafts only with StairCut off. A draft moves about 6 GB (3.6 GB of drafter layers plus one read of the head for all rows), about 25 ms.

The drafter’s head gives the top 16 candidates per slot, and a small bigram selector scores how each candidate follows the one before. That is a lattice of 15 slots by 16 candidates. The greedy path through it becomes the spine of the tree, and the next-best candidates become branches.

11.3 The lookup drafter

engine/drafters/ngram.py copies text instead of predicting it. It looks for the longest match of the last few tokens in three places:

• the request itself, prompt and output so far (orders 2 to 8);
• a corpus of public text, memory-mapped with a suffix array;
• the persistent suffix store, a log of what this engine has read and written before (Section 13.10).

Every continuation found is a vote, and the votes form a tree whose node scores are path probabilities. A lookup costs about 0.2 ms against about 25 ms for the block drafter. On fresh prose it rarely fires. On an edit of a file already in the context, or on a quotation, it can commit 15 tokens in a round. It fires only when the tree it would propose is worth its verify cost, because a drafter that proposes on every step and is usually wrong costs the round. After two wide rounds in a row that commit their whole width, the lookup drafter may propose one deep chain of up to 32 rows (QWEN38_DEEP).

11.4 StairCut

StairCut is the served behaviour: QWEN38_LEN_SWITCH=1 and QWEN38_LEN_MODE=wide, off in the code and on in ops/serve.env. Every round drafts with the wide drafter and builds four candidates of up to 31 nodes: its lattice tree, its chain, the lookup tree, and their merge. Each candidate is cut to the node count that maximises its expected committed tokens per millisecond,

and the round runs the best of the four.

A node’s $q$ is its path probability from the drafter’s lattice (read at temperature 1.4), or the lookup’s vote share, scaled by a calibration the router learns online and capped at 1. Nodes are admitted in the order of the drafter’s own score, each with the ancestors it needs. The price of a tree is verify plus draft plus commit; a chain pays the rollback instead of the commit, weighted by the chance it is rejected. The verify price is a staircase, not a line, so the router tries every count instead of a search that assumes a smooth cost. A cut ends at 7, 15, 23 or 31 nodes (8, 16, 24 or 32 rows), or at the whole candidate when it is smaller.

QWEN38_STAIR_TABLES names the JSON file of verify prices by context length, chain and tree apart (tools/verify_curve.py measures them). Figure 11.2 draws the served file. At 32k a 24-row tree costs about 16 ms more than a 16-row one, against 9 ms at 1k, and a tree costs 2 to 4 ms more than a chain of the same rows. QWEN38_STAIR_RHO names a prior for the lookup’s continuation rate on 8-token matches (ops/lookup-rho.json), which lets a long verbatim copy keep its whole line; the router updates it online against every committed block. QWEN38_STAIR_SKIP=1 decides on the same prices whether to draft the head at all once a copy run has started: on a copy round the lookup line alone often wins, and at 32k a draft costs more than at 1k.

The draft cost is the drafter’s own measured time at the current context length, learned online per context class. What the router learns (the draft cost per context class, the calibration, the copy counts) belongs to the router object, and the server keeps one router for its whole life, so what one request teaches reaches the next. --no-learn-cost keeps the served price table instead, so what earlier requests took no longer moves a block’s choice and a speed measurement repeats much more closely (Chapter 15). The timings inside the request still move it, so two runs can still part at a near tie (Section 12.3).

With the node count calculated, the wide drafter’s lattice cut to 16 nodes is a bush near the root on fresh text and a long line on copies, and the narrow drafter has nothing left to add. The router releases it at the first round.

11.5 Longer contexts

Every token of context adds 64 KB of KV rows that each round reads, and the drafter reads the whole context too, so a round costs more as the conversation grows. The verify itself shows it: a 16-row tree with its commit takes 74.4 ms at 1k tokens of context and 81.5 ms at 16k, and 88.1 ms at 32k (measured, the price table). Figure 11.4 puts the two measured decode rates the public record holds beside that curve and beside the bandwidth bound of plain decoding.

Two points are all the public record holds for decode speed against prompt length on the published weights, and they come from two builds of the same series. A sweep over prompt lengths, under the rules of Chapter 15, is not published yet.

11.6 The length router

Before StairCut, the served profile picked the block width once per request, and with the StairCut flags off engine/lenrouter.py still does. It runs four wide rounds first. A wide round prices both widths at once, because a narrow block is a prefix of a wide one and the target’s token at row $i$ does not depend on rows after $i$. Where that says the narrow width still has room, it runs up to four narrow probes. Then it decides, keeps the decision for the rest of the request, and releases the losing drafter (--drop-idle), which then stops syncing its cache. It decides once because switching costs acceptance: a drafter keeps its own cache of the context, and a switched-in drafter starts behind. --len-fixed 8 or 16 pins one width; those are the fixed baselines the paper compares StairCut with.

11.7 Building the tree

engine/router.py merges the block drafter’s lattice and the lookup tree into one DraftTree (engine/tree.py), where a shared prefix becomes a shared node. With StairCut on, the merge is built at up to 31 nodes and the staircase cut sets its size. With StairCut off, the router prunes it to a node budget: a node stays only while its expected gain in accepted tokens pays for its share of the verify cost, priced on a verify curve (QWEN38_TREE_MS, or a price table). The budget is 16 nodes for either drafter, and the wide drafter gets 24 once the request has committed 32 tokens, so a short answer never pays for branches it cannot use.

The router’s cost estimates are learned in the loop. It scores the lookup tree against the tokens the target actually wrote on every round, chosen or not, so its calibration settles before the lookup drafter has had a turn.

11.8 Tree verify through recurrent layers

Verifying a tree through attention layers is known work: give each node an attention mask of its ancestors and a position equal to its depth. The 48 Gated DeltaNet layers are harder. Their state after node $t$ is the state after $t$’s parent, updated by $t$, and a plain left-to-right walk overwrites the parent’s state before $t$’s sibling needs it.

engine/tree.py stores a tree in depth-first pre-order: every node’s parent has a lower index, and every subtree is a contiguous range (Figure 11.5). In that order the ancestor relation is part of the strict lower triangle, and every prefix operation of a chain verify has an exact tree counterpart.

chain	tree
causal mask	the committed prefix plus the node’s ancestors
position = index	position = depth, so two siblings share a position
the convolution slides over neighbours	it gathers each node’s 3 nearest ancestors
cumulative gate along the sequence	the sum of gates along the node’s path
the delta rule’s lower-triangular solve	the same solve on the ancestor relation
accept a prefix, roll back to a length	accept a path, commit it by a gather

The state after any accepted path can be written from quantities the verify already computed. Let $S_0$ be the state before the round, and for each node $t$ on the path let $k_t$ be its key, $u_t$ its pseudo-value and $g_t$ its cumulative gate. Then

$u_t$ depends only on $t$’s own path, because the inverse of an ancestor-masked matrix is ancestor-masked too. The formula holds for every root-to-node path, not only for prefixes. The commit is therefore one read of the state and one small matrix product per layer, and it reads no weights. For a chain the same identity is the rollback: keeping the first $n$ tokens of a block is the path of length $n$. The KV side is simpler: the cache holds every node at its depth-first slot, and a commit gathers the accepted rows down into place.

11.9 Sampling

Greedy requests (temperature 0) take the path above. A sampled request stays exact in distribution: the target draws its own token at each node, and a drafted token survives exactly while the draw lands on it. For a drafter that proposes one fixed token this is the standard rejection sampler, with no second draw. With --sampled-tree det (served) a sampled request keeps the tree. A request with a seed draws with noise keyed by position (the Gumbel-max trick), so the same seed gives the same text whatever the drafters and the router did, up to the same near-tie rule as greedy decoding. Chapter 12 covers both.

11.10 Reasoning

The model thinks inside <think> tags. A reasoning budget closes the block when it is spent: the engine appends the vendor’s closing sentence and </think>, runs those tokens through the model at their real positions, and the model answers from there. This changes the answer, by design. The budget comes from the server’s --think-budget, from the client in any of the shapes clients send (Section 14.3), or from the conductor’s effort level. With the conductor on, the closing phrase is hidden from the client, which receives </think> and the answer; the conductor’s loop guards close a reasoning block that goes in circles (Chapter 24).

11.11 The same method on Flash-Next

Flash-Next drafts with the multi-token-prediction layer its checkpoint ships, and its StairCut is written once for every model (engine/methods/staircut.py). Each round the MTP head drafts a chain of 4 tokens with its top two a depth, from one captured graph. The cut keeps the spine’s prefix and, with siblings, the second choices that maximise

where $q$ comes from ProbDeciles, the head’s probability mapped to the acceptance measured per tenth of probability and learned from every verified round. With QWEN38_FN_LOOKUP=1 the lookup’s best continuation joins the same tree: where its tokens match the head’s chain the two share nodes, where they differ it hangs under the last shared node, and the cut admits nodes best-first up to 32 rows. A round still runs one verify, and on a copy the lookup lifted the rate from 102.3 to 132.9 tok/s (Table 17.3).

$V(\text{rows})$ comes from PriceModel: a prior from a price file in the format of the 27B’s stair tables, ops/prices/<model>/<profile>.<device>.json, followed online and never cheaper for more rows. The device is dgx-spark for the GB10 (engine/methods/device.py); a profile without a file for its device learns every price from the first rounds. On this model the verify is a ramp, not a staircase: every row can route to other experts, and at about 1k of context 2 rows cost 27.8 ms and 16 rows 68.8 ms (measured, learned online).

The cut also decides whether to draft at all. Prices, the plain step’s cost and a drafted round’s cost a token are kept per context class, by wall time; a class whose drafts cost more a token than its plain steps takes the plain step, with probe rounds at a doubling interval, and a turn drafts only when the saving pays for the prompt rows the head has not read yet. Section 17.6 has the rule and the runs that set it. A verified row on Flash-Next is the decode step’s row bit for bit, so none of this moves the text.

11.12 Knobs

knob	what
--drafter lenrouter, --dflash2-ckpt, --dflash2-ckpt16	the block drafters (both loaded; StairCut drafts with block 16)
--narrow-arm on|off	build the block 8 drafter at all
QWEN38_LEN_SWITCH=1, QWEN38_LEN_MODE=wide	StairCut
QWEN38_STAIR_TABLES, _RHO, _SKIP	its prices, the lookup prior, the head-skip rule
--len-fixed 8|16, --len-latch, --drop-idle	with StairCut off: pin a width, decide once a request, release the loser
--tree, --budget, QWEN38_TREE_NODES*, QWEN38_TREE_MS	the tree and its node budgets (StairCut off)
--corpus, --suffix-store	the lookup drafter’s corpus and persistent store
QWEN38_DEEP, QWEN38_DEEP_AFTER	the deep chain after long accepted paths
--sampled-tree det|mixed, QWEN38_DRAFT_TEMP	sampled requests on the tree, and the drafter’s temperature
--learn-cost, --no-learn-cost	learn verify and draft costs online, or keep the price table
--think-budget, --reasoning-effort	the server’s reasoning defaults
--drafter mtp, QWEN38_FN_DRAFT_HOT	Flash-Next: its MTP layer as the drafter, through a head of the 32,768 most frequent ids
QWEN38_FN_LOOKUP	Flash-Next: the lookup in the MTP round’s tree (on in its profiles)
ops/prices/<model>/<profile>.<device>.json	the shared StairCut’s price file