15Measurement

Speed claims for inference engines are easy to inflate by accident. The suffix store once held the benchmark’s own prompts and answers, and it added about 21 % to the benchmark row without the engine getting any faster. This chapter describes how TandemLLM measures itself, from the benchmark row and the teacher-forced bench to the release gate, and the traps each rule answers.

15.1 The row

The benchmark row is serve-single-i256-o256-v1: one request at a time, 256 prompt tokens, 256 generated tokens, thinking off, temperature 0, seed 42, 50 requests after 3 warm-ups. Every configuration runs with the same runner and the same prompt set. Each request’s speed comes from its own record:

The first token belongs to the prefill, so it is subtracted, and the decode time is the request’s wall clock after its first token. A row reports the mean, p50, p90 and maximum over the 50 requests, with the median time to first token.

Two readings of a row are easy to get wrong. The gaps between tokens say nothing about round time: a verified round writes several tokens into the socket at once, so half the gaps are microseconds and the other half a whole round. And the mean is not the median: some prompts ask the model to repeat text already in the context, the drafters commit 15 tokens a round on them, and that lifts the mean. The median reflects fresh text.

15.2 Two views of every row

A row runs in two views of the suffix store. Store off: the engine’s persistent suffix store is off, so nothing the engine has seen before can help it. Clean store: the store is on but holds no text of the benchmark. A row with the benchmark’s own text in the store measures the store, not the engine. Benchmarks that read a store of real traffic run with --suffix-store-readonly, so they cannot write themselves into it.

With the conductor on, a benchmark also needs the raw path: X-Tandem: raw, or the locked bench profile. tools/row3.py --no-server refuses a server that lists TandemLLM, because a runner that sends no headers would measure the router, the effort cap and the notes along with the engine.

15.3 Noise, and what a row can decide

Two server processes of the same configuration can differ by several per cent. StairCut learns its costs and copy rates online, so two runs can make different decisions on the same prompts. tools/row3.py runs N rows per configuration through one server, recomputes each from its own records, and reports the median with the spread of the runs. --compare calls a difference resolved only when it is larger than the noise and the run ranges of the two sides do not overlap. A change worth less than about 3 % of the median cannot be settled by the row at any number of repeats.

Speed is a quotient: tokens a round over time a round. A change can trade one factor for the other and leave the quotient flat. The served loop therefore counts its own rounds, and every row reports tokens per round and milliseconds per round next to tok/s. Both are gated: a kernel change must not move tokens per round, and a drafter change must move it up.

A server run can hold the costs still. --no-learn-cost keeps the router’s served price table, so what earlier requests took no longer moves the choices. The router still times the blocks inside the request, so two runs can still choose differently and, at a near tie, write different text (Section 12.3). With learning on, one prompt took 366, 468 and 504 blocks for the same 2,000 greedy tokens in three runs, and the runs spread 37 %; with the flag, six runs wrote one text in 514 blocks each and spread 0.19 % (measured, the 0.4.0 series). An A/B on a server therefore uses the flag, two warm-ups an arm, and compares only runs that wrote the same text.

Every report names the code it measured with code_sha256, a hash over every file of the four code folders. The same commit hashes the same everywhere, so a report can be matched to a commit.

15.4 Same hour, both sides

Nobody compares a candidate with a number from another day. It runs against the current release in the same session on the same box, alternated: base, candidate, base, candidate. A board drifts over hours (its temperature, its memory, whatever else runs), and alternation spreads that drift over both sides. The adoption rule is “never worse”: a candidate ships only if nothing resolves worse against the same-hour base, in both store views. Not the mean, p50, p90 or maximum, not the time to first token, and not tokens or milliseconds per round.

15.5 The block A/B

Most kernel changes are worth 0.5 to 3 ms of a round of about 93 ms, below what the row can resolve. For a change whose output is bit-identical, only time can differ. tools/block_ab.py loads the served engine once, switches each state’s flags inside the process, warms each state, freezes the router’s learned costs, and alternates base and candidate over three workloads for N pairs. Every run’s tokens must equal the base’s first run, and the rule is applied to milliseconds per round. Two states of the same code read “not resolved”, with a spread of 0.35 to 0.87 %.

15.6 The release gate

ops/gate.sh is the whole protocol in one command, run on the board inside a hold. It stops at the first failing step:

1. every CPU test, tests/test_*.py, in a clean environment;
2. every GPU test, tests/gpu/test_*.py;
3. identity: the served configuration with the new flags off is bit-identical to the build it replaces, on a fixed scenario of fixed shapes (tools/flagoff_identity.py);
4. agreement: tools/verify_spec.py with the served flags and the candidate’s, under the near-tie rule;
5. optionally, the block A/B;
6. rows: two store-off rows and one clean-store row with the candidate’s flags;
7. compare: tools/gatecheck.py against the base reports, with the never-worse rule.

It writes a dated report with every command and its output. Rows are not enough on their own: a release candidate once passed every row and broke a real client’s request shape. Before a deploy the gate is followed by tools/client_smoke.py (the request shapes real clients send, through the official openai client: tools validated against their schemas, every tool_choice mode, streamed tool-call deltas, prompts from 8k to over 100k tokens), by tools/api_text_check.py, and by a real agent session whose tool calls must all go through.

15.7 The teacher-forced bench

A change of the verified tree’s shape moves near ties, so two configurations rarely write the same text on the same prompt, and their rows measure different texts. tools/forced_bench.py removes that. Each workload has one fixed reference text, and every configuration drafts and verifies with the real kernels at the real context but accepts along the reference, so all of them commit the same tokens and differ only in rounds and time per round. The paper’s run covers 25 workloads: prose, chat, code, twelve prompts of the benchmark row, three edit tasks, a quotation and a copy, and fresh text, a copy and a code edit at 32k tokens of context. StairCut averages 90.45 tok/s against 81.95 for the fixed block 16 and 59.31 for the fixed block 8 (Figure 11.3).

15.8 Traps

Each of these cost a wrong number once.

• A standalone kernel timing is a ranking, never a budget. The same configuration read 0.254 ms in one process and 0.627 ms in another.
• The first configuration in a fresh process pays for autotuning. It once read a verify of 260.2 ms against 115.5 ms for every later one, and the policy under test “won” by 35 %.
• A bandwidth-bound engine measured beside another engine measures the other engine. Two engines halve the bandwidth each gets. Take the board’s lock.
• A speculative decoder judged on a short generation hides bugs. One drafter bug read 81 % acceptance at 48 tokens and 24 % at 128.
• Acceptance with the reasoning block open differs from acceptance with it closed. Measure in the serving regime.
• A kernel probe whose working set fits in cache measures the cache. 266 GB/s on a 273 GB/s board is not a real reading.
• Two runs that agree to six decimals have not agreed; they are the same run. Once, a test server had bound its port beside another.
• A quiet log is not a hung process: Python block-buffers into a pipe, so use python -u.

15.9 Measuring Flash-Next

Flash-Next’s numbers in this manual come from tools of their own (tools/fn_*.py), each answering one question, and every one ran under the board’s lock with one model loaded. Most are single runs of one request, labelled so where they appear; none is a benchmark row yet.

tool	the question it answers
fn_server_check.py --compare	do 20 request shapes give the same text, tool calls and logprobs with and without speculation; --sampled: do seeded sampled answers
fn_spec_check.py	does speculative greedy equal plain greedy in the engine, with the state bit-equal after every run
fn_fit_run.py	does a pack hold its largest window: one long request, MemAvailable at 10 Hz, every limit knob the server reports
fn_cache_check.py	does a request restored from each cache tier answer as on a cold server
fn_resume_check.py, fn_guest_check.py	does a resumed prefill equal a cold one; does a short guest request leave the resident conversation’s rows intact
fn_lookup_ab.py	one greedy request a class through a server, for an A/B of the drafters
fn_memory_check.py	memory after the load, the prefill workspace, the table’s resident bytes under each policy
fn_exl3_xcheck.py, fn_refcheck.py	our reading of a pack against the converter’s; our forward against transformers’

Every speculative rate in Chapter 17 comes with the statement that its texts equalled plain decoding, because a rate whose text changed measures another answer. The runs by context length started a fresh server and report its second round, so the first round’s one-time graph captures do not count in them. What Flash-Next has not had yet is the release gate: no row of serve-single-i256-o256-v1, no block A/B and no task suite.

15.10 Limits

The published numbers come from the test board and the test model (Section 1.1), one request at a time. Flash-Next’s come from single requests on the same board (Section 15.9). Long prompts are measured separately (Chapter 13), and parallel requests in single runs with tools/parallel_bench.py (Section 14.13). The quality gates are held-out loss, confident argmax agreement and free generation; no task benchmark suite has been run, so this manual makes no claim about task accuracy.