1What TandemLLM is

TandemLLM runs the models your agents need on your own hardware, behind one OpenAI-compatible address. A coding agent such as opencode or a chat client such as Open WebUI sends its request to that address. TandemLLM picks the model that answers, plans the turn, and watches the turn while it runs.

Several models work on one request. Qwen3.8-27B writes the answer. Two small drafters guess its next tokens and a lookup drafter offers text it has seen before, so one step of the big model commits 4.58 tokens on average on the benchmark row (measured, the paper). More models can run beside them in the same process. Laya’s decision models answer typed questions such as “is this turn code or prose”. A Qwen3.5-2B narrator writes one short line about what the big model is doing, and a Qwen3.5-0.8B shadow reader keeps its own copy of each coding session. A residency planner refuses any model that would not fit in memory (Section 6.8).

You get on your own machine what a cloud provider gives you. Clients use the OpenAI API with streaming and tool calls. A proxy routes each request by its model id, to this engine or to another backend: a vLLM or SGLang server, or a cloud API (Chapter 5). Every request leaves one row in a usage ledger with its tokens and its cost. Keys for cloud APIs sit in the config database as AES-256-GCM ciphertext.

An agent’s next turn does not read its whole prompt again. The resident prefix keeps the last prompt’s rows with a snapshot of the recurrent state every 1,024 tokens, and the end anchor keeps the end of the last answer. A parked session holds a second conversation while another one runs, and the state store keeps whole snapshots. Two caches live on the NVMe and survive a shutdown. The disk tier keeps prefixes that come back. The session spill writes live conversations at a stop and reads them back at the next start: a 41,587-token conversation came back in 0.94 s (measured, the 0.4.0 series). Last, the persistent suffix store keeps the n-gram tables the lookup drafter copies from. Chapter 13 describes every cache.

With memory notes on, the model keeps notes as files, one fact per file and at most 400 words. Notes about the user and about each project last across sessions. Session notes hold what is true for one conversation only (Chapter 24).

One dashboard watches this engine and the others. It reads the metrics pages of vLLM and SGLang every 15 s, and small plugins inside them report each request’s own times. For a cloud API it measures time to first token and decode speed at the proxy, and it shows the machines behind every backend (Chapter 28).

Requests run in parallel. With --parallel 4 the engine decodes up to four requests in the same rounds (Section 14.13), and the proxy relays any number of requests to other backends at once.

TandemLLM is not tied to one model. Its main model is Qwen3.8-27B, and a second, Qwen3.8-Flash-Next, runs on the same board in its place (Chapter 17). The small models are Qwen3.5 and Laya checkpoints, the proxy serves any model behind an OpenAI-compatible API, and Chapter 16 lists the seams where a main model plugs in.

1.1 The test environment

Every number in this book comes from one machine with one main model: an NVIDIA DGX Spark running Qwen3.8-27B. The Spark has a GB10 GPU and 128 GB of memory that the CPU and the GPU share. That is the project’s test hardware and its test model, and the project has tested nothing else so far. Its kernels’ tile tables and StairCut’s price tables come from that board. Much of the engine follows that model’s mix of 48 Gated DeltaNet layers and 16 attention layers (Chapter 8).

The project is growing past both. Other CUDA GPUs run the engine with the one Spark-only kernel turned off; their speed has no measurement yet. The proxy and the dashboard run on any machine with Python 3.11. The roadmap (Chapter 35) names the next models and boards. When this book says “the board” or “the model” without a name, it means the DGX Spark and Qwen3.8-27B.

1.2 Two levers

A decode step of Qwen3.8-27B reads about 27 GB of weights in the vendor’s FP8 format. The DGX Spark reads its memory at 273 GB/s on paper and about 240 GB/s in practice, so every step costs at least 100 ms before any code runs (arithmetic). That is a ceiling of 10 tokens a second for one request. The engine is built to get under that floor while writing the model’s own greedy text.

With the published NVFP4 weights and StairCut on, it writes 49.89 tokens a second for one request on the benchmark row, against 13.69 for plain greedy decoding on the same box (measured, the StairCut paper, 50 requests each). The rest of this section says how.

The engine pulls two levers on the same cost.

The first lever cuts the bytes a step reads. The served weights store every projection in NVFP4, four bits a weight with a scale per 16 weights, and the output head in FP8. A step then reads 15.01 GB instead of 26.93 GB (arithmetic, from the checkpoint’s tensor sizes). Chapter 9 shows how the 4-bit copies were made and what they cost in quality.

The second lever makes each step produce several tokens. Small drafters guess the next tokens, and the big model checks all the guesses in one forward pass. A guess the model agrees with is a free token. At the first wrong guess the model puts in its own token, so nothing a drafter proposes reaches the output unless the model would have written it. On the benchmark row a round takes 93.4 ms and commits 4.58 tokens (measured, the paper). That is where the name comes from: two models working in tandem, one guessing and one checking.

The verify is cheap because of the memory floor. A forward pass over 16 tree rows reads the same weights as a pass over one row, so on the published weights at 1k tokens of context it takes 74.4 ms against 75.5 ms for 8 rows (measured, ops/stair-tables-nvfp4.json). Extra guesses in the same pass are nearly free until the row count crosses a kernel’s tile boundary. StairCut, the router that sizes each round’s draft tree, is built on that staircase (Chapter 11).

1.3 What is in the repository

TandemLLM is several programs that share one repository and one release number.

The engine

engine/, tools/ (kernels) and server/app.py: the model, its kernels, the drafters and routers, the caches, and an OpenAI-compatible HTTP server built on Python’s standard library. It needs a CUDA GPU (measured on the DGX Spark only), PyTorch, Triton and the weights. The small models run on its own code: the Qwen3.5 narrator and shadow reader on its Qwen forward, Laya’s decision models in the same process.

The conductor

server/conductor/: a layer in front of the model. It classifies each turn, sets the thinking effort, adds hidden notes and skills, watches the generation for loops and broken tool calls, keeps memory notes and shows the client what it does. It is off unless a configuration directory is given.

The dashboard

dashboard/: a web page the server serves at /dashboard/. It shows what each request is doing now, a year of usage, the log, a playground, the conductor’s settings, the backends and the machine.

The proxy

proxy/: the same dashboard in front of other OpenAI-compatible engines and cloud APIs, routed by model id. It installs with pip and needs no GPU.

The plugins

plugins/: two small packages that run inside vLLM and SGLang and report each request’s queue, prefill and decode times to the proxy.

The installer

install.sh: one command that sets up the served profile on the test board, and best effort on another CUDA GPU. With --setup it starts setup mode instead: the Setup Guide in the browser, or ops/use-profile.sh from a shell, then downloads and starts any weight profile (Section 3.3).

1.4 What runs today

The roadmap (Chapter 35) widens each of these, one step at a time.

• Models: Qwen3.8-27B is the main model the engine serves, a hybrid of 48 Gated DeltaNet layers and 16 attention layers; Chapter 8 explains why much of the engine follows that mix. Qwen3.8-Flash-Next, a mixture of experts on the same hybrid, is the second, served from turboderp’s EXL3 packs one model at a time (Chapter 17). Beside it run the two fine-tuned DFlash2 drafters, the lookup drafter and, when you turn them on, Laya, the narrator and the shadow reader. Behind the proxy, any model a backend serves.
• Hardware: the DGX Spark is the only board with measured kernels and prices (Section 1.1). Other CUDA GPUs are best effort.
• Requests: --parallel N gives the engine N sequences, and requests that decode at the same time share one draft pass and one verify pass a round (Section 14.13). The shipped profile keeps one sequence, and the measurements go up to four. With one sequence a request that finds the engine busy waits in a bounded queue, and a short request is served at a safe point inside the running one (the side lane, Section 14.11). The proxy relays any number of requests to other backends at once (Chapter 25).

1.5 Two rules

Every change to the engine keeps two rules, and the release gate checks both before anything ships.

The first rule is agreement with greedy decoding. For every greedy request the engine’s output tokens equal what plain decoding on the same weights writes, one token at a time, up to the first near tie where the two best tokens are within one unit in the last place. There the batched verify can round the other way. The paper measured how often: of 30 prompts, 17 outputs equal plain greedy decoding to the end of the answer, and 13 diverge at a near tie with a gap of 0.04 to 0.76 ulp (measured). Chapter 12 has the full statement and the checks.

The second rule is that no measured number gets worse. A candidate runs against the current release in the same hour on the same box, alternated, and ships only if no figure resolves worse: not the mean, the median, the 90th percentile or the maximum, not the time to first token, and not tokens or milliseconds per round. Chapter 15 describes the method and the traps it avoids.

1.6 The headline numbers

Table 1.1 collects the numbers that describe the served engine on the test board. Each comes from the StairCut paper’s measurements on the published NVFP4 weights, with the lookup drafter’s persistent store off.

These are single-request numbers. Parallel requests on the engine are measured in Section 14.13. Long prompts are measured separately in Chapter 13. No task benchmark suite has been run, so this manual makes no claim about task accuracy.

1.7 The paper

Khaled Bakeer’s paper describes the engine’s decoding method:

Khaled Bakeer. StairCut: sizing speculative draft trees on a measured verification-cost staircase. Zenodo, 29 September 2026. doi:10.5281/zenodo.23045275.

The paper measured the release tagged v0.2.0-staircut (commit c111fa7) with the published NVFP4 weights on the test board. It backs these parts of this book:

• the draft tree merged from the block drafter and the lookup drafter, and StairCut, which cuts it each round on the measured verify-cost staircase (Chapter 11);
• the commit of an accepted path through the Gated DeltaNet layers in closed form, and the NVFP4 kernels with a fixed reduction order (Chapter 10);
• the NVFP4 quantisation from Qwen’s BF16 release and its held-out loss (Chapter 9);
• the speeds of Table 1.1: 49.89 tok/s for StairCut, 47.13 and 44.80 at fixed blocks of 8 and 16, 13.69 for plain greedy decoding, 50 requests each on the benchmark row, and the teacher-forced bench of 25 workloads;
• the agreement with plain greedy decoding: of 30 outputs, 17 identical to the end of the answer and 13 that part at a near tie (Chapter 12);
• the time to first token of a 32k to 128k-token prompt from cold, and again when it resumes from the anchors (Chapter 13).

What came after that release is not in the paper: image input, the conductor, the caches on the NVMe, memory notes, parallel requests, the proxy and the plugins. Their numbers come from the release series, and the text labels them that way. The release gate holds every later release to the paper’s single-request numbers and to the agreement rule. CITATION.cff in the repository carries the citation; in BibTeX:

@misc{bakeer2026staircut,
  author    = {Bakeer, Khaled},
  title     = {StairCut: sizing speculative draft trees on a measured
               verification-cost staircase},
  year      = {2026},
  publisher = {Zenodo},
  doi       = {10.5281/zenodo.23045275},
  url       = {https://doi.org/10.5281/zenodo.23045275}
}

1.8 Credits

Qwen made the model weights and their reference implementation. The base block drafter is DFlash2 for Qwen3.8-27B, released by its vendor under Apache-2.0. The engine runs on PyTorch and Triton over CUDA, and its own kernels are written in Triton and CUDA. The lookup corpus that the installer builds comes from public, permissively licensed text. The decision models the conductor can ask are Laya’s published checkpoints (Apache-2.0).