2One request, end to end

This chapter follows one chat completion from a coding agent through every part of TandemLLM. Each step names the chapter that explains it. Figure 2.1 is the map.

2.1 The client sends a request

The client is any program that speaks the OpenAI chat completions API: the OpenAI SDK, opencode, the Hermes agent, Open WebUI or curl. It sends POST /v1/chat/completions with its messages, its tools and its options, usually streamed. The engine’s server reads every OpenAI field it can honour, accepts the ones that have no effect here, and refuses the rest with a 400 that names the field (Section 14.1).

2.2 The proxy routes it

When the server runs with --backends, or when the standalone proxy (tandem-dash) is in front, the request is routed by its model field. A model of another backend (vLLM, SGLang, a cloud API) is relayed there: the answer’s bytes go back to the client unchanged, and the proxy writes one ledger row with the tokens, the time to first token, the decode rate and the cost. A request for TandemLLM stays in the engine’s server. Chapter 4 sets this up and Chapter 26 covers the conductor in front of other backends.

2.3 The conductor plans the turn

With a conductor configuration the server serves one model name, TandemLLM: the weights with a conductor in front. Before the request queues, the conductor resolves a profile (by header, by client, or the default), and its router decides what kind of task the turn is. A tool-result turn keeps the session’s class; only a new user message asks a classifier, and the default classifier is Qwen itself under a choice grammar. The class sets the thinking budget and adds a hidden note at the end of the last message: the effort sentence, rules and packages (skill texts). Later turns replay that note at the same token position, so the cached prefix keeps matching. Chapter 21 explains the router and Chapter 22 the packages.

A client’s side request, such as opencode’s title call, is recognised by its shape and served small: no classifier, no note, at most 64 output tokens for a title.

2.4 Admission decides when it runs

The engine holds one sequence unless --parallel N gives it more (Section 14.13). A request that finds it busy waits in a bounded queue (8 by default). A streamed request that waits more than 10 s gets its stream opened and an SSE comment every 10 s, with an estimate of the wait. A short request (at most 2,048 prompt tokens and 64 output tokens, thinking off) does not wait at all: the running generation serves it at its next safe point, between two decode blocks or two prefill chunks, and gets its state back byte for byte. Before every prefill a memory gate checks that the prompt’s workspace fits under the declared peak, giving caches back cheapest first. Chapter 14 covers the queue, the side lane and the memory gate.

2.5 The caches decide where the prefill starts

A prefill is the most expensive thing the engine does: 208.3 s for a 128k-token prompt from cold (measured, the paper). Before it prefills, the engine looks for state it already has. In an agent session each turn’s prompt is the previous prompt plus the answer plus a tool result, and the resident prefix still holds the previous turn’s rows in the engine’s own buffers, with recurrent-state anchors every 1,024 tokens and an end anchor at the end of the last generation. The next turn resumes there and forwards only what is new. Further down are a state store of snapshots, a disk tier on the NVMe for prefixes that come back, and a session spill that keeps whole conversations through a restart. Chapter 13 describes them all.

2.6 The decode loop

Then rounds run until the answer ends. In each round the block drafter proposes up to 15 tokens and the lookup drafter proposes continuations it has seen before; StairCut merges them into one tree and cuts it to the size that maximises expected tokens per millisecond on the measured verify cost. One forward pass of the target model verifies the whole tree through all 64 layers. The engine walks the tree with the model’s own tokens, keeps the longest agreeing path plus one token of the model’s own, and commits that path’s KV rows and recurrent state. Chapter 11 explains the round and Chapter 10 the kernels that make the verify cheap.

2.7 The monitor watches the output

While the model writes, the conductor’s session monitor reads the output at four safe points: before the prefill, after every committed block, at the end token, and when the answer’s tool calls are complete. It can close a reasoning block that loops, continue a turn that ended empty, repair or regenerate a malformed tool call, and stop a tool call whose arguments repeat. Everything it adds is hidden from the client and never counted in usage. Chapter 23 lists its tasks.

2.8 The stream goes back

Committed tokens stream to the client as they are verified, several at a time. A tool call streams as argument deltas while the model writes it, with values typed by the request’s JSON schema. With a profile’s status lines on, the client also sees one short line for each step the conductor takes (“› reading 20,745 tokens, about 18 s left”), which the engine strips again from the next prompt (Chapter 24).

2.9 After the request

The request leaves one log line (counts, finish reason, rate, never text) and one row in the usage ledger. The conductor writes its decisions to a journal, a transcript block for a request with tools, and, when memory notes are on, any <tandem_memory> block the model wrote. When the session goes quiet, a background writer may write the session’s facts as notes (Chapter 25). The dashboard shows all of it live (Chapter 27).