13Caches

A long prose prompt sent twice took 44.4, 93.7 and 208.3 s to its first token at 32k, 64k and 128k tokens from cold, and 0.3, 0.6 and 1.1 s the second time, resumed from the resident prefix (measured, the paper, one request each). A prefill is the most expensive thing the engine does, and the caches exist so that it runs once per token of a conversation instead of once per turn. Every cache here restores exact state, and none of them is meant to change the output (Chapter 12).

13.1 What a state is

To continue a sequence at token $L$ the engine needs three things:

• the KV rows of the 16 attention layers for tokens 0 to $L$, 64 KB a token;
• the 48 recurrent states and their convolution tails, about 147 MiB whatever $L$ is;
• the drafters’ own position-indexed rows, about 40 KB a token for both drafters.

The recurrent part is the hard one. A KV row depends only on the tokens up to it, so rows written for an earlier request stay valid. The recurrent state has absorbed every token and cannot be rewound, so a cache must have taken a copy at $L$.

The caches know no model. Every tier reads a sequence through the engine’s adapter (eng.cache_state(), engine/models/state.py): its recurrent sections by name, copied whole, and its position-indexed views with their token axis, sliced to $L$. For the 27B the sections are gdn.S and gdn.conv and the views K and V. Flash-Next has more parts, and the tiers carry all of them with no code of their own for it:

• sections: 36 GDN states and convolution tails, the n-gram embedding’s convolution and token context, and mtp.h_last, the last committed row’s streams that its draft layer reads next, 115,662,864 bytes (110.3 MiB) whatever $L$ is;
• views: K, V and the sparse-attention indexer’s keys of its 12 attention layers, and the draft layer’s own, 29,952 bytes a token.

Flash-Next’s grid is 4,096 tokens, its prefill chunk, so a resumed prefill runs the chunks a cold one runs and restores from grid anchors, the state store, the disk tier and the spill equal a cold prefill bit for bit. A disk anchor of format 2 names its sections; the 27B’s older files of format 1 are read as gdn.S and gdn.conv. Section 17.8 has the measured restores.

13.2 The tiers

Figure 13.2 shows the caches in the order the engine looks them up before a prefill. A deeper tier is taken only when it starts later than every tier above it, and the NVMe tiers only when their read costs less than the prefill it saves.

cache	what it keeps	where	default
resident prefix	the KV the last prefill and generation wrote, in place, with anchors every 1,024 tokens and one at the end of the last generation	device	on
state store	whole state snapshots at prompt boundaries (session cache) and chunk boundaries (prefix cache)	device	on, 2 GiB
persistent suffix store	token ids the engine read and wrote, with a suffix array	disk	on, 192 MiB
response cache	the tokens of an identical greedy request	memory	off
disk tier	rows and anchors of prefixes prefilled more than once, and of a session’s first long request	NVMe	on, 50 GB
session spill	whole live conversations, at a planned stop and before an eviction	NVMe	on, 40 GB

13.3 One conversation through the tiers

Figure 13.3 follows one opencode session through a day and names the cache that answers each turn. The pattern holds for any agent client whose conversation only grows: the engine’s own buffers answer the next turn, the NVMe answers after a restart or an eviction, and a cold prefill reads only what no tier holds. Each case is described in the sections that follow; the numbers in the figure are the cases those sections measured.

13.4 The resident prefix

An agent client such as opencode sends one conversation that only grows: each turn’s prompt is the previous prompt, plus the previous answer, plus a tool result. At 190k tokens a whole snapshot would be about 20 GB, so the state store cannot keep one.

engine/cache.py::ResidentPrefix uses the fact that the previous prompt’s KV is still in the engine’s own buffer, row for row. It leaves the KV in place and clones only the recurrent state and convolution tails (146.8 MiB) at every 1,024-token boundary of the prefill, the grid. These copies are the anchors. A new request that shares the first $c$ tokens of the resident prompt resumes at the largest anchor at or below $c$: the engine copies the recurrent state back, sets the KV length, tells the drafters their caches are valid up to there, and prefills the rest.

Resuming this way computes the same state as a cold prefill on the same grid. Anchors come only from prefill chunks, never from rows a verify round wrote, and the resumed request forwards the rest of its prompt on the same 1,024-token grid a cold prefill uses. tests/test_resident.py checks on a small random model that the resumed logits, recurrent state, convolution tails and KV rows equal a cold prefill’s bit for bit. The anchor budget (--resident-gb, 2.5 GiB) holds 17 anchors, and the newest 4 are never evicted, because the next turn resumes near the end.

Resuming at a finer line than 1,024 was built and measured: a forward that starts inside a grid chunk does not reproduce the rows of the cold forward over that chunk on the served kernels, so it is not in the engine.

Guests.

A short unrelated request between two turns, such as a title request, would overwrite rows the conversation needs. Such a guest (at most --resident-guest-max tokens, 8,192, and under a quarter of the resident prompt: both must hold) gets the rows it can reach copied aside first, into a stash (--resident-stash-gb, 1 GiB), and they are copied back before the next request. The stash holds what the guest can write: its prompt, its max_tokens, 2,048 rows for the monitor’s hidden tokens when the monitor runs, and two margins of 512 rows, not every row the 1 GiB holds (a 410-token question used to ask the memory policy for 1 GiB, and a refused stash cuts the resident rows at the guest’s start). With --parallel N a guest takes an idle side slot first and needs no stash (Section 14.13). A guest keeps no end anchor of its own: a short conversation that starts while a longer one is resident re-reads its first answer at its second turn.

The pinned head.

A request too long to be a guest replaces the resident prompt. Before it cuts the rows, the head of the old prompt is pinned: its KV rows and up to 8 anchors spread over it are copied aside, as long a head as --resident-pin-gb holds (4 GiB, about 33,000 tokens). A later request that shares more with the pinned head than with the rows in place gets them copied back and resumes there. opencode’s compaction request is the case this was built for.

The parked session.

Two sessions used in turn (A, then B, then A) share their head and nothing after it. When a request of another session replaces the resident prompt, the rows it is about to overwrite, from where the two part, are parked with the newest 4 anchors (--resident-park-gb, 4 GiB, about 35,000 tokens past the head). A later request that shares more with the parked prompt gets them back: the parked rows and the rows in place are swapped a block at a time, so B is parked while A comes back. In a hold on 2026-10-01, A came back after B with 41,984 of 53,209 tokens reused and 11,225 re-read in 16.4 s, where it would have re-read about 45,000 (measured). tests/test_resident_park.py checks every swap against a cold prefill bit for bit.

13.5 After a long generation: the end anchor

The rows a generation writes are never anchored on the prefill’s grid. Until the end anchor existed, the next turn shared at most the previous prompt with the resident rows and resumed at its last anchor, so it re-read the whole previous output. On 2026-10-02 one opencode turn prefilled 34,038 tokens and thought for 32,773; the next turn re-read 33,172 tokens in about 44 s (measured).

The end anchor (--end-anchor, on) fixes that. At the end of every generation the resident prefix keeps one more anchor: the recurrent states and convolution tails at the last position the engine forwarded (147 MiB, the same two buffers turn after turn), with the rows in place marked valid up to there. A generation that stopped part way through an accepted block has its pending commit cut back to the committed tokens first. The next request that starts with every one of those tokens (the prompt and the answer as the client sends them back) resumes there and forwards only what is new: the turn after a 33k thinking forwards its tool result and the template’s glue. The ledger’s cache source is then end. Figure 13.4 shows the idea.

An answer’s reasoning the client does not send back. Open WebUI sends a finished answer back without its reasoning, and the Qwen3.8 template renders such a message with an empty <think> block, so the copy parts from the rows right after <think>. The end alignment keeps where each generation’s reasoning ends and aligns such a copy with the reasoning taken out; the prompt is served with the rows’ tokens up to the end anchor, the reasoning included, and resumes there (QWEN38_END_ALIGN_REASONING). Before it, every Open WebUI chat whose first answer thought re-read that answer on its first follow-up (cached 0.71 to 0.89, 2.5 to 5.0 s to the first token); with it the first follow-ups took 0.97 to 0.998 of the client’s tokens from the cache (measured).

Verify blocks wrote that state, so resuming there is a session resume, held to the greedy gate rather than to bit equality (Chapter 12). The engine tracks the lowest such row (taint) and keeps the rest of the caches exact around it.

Two details make the client’s copy match. The tokens the conductor hid in an answer (a hidden close’s phrase, a note) are in the rows and not in the client’s copy, so the server puts them back where they sit before the prefill. And the client’s copy is text the chat template renders again, whose whitespace next to a hidden span can differ from the rows’; the replay aligns runs of whitespace-only tokens there (--end-align). In the served engine’s suffix store on 2026-10-02 (4,607 turns that followed another), the next prompt held the engine’s visible output token for token to its end in 3,844, 83 % (measured); where it did not, it parted late, and such a turn resumes at the prompt’s last grid line as before.

13.6 The state store

engine/cache.py::StateStore keeps whole snapshots, keyed by the tokens that produced them, for two uses. The session cache keeps a conversation’s state after its turn ends, and the next turn forwards only the new message. The prefix cache takes a snapshot at the longest chunk boundary of a prefill that fits the per-entry cap, so a shared system prompt costs nothing from the second request on.

A snapshot is about 151 MB of recurrent state plus about 104 KB a token of target and drafter KV. The budget therefore counts conversations long before it counts tokens. The served budget is 2 GiB (CACHE_GB) and one snapshot may take at most 1 GiB (--store-entry-gb, about 7,000 tokens): under opencode the store serves short chats, and longer sessions are the resident prefix’s. Room is made before a snapshot is cloned, not after. Every hit re-checks the stored tokens element by element before restoring anything: a 64-bit hash is a hint, never an answer.

13.7 The disk tier

Every opencode session starts with the same 24,000 to 30,000 tokens: the reasoning header, opencode’s system text and the tool block. The resident prefix serves that head only while its rows are in the engine’s buffer; a restart, a deploy or a long unrelated request throws them away. engine/disk_cache.py::DiskTier keeps them on the NVMe.

It stores what a restore needs in two kinds of file on the prefill’s 1,024-token grid. A chunk file holds rows $b - 1024$ to $b$ of every position-indexed buffer (the attention KV and the drafters’) and those 1,024 token ids, keyed by the hash of the whole prefix up to $b$: 104 MiB a chunk on the served stack. Two prompts that share 30,000 tokens share 30 chunk files. An anchor file holds the 48 recurrent states and their convolution tails at one line (147 MiB), with the hashes of the chunks below it; it is usable when every chunk below it is on disk.

What is written is decided before the prefill runs. Every grid line a prefill forwards cold is counted by its hash, and the longest line of the prompt counted twice (admit_after) gets an anchor. The first long request of a session gets anchors at the last 4 lines below where its user message starts, because the next session in the same folder parts there. A line where an earlier prompt read this far and went on differently (a branch) gets one too. Prompts under 4,096 tokens are not written, and neither is anything a session resume wrote.

With --parallel N the side slots read the tier too: a side slot’s prefill restores an anchor into its own buffers, the same bytes as on slot 0, and writes, counts and fences nothing of slot 0’s, so a new chat sent to a side slot starts from its client’s head on the NVMe (QWEN38_SIDE_DISK). The writes never hold a request. An anchor’s state is copied to a staging slot as the prefill passes its line; a writer thread copies the rows out after the prefill and writes them while the request decodes. Each file goes to a temporary name first and is renamed after its fsync, with a crc32 per section. A read compares every stored token id with the prompt’s before a byte reaches the engine, checks every section’s crc32, and makes the restored prompt the resident one with the disk anchor as its anchor. A file that fails its check is deleted with every longer chunk of its chain, and the request resumes from what is in place or starts cold. Files of another fingerprint (weights, engine code, arithmetic flags, grid, library versions) are swept at the start.

On the board, O_DIRECT writes ran at 4.3 GB/s and reads at 7.35 GB/s (measured, tools/disk_cache_bench.py). On 2026-10-02 a 15,360-token head was restored in 334 ms and the server’s time to first token went from 23.1 s cold to 2.7 s (measured, tools/disk_cache_live.py).

The dashboard’s Memory screen, Disk cache tab, sets the budget (50 GB), how long an unused entry is kept (7 days) and the free space the tier leaves the file system, never under 50 GB. When it is full, the least valuable entry goes first: its hits plus one, times the tokens it saves per byte it holds, halved for every day unused.

13.8 Live conversations on the NVMe

The disk tier keeps what a cold grid prefill wrote, which in an agent conversation is the head. Everything past the first answer is session state. Two re-reads on 2026-10-03 show the gap: a deploy restarted the engine and the next turn of a 130,000-token conversation read 120,102 tokens again (158 s); minutes later a 10,216-token request of another client replaced the resident prompt and the conversation’s next turn re-read everything above 32,768 tokens (165 s).

engine/spill.py::SessionSpill keeps whole conversations on the NVMe: the rows of every position-indexed buffer, the end anchor with its hidden spans, the newest 4 grid anchors, the token ids, taint and the session key. These are the bytes that were in the engine, so a restore is held to the gate the rows were held to in RAM. It writes at two moments. A planned stop (SIGTERM’s drain) writes the resident and the parked conversation, and the next start reads the resident one back before its first request. A request of another session that is not a guest writes the resident conversation first when it has at least 16,384 tokens and the park cannot hold it. The spill writes by content, in blocks of 1,024 rows with two 64-bit sums per buffer computed on the device, so a block that did not change is not written again.

Measured on the board (the 0.4.0 series): a 41,587-token conversation evicted by another session was written in 1.03 s (5.20 GB) and read back in 0.94 s. After a restart through the drain, the next turn of a 41,719-token conversation resumed at its end anchor with a time to first token of 635 ms (measured). The NVMe’s own write cache sets the bound: a sustained write ran at 4 to 5.6 GB/s for the first 16 to 24 GB and then at 0.3 to 1.3 GB/s until the drive rested.

13.9 Compaction as a continuation

opencode’s /compact sends its compaction agent’s prompt and one user message that holds the whole history as text. That text shares no token with what the engine read, so a compaction read tens of thousands of tokens cold while the conversation sat resident. server/compaction.py serves it instead as the continuation of the conversation the engine holds for the session: the tokens up to the end anchor of its last generation (in RAM, or its entry on the NVMe), then one user turn with the client’s system text and instructions, thinking off. The prefill reads only that turn. A guard checks that the first and last user lines of the client’s history are in the record; any miss serves the request as sent.

A summary still has to be written, about 2,000 tokens, and the server writes it ahead: when no request has come for 30 s and the resident conversation is an opencode session of at least 32,768 tokens, it runs the continuation in scratch rows, under the board’s lock taken without waiting. A request that comes for the engine, or a hold that asks for the lock, stops it at the next chunk or block, and the conversation’s rows and anchors stay as they were. A compaction request of that session at that end anchor is then answered from the summary without the engine.

13.10 The persistent suffix store

engine/cache.py::PersistentSuffixStore is an append-only log of the token ids this engine has read and written, with a suffix array over it. The lookup drafter reads it beside the corpus. It holds token ids and never text, in a directory of mode 0700 outside the repository, capped by --suffix-store-mb (192 MiB, about 48 million tokens); over the cap it forgets the oldest half. A request adds only what one of the last four requests did not already add, so an agent conversation that resends its growing prompt is not written 200 times. A store that has seen a benchmark’s prompts inflates that benchmark, so benchmarks run with the store off or clean (Chapter 15).

13.11 Images in a prompt

An image’s prompt rows are placeholder tokens, and the placeholder is the same token for every image, so a cache keyed on tokens would answer a request with another image’s state. The caches therefore compare key ids in which each image’s rows are replaced by an id taken from a digest of its preprocessed patches. The same URL with new pixels misses; the same pixels under another URL hit. The vision tower’s output is cached by the same digest (--image-cache-mb, 512), so a conversation that sends an image on every turn encodes it once.

13.12 Memory limits

All caches share the board with the weights and the KV buffer. engine/cache.py::Budget holds them to three limits, and Section 14.10 describes the gate that enforces them before every prefill. Every limit decides only where a prefill starts; a cache given back is the cold path, never a different token.

13.13 Routes and knobs

GET /v1/cache/stats reports every cache’s counters, and POST /v1/cache/clear empties them (the admin rule; ?tier=disk the disk tier, ?tier=all both). The flags are in Table 31.1; the main ones are --resident-gb, --resident-stash-gb, --resident-guest-max, --resident-pin-gb, --resident-park-gb, --end-anchor, --cache-budget-gb, --store-entry-gb, --prefix-chunk, --disk-cache, --session-spill and the --compaction family.