3Installing TandemLLM

TandemLLM has two parts to install. The engine runs the models and needs a CUDA GPU; its test board is the DGX Spark (Section 1.1). The dashboard and its proxy need only Python, run on any machine, and also run inside the engine’s own server. Most setups use both. Find your setup in Table 3.1, then follow the sections it names.

3.1 Setups, and which ones work

The parts combine. One engine can sit behind one address together with other engines and cloud APIs, and one dashboard watches all of them. Table 3.1 says which combinations work today, and how far each one is tested.

3.2 What the engine needs

The engine is tested on one NVIDIA DGX Spark, a GB10 GPU (sm_121) with 128 GB of memory that the CPU and the GPU share, running Linux on aarch64. On another CUDA GPU the installer turns off the one kernel built for the GB10, and the engine runs best effort. The installer checks four things before it changes anything: an NVIDIA GPU and driver, the CUDA compiler (one kernel builds on its first use), about 70 GB of free disk, and 64 GiB of free memory at the start. With --setup it downloads only Wren’s model, so it asks for about 5 GiB of disk beside the Python environment and 8 GiB of free memory; use-profile.sh checks the disk and the memory a profile needs before it downloads or starts it.

The software is pinned. The engine imports PyTorch 2.13 built for CUDA 13.0, torchvision 0.28 (the CUDA 13.0 build that matches that torch), Triton 3.7, transformers 5.12, Pillow 11.3, safetensors and numpy, and nothing else. Python 3.11 is the tested version; under 3.12 one grammar test fails. The installer’s lock lists every package with its version, for aarch64 and x86_64.

Two of those packages are there for images. Pillow decodes a request’s images, and torchvision is the backend of the image processor that the checkpoint ships. The engine pins that backend (--image-backend torchvision) and refuses to start without it, because transformers would otherwise fall back to its Pillow backend, whose patches differ by up to two 8-bit levels after a resize (measured, 0.4 % of the values of one resized image). Every cache keys on those patches, so the backend has to be fixed. --image-backend pil is the explicit choice without torchvision, and --vision off serves text only.

Run one engine per board. Two engines loading side by side run the board out of memory, and a DGX Spark in that state stops answering until it is power cycled. Section 6.1 explains how the scripts enforce it.

3.3 A new box: setup mode, then the Setup Guide

Since 0.8.1 a new DGX Spark needs two commands and no Hugging Face token. Every file a profile reads is public on Hugging Face: our own packs, the vendors’ releases, and turboderp’s Flash-Next packs (Chapter 17).

curl -fsSLO https://raw.githubusercontent.com/0xBakeer/TandemLLM/v0.13.0/install.sh
bash install.sh --setup --yes

The first line fetches the installer from the release tag. With --setup the installer puts the code and the Python environment in place and downloads only Wren’s model (Qwen3.5-2B, 4.3 GB), the setup guide’s assistant. It writes the configuration with no weights and starts setup mode on 127.0.0.1:8000: the dashboard and its #/setup page, Wren, and no model loaded.

Then open the Setup Guide in a browser and pick a model. The guide shows each model with its weight packs, checks that the choice fits in memory, downloads it and starts the engine; it walks through the model, the weights, the tools and the parallel slots in eight steps. docs/setup.md describes the page.

• On the box itself, open http://127.0.0.1:8000/dashboard/#/setup.
• From another computer, open an SSH tunnel to the box first, ssh -L 8000:127.0.0.1:8000 <user>@<box>, and open the same address on that computer.
• On your network, install with --host 0.0.0.0 and open http://<box>:8000/dashboard/#/setup from any computer that reaches the box. Until an admin password is set, anyone who reaches the port can run the setup.

The installer prints the addresses when it finishes, and while the box is in setup mode the dashboard opens on the Setup Guide by itself.

3.3.1 Advanced: from the command line

The Setup Guide runs ops/use-profile.sh for you. To pick the weights from a shell instead, run the copy in ~/TandemLLM/run/ops/, next to the serve.env the installer wrote:

bash ~/TandemLLM/run/ops/use-profile.sh --list
bash ~/TandemLLM/run/ops/use-profile.sh <profile> --dry-run
bash ~/TandemLLM/run/ops/use-profile.sh <profile>

--list prints every profile and marks the active one. With --dry-run the script prints every file’s URL and size and changes nothing. Without it, the script downloads what the profile needs and checks every weight file against the sha256 the hub lists. It then leaves setup mode, starts the engine on port 8000 and sends one smoke request. Since 0.8.2, community/<id> runs a pack someone else published (Chapter 7).

A folder appears only when all its files are there. If a download stops, run the same command again: it keeps the files that finished and continues. The engine then needs about 80 GiB of free memory for a 27B profile and 70 GiB for flashnext-3.05. If the profile does not come up, the script brings setup mode back.

The installer checks out the tag v0.13.0 from the repository --repo names, the public one by default. Where the tag is missing, curl fails with a 404, and the installer stops at its clone step with “no tag, branch or commit ’v0.13.0’”. --ref picks another release, and --repo another repository.

3.4 Updating a box in setup mode

A box with a running profile counts as configured, and --setup alone refuses to touch it. To update, add --force-setup: the installer stops the engine, checks out the new tag and comes back in setup mode. Then switch to the profile again; its files are on disk already, so nothing downloads twice.

bash ~/TandemLLM/src/install.sh --setup --force-setup --yes --ref <new tag>
bash ~/TandemLLM/run/ops/use-profile.sh flashnext-3.05

3.5 The installer

Without --setup, one command sets up the NVFP4 profile, StairCut included:

curl -fsSL https://raw.githubusercontent.com/0xBakeer/TandemLLM/v0.13.0/install.sh | bash

The installer works in seven steps, and each step is skipped when it is already done, so running it again updates an install and resumes a download that stopped.

1. Checks. The GPU needs its driver and the CUDA compiler, the disk and the memory need room, and Hugging Face must answer.
2. The engine: a clone of the repository --repo names into ~/TandemLLM/src at the pinned release, v0.13.0.
3. A Python venv in ~/TandemLLM/venv with the pinned packages; torch and torchvision come from the CUDA 13.0 index. If neither uv nor Python 3.11 is on the machine, the installer fetches uv first.
4. The weights, into the normal Hugging Face cache: the vendor’s FP8 checkpoint (it supplies the tensors the NVFP4 overlays do not replace, and the tokenizer), the four NVFP4 files and both fine-tuned drafters. With --setup, only Wren’s model; use-profile.sh fetches the rest later.
5. A lookup corpus built from public text (one English and one German Wikipedia shard, plus Python sources from the venv’s own packages, about 31 million tokens). --no-corpus skips it.
6. The configuration: ~/TandemLLM/run/ops/serve.env, the release’s ops/serve.env with this machine’s paths, and a control script, ~/TandemLLM/bin/tandem.
7. The start: the engine starts, the script waits for /health, and a smoke test prints tokens a second. --no-start stops after step 6.

The weights are public. --hf-token (or HF_TOKEN) is needed only for a repository that is private or gated, and a stored hf auth login works too. Table 3.3 lists every option; the most used ones are --setup (Section 3.3), --repo (another repository, such as a fork), --dir (default ~/TandemLLM), --port and --host (default 127.0.0.1:8000), --ref (any git ref; v0.2.0-staircut is the paper’s release), --hf-token, --systemd for a systemd user unit, --dry-run to print every step and change nothing, and --uninstall.

Environment variables the installer reads: TANDEM_REF, TANDEM_REPO, TANDEM_DIR, TANDEM_PORT, TANDEM_HOST, HF_TOKEN, HF_HOME.

The installer’s corpus is not the corpus the paper’s numbers come from; that one is not published. With the installer’s corpus, or with --no-corpus, the lookup drafter proposes different continuations, so speeds can differ from the paper’s. How much is not measured. On a CUDA GPU other than the GB10 the installer turns off the skinny NVFP4 kernel, which is built for sm_121a only, and StairCut’s cost tables still come from the Spark, so such GPUs are best effort.

3.6 After the install

The install directory holds everything the installer wrote:

The control script takes one command:

~/TandemLLM/bin/tandem start      # ops/start.sh with run/ops/serve.env; waits for /health
~/TandemLLM/bin/tandem stop 120   # SIGTERM, wait for the drain, kill only after it
~/TandemLLM/bin/tandem restart
~/TandemLLM/bin/tandem status
~/TandemLLM/bin/tandem smoke      # /v1/models and one short completion, with tok/s
~/TandemLLM/bin/tandem logs       # follow the engine's log

3.7 The first request

The server speaks the OpenAI API on port 8000. With the conductor on, as in the shipped profile, it serves one model called TandemLLM, and any model name a client sends is served as TandemLLM:

curl -s localhost:8000/v1/chat/completions -H 'Content-Type: application/json' \
    -d '{"model": "TandemLLM", "messages": [{"role": "user", "content": "Say hello."}]}'

Point other clients at the same address:

For the engine without the conductor, send the header X-Tandem: raw. That is the release’s raw path, the one the benchmarks and the paper measure. Chapter 21 explains the profiles.

The dashboard is at http://<host>:8000/dashboard/. It asks for no password; read Section 3.8 before anyone else can reach the port.

3.8 Access and sign-in

TandemLLM has no sign-in turned on by default. The engine’s OpenAI routes are always open, and so is everything else until you set an admin password: the dashboard, the conductor’s settings, the memory notes, the transcripts, the decision models, the configuration and the cache routes. Anyone who reaches the port can read and change them. The installer binds the engine to 127.0.0.1, so only the machine itself reaches it. With --host 0.0.0.0, or when you start ops/serve.env from a checkout (its HOST is 0.0.0.0), the whole local network does.

Table 3.6 has the switches. Only /metrics has a token of its own, for a Prometheus scraper, and ops/make-secrets.sh writes that one alone unless --admin asks for the password too.

Setting a password has a cost in this release: the dashboard has no sign-in screen. Its read-only views keep working while the login is off, but its edits and the Memory screen show the server’s 401 and a banner. Remove the line from secrets.env and restart to open it again. A request from the machine itself (loopback, without proxy headers) needs no token, because the watchdog and the measurement tools rely on that.

TandemLLM serves one user. Every client that reaches the engine is that user, so a “remember that” from any client lands in the same memory notes. A request that must not touch them sends X-Tandem-Memory: off. Until sign-in and several users arrive, keep the port on loopback or a trusted network. To reach it from elsewhere, put a reverse proxy with its own sign-in, or a VPN, in front of it.

3.9 The small models

Three small models can run beside the main model. They are off in the shipped profile, because each needs its weights on the disk. Each is one line in local.env and a restart.

Download a model with hf download <repo> --local-dir <dir>. The small models load after the engine, one at a time, through the residency planner, which refuses a load that would leave the board short of memory (Section 6.8). The dashboard’s System view shows what is loaded.

3.10 Other engines and cloud APIs beside the engine

The engine’s own server can be the one address for everything. Point QSE_BACKENDS in local.env at a config database and restart:

echo 'QSE_BACKENDS=$HOME/.tandemllm/config.sqlite3' >> ~/TandemLLM/run/local.env
~/TandemLLM/bin/tandem restart

Then add the backends on the dashboard’s Backends screen, or as TANDEM_* variables (Chapter 5). The engine keeps serving TandemLLM as before, and a request for another model goes to its backend on the handler’s own thread: it never takes the engine’s lock or a place in its queue, so a slow cloud answer cannot hold up the engine. The engine’s conductor can run in front of each backend (harness.mode, Chapter 26), and one ledger and the Parallel view cover them all. On a DGX Spark the other backends are cloud APIs or engines on other machines; a second GPU engine on the same Spark is not supported (Table 3.1). No live run of this in-process relay is recorded yet; the standalone proxy in front of the engine is the setup that ran live.

The other way round works too: tandem-dash on a laptop or a small server, with the engine as a backend of kind tandemllm (Chapter 4).

3.11 Running from a checkout

The installer does nothing you cannot do by hand. In an environment with the packages above, download the published weights and drafters with hf, the Hugging Face CLI (pip install -U huggingface_hub):

hf download Qwen/Qwen3.8-27B-FP8
hf download 0xBakeer/TandemLLM-Qwen3.8-27B-NVFP4 --local-dir ~/tandem/nvfp4
hf download 0xBakeer/TandemLLM-Qwen3.8-27B-DFlash2-b8 --local-dir ~/tandem/ft-b8
hf download 0xBakeer/TandemLLM-Qwen3.8-27B-DFlash2-b16 --local-dir ~/tandem/ft-b16

Then edit the path lines at the top of ops/serve.env: REPO (the checkout), PY (a Python with the packages), NV (the three NVFP4 files, mlp, gdn and attn, comma-separated), HEAD (head-fp8.safetensors), CKPT8, CKPT16 and CORPUS. Start it with bash ops/start.sh. That is the served profile.

In a PyTorch container that already has torch 2.13.0+cu130, add the two image packages:

pip install pillow==11.3.0 torchvision==0.28.0+cu130 \
    --extra-index-url https://download.pytorch.org/whl/cu130

A server started directly with python server/app.py and no further flags is the raw engine. The conductor, the disk tier and the usage ledger come only with --conductor-config, --disk-cache and --usage-ledger, which ops/start.sh passes from QSE_CONDUCTOR_CONFIG, QSE_DISK_CACHE and QSE_USAGE_LEDGER. To try the engine with the vendor’s base drafter instead of the fine-tuned ones:

hf download z-lab/Qwen3.8-27B-DFlash2
python server/app.py --port 8000 --max-len 32768 --drafter dflash2 --dflash2-path greedy \
    --nvfp4 ~/tandem/nvfp4/mlp.safetensors,~/tandem/nvfp4/gdn.safetensors,~/tandem/nvfp4/attn.safetensors \
    --fp8-head ~/tandem/nvfp4/head-fp8.safetensors

3.12 Building the weights yourself

The published NVFP4 set was quantised once from Qwen’s BF16 release with GPTQ on the NVFP4 grid. Rebuilding it needs the BF16 checkpoint and a GPU with about 80 GB of memory at the default --h-budget-gb 22; it took 25 minutes on an RTX PRO 6000 (measured):

hf download Qwen/Qwen3.8-27B
python tools/calib_corpus.py --out ~/calib
python tools/quant_nvfp4.py build --model <the BF16 snapshot> --methods gptq --out-dir ~/nvfp4-bf16 \
    --corpus ~/calib/calib-code.txt:80000,~/calib/calib-en.txt:60000,~/calib/calib-de.txt:16000
python tools/quant_head.py build --out ~/nvfp4-bf16/head-fp8.safetensors --ratios 1.0,0.95,0.90

The build writes gptq/mlp.safetensors, gdn.safetensors and attn.safetensors; tools/quality_gate.py --plan scores them against the BF16 model. The engine loads them over the FP8 checkpoint, as it does the published files. Chapter 9 explains the method and the gate.

3.13 Profiles

A serving profile is a serve.env file: which weights, which drafters, which settings. Three ship with the NVFP4 set. They differ only in their weights; drafters, StairCut, caches and the exactness gate are the same in all three. The weight profiles under ops/profiles/ are the other kind: nvfp4, six exl3-<bpw> sizes of the 27B and five flashnext-<bpw> packs, which ops/use-profile.sh switches between one engine at a time (Section 6.3).

The balanced and FP8 profiles source ops/serve.env and change two lines, NV and HEAD. They inherit StairCut’s verify prices, which were measured on the NVFP4 profile. The router corrects their level online, but the staircase’s shape for those two profiles is not measured, and neither is their speed. Do not confuse these files with the conductor’s profiles (coding, chat and the others), which are documents the dashboard edits.

3.14 Upgrading and uninstalling

To upgrade, run the installer again with the same --dir. It checks out the new release, installs the new lock and rewrites serve.env from the release’s template; local.env stays. The usage ledger migrates in place at the first start: columns are added and every row is kept. A box installed with --setup and switched to a profile since then upgrades with --setup --force-setup and the profile switch after it, as Section 3.4 shows.

Coming from 0.3.0, a client sees these changes:

• The conductor is on in the shipped profile. /v1/models lists TandemLLM alone, every response carries the headers X-Tandem and X-Tandem-Profile, and every request runs through the router, with a real cap on the thinking per effort level. X-Tandem: raw gives the release path of 0.3.0. A benchmark client that cannot send a header should run against a server with CONDUCTOR=raw and an empty QSE_CONDUCTOR_CONFIG in local.env.
• The disk tier and the session spill are on: up to 50 GB under ~/.qwen38-spark-engine/cache/disk, never below 50 GB free on the disk. QSE_DISK_CACHE=off turns both off.
• A streamed request that finds the engine busy waits as long as a request may run, with an SSE comment every 10 s, instead of a 429 after 240 s. A short request is served between two blocks of the running turn.
• The image processor’s backend is pinned to torchvision.
• A malformed tools field is a 400 that names the item.
• The memory limits are enforced: 70 GiB between requests, 77 GiB during a prefill.

To run 0.4.0 as 0.3.0 was served, put CONDUCTOR=raw, an empty QSE_CONDUCTOR_CONFIG= and QSE_DISK_CACHE=off in local.env.

To roll back, run the installer with --ref set to the earlier tag; the ledger keeps its added columns and the older release ignores them. To uninstall:

bash ~/TandemLLM/src/install.sh --uninstall --yes

This stops the engine, removes the systemd unit if one was installed, and deletes the install directory. It keeps the weights in the Hugging Face cache, which other tools may share, and ~/.qwen38-spark-engine/ with the conductor’s documents, the journal, memory notes, transcripts, the disk tier and the small models. Delete that directory to remove them too.