6Operations

A DGX Spark has 128 GB of memory that the CPU and the GPU share, and a board that runs out of it does not recover by itself: the GPU driver locks, the machine stops answering, and only a power cycle brings it back. Most rules in this chapter exist to keep that from happening on a machine nobody can reach.

6.1 One engine per board

The served engine reserves about 69 GB. A second engine of the same size does not fit, and two engines loading side by side wedge the board. Every script in ops/ checks for a running engine first, and a loading engine counts although it answers nothing yet: the server binds its port only after the weights are in, so for the minutes of a load neither /health nor the port shows it. ops/engines.sh finds engines by their process instead (. ops/engines.sh; engine_pids). The same rule holds for a vLLM or SGLang started on the same board.

6.2 Memory rules

Keep at least 15 GiB of MemAvailable at the worst case of anything you run. Before any change that can raise memory (a cache budget, the context length, a new resident copy), write down its worst-case bytes first. During a trial, run a guard that kills your test process when MemAvailable drops under 10 GiB.

Page cache counts as available memory but is not always given back in time. After a load the engine never reads the weight files again, so ops/start.sh drops their page cache (tools/drop_page_cache.py, posix_fadvise with DONTNEED, no root needed). Without that drop an 8,192-row prefill once took free memory to 1 GB and the driver logged NV_ERR_NO_MEMORY.

Inside the engine three limits hold the caches and the prefill workspace: 70 GiB between requests, 77 GiB during a prefill, and 2 GiB of free allocator blocks kept between requests. Section 14.10 explains how the memory gate enforces them.

6.3 Two models, one at a time

The board serves one big model at a time, the 27B or Qwen3.8-Flash-Next, and ops/active-profile names the profile it serves. The 27B’s profiles are nvfp4 and exl3-<bpw>; Flash-Next’s are flashnext-2.05 to flashnext-6.05, one per EXL3 pack (ops/profiles/_flashnext.env and one small file each). The same scripts serve both:

• ops/use-profile.sh flashnext-3.05 checks or fetches the pack, stops the engine under the box lock, starts the new profile and runs the smoke check. A failed smoke starts the previous profile again, 27B or Flash-Next. A Flash-Next profile needs no 27B drafter, draft head or base snapshot. Since 0.8.1 a 27B profile’s switch fetches those too, when they are missing: the K3 drafters, the K4 draft head, the BF16 drafters into train/ and the FP8 base checkpoint (Section 3.3).
• The fetch takes one branch of turboderp/Qwen3.8-Flash-Next-exl3, pinned to its commit when the download starts. The branch has no checksum list, so ops/fetch_weights.py records the hub’s sha256 of every file and checks against it. The 4.05, 5.05 and 6.05 packs share one n-gram table file, which a second pack hard-links instead of downloading 39 GB again.
• ops/start.sh passes the profile’s QSE_ENGINE_FLAGS (--model <pack> --drafter mtp) and --corpus in place of the 27B’s model and drafter flags; without that key the 27B’s command line is the one it was.
• The page-cache drop after a start drops the pack’s pages and keeps a warm n-gram table, which the engine keeps reading.
• The watchdog and the reboot line run ops/start.sh of the tree that serves, so a crash or a reboot brings back the active profile, whichever model it is. The status page names the profile that is loading.
• A start with a --max-len above the profile’s largest window stops before the load and names both numbers (Table 17.2).

The memory rules hold for Flash-Next with two additions. Under the warm policy the n-gram table is page cache that the engine reads on every token, and MemAvailable counts it as free; every floor check therefore takes the table’s resident bytes off first, and the dashboard shows them beside the memory split. Each profile sets its own ceiling and peak, with the arithmetic in the file: 66 and 77 GiB for 3.05, 81 and 92 GiB for 4.05, against 70 and 77 for the 27B. A 3.05 engine held 46.9 GiB after its load, with MemAvailable at 68.0 GiB (measured). Chapter 17 has the windows each pack fits and the runs behind them.

6.4 Starting and stopping

ops/serve.env is the served configuration: about 60 settings, from paths and the port to every engine switch, each with the reason for its value in a comment. The scripts read it.

ops/start.sh is safe to run at any time. If the engine is healthy it says so and exits. Once the server answers, it drops the weights’ page cache, asks the conductor to load the decision models named for the start, and loads the narrator’s model, each through the residency planner.

Stopping drains. SIGTERM and SIGINT make the server stop taking work: /health answers draining with a 503, new requests get a 503 with Retry-After: 30, the generation in flight finishes with its last chunk and [DONE], and the process exits 0. After the work in flight, the drain writes the resident conversation and the parked one to the NVMe (about 3 s for 130,000 tokens), and the next start reads the resident one back before it serves, so that conversation’s next turn resumes at its end instead of reading everything again (Section 13.8).

6.5 The watchdog

The watchdog restarts only what it is sure about. A healthy server is left alone, and so is a draining one, because killing a graceful stop halfway is worse than a slow stop. A loading engine is left alone for up to 900 s. Only a server that fails three checks in a row is restarted, so one slow check during a long verify does not bounce a working server.

At boot two cron lines bring the service up:

@reboot sleep 90 && $HOME/<checkout>/ops/start.sh >> $HOME/<checkout>/logs/boot.log 2>&1
* * * * * $HOME/<checkout>/ops/watchdog.sh

ops/qwen38-engine.service is a systemd user unit for the same job, and the installer can write one (--systemd). A user unit survives logout only with loginctl enable-linger, which needs root. Install one mechanism, never both: two supervisors for one port start the service twice.

6.6 Holding the board

Measurements need the board to themselves, and so does anything that loads a second copy of the model. ops/hold.sh makes the safe sequence the easy one:

flock ~/.qwen38-box.flock ops/hold.sh 30 python tools/verify_spec.py ...

It arms a pause file so the watchdog stands back, keeps it fresh for the whole hold, stops the service and runs the command. Afterwards it restarts the service and waits for /health. On every exit path, a killed command included, it removes the pause. The lock file makes two holds wait for each other, and a pause file older than 2,400 s is ignored, so a crashed hold cannot leave the service unsupervised for long. Background jobs of the engine itself (the summary written ahead, the shadow reader, the memory writer) take the same lock without waiting and give it back when a hold asks for it. Taking the service down stops everyone who uses it: hold the board when nobody needs the service, and keep holds short.

6.7 While the engine is away

A stop, a hold or a start takes the engine away for minutes. ops/start.sh, ops/stop.sh and ops/hold.sh write what they do to logs/engine-status.json: the state (stopping, held, starting, up, failed, down), since when, a hold’s reason (HOLD_REASON, or the first line of the file HOLD_NOTE names) and its budget, and the last five start times, whose median is the next estimate. With QSE_STATUS_PORT in serve.env the scripts keep a small responder on 127.0.0.1 (ops/engine_status.py). It answers what a reverse proxy sends it when the engine does not: a page for /dashboard/ that names the state, the reason and about how long is left, checks /health every 5 s and opens the dashboard when the engine is back; and for every other request the OpenAI error shape, status 503, Retry-After, with the same facts under engine. An open dashboard shows them in its offline banner. GET /v1/models is the exception: it answers 200 with the engine’s last model list, which set up keeps in the status file, so Open WebUI keeps TandemLLM in its list through a restart and a chat shows the engine’s sentence instead of “Model not found”. The nginx lines are in docs/operations.md.

6.8 Roles beside the engine

The engine loads first and holds most of the board. Anything else that runs beside it (a decision model, the narrator, the shadow reader) loads after it, one at a time, through the residency planner (server/conductor/residency.py). A role may load when

with the engine’s declared peak at 77 GiB and the floor and margin at 15 and 6 GiB by default. It never loads during a hold, and never while another load runs. A refusal names its reason (refused_hold, refused_load_in_progress, refused_memory, refused_unknown_role) and is journaled in residency.jsonl under the conductor’s directory.

A claim is the role’s peak, not its idle size, and it is what the role takes from MemAvailable, not what nvidia-smi shows. The floor, the margin, the settle time, the load order and the claims are the conductor’s residency document, versioned like the others and edited in the dashboard. The System view shows the plan: the engine’s own memory split, MemAvailable, every role with its claim and whether it fits now, and the last refusal. Nothing unloads on its own when MemAvailable falls after a role loaded; the plan shows it.

ops/roles.sh loads external roles. ROLES_AUTOLOAD in serve.env names the roles start.sh brings up after the engine (none in the shipped profile). A role served by vLLM runs BF16 weights with an unquantised KV cache, and its --gpu-memory-utilization is the planner’s byte math, never vLLM’s default of 0.9, which on this board would take the engine’s memory too.

6.9 Logs

Each request leaves one log line with its token counts, its finish reason and its rate, and no line carries the text of a prompt or an answer. --log-content lets exception messages that quote a request into the log, and --log-request-keys logs parameter names and scalar values, never messages. The installed engine’s log is ~/TandemLLM/src/logs/engine.log (bin/tandem logs follows it), and the dashboard’s Dev view shows it live.

6.10 Monitoring

/health returns the status, the queue counters (running, waiting, served, refused, errors, timeouts, abandoned), the cache report and the allocator’s memory figures. On this board nvidia-smi reports no used memory, so torch.cuda.memory_allocated and memory_reserved are the numbers to watch. Allocated memory that climbs across requests is a leak; reserved memory that climbs while allocated memory stays flat is fragmentation.

/metrics serves Prometheus metrics (Chapter 31). ops/monitoring/ has a scrape configuration, alert rules with their tests, and a Grafana dashboard that tools/grafana_dashboard.py generates. The conductor’s stall watchdog can send a push notification when an unattended session’s client has gone; the URL comes from TANDEM_NOTIFY_URL or the policy, and is empty in the shipped policy.

6.11 Settings

engine/settings.py is the one registry of every QWEN38_* switch the engine reads, with its default and the module that reads it; tests/test_settings.py fails if any module reads a switch another way. To see what is in effect:

python -c "from engine.settings import SETTINGS; import json; print(json.dumps(SETTINGS.describe(), indent=1))"

Server behaviour (ports, limits, caches, reasoning) is set by command-line flags, which ops/start.sh builds from serve.env. Chapter 28 lists every flag and Chapter 29 every variable.

6.12 Deploying and rolling back

An installed engine upgrades by running the installer again (Chapter 3). A checkout that serves the service upgrades in four steps:

1. Copy the serving directory to a dated backup.
2. Copy the new files in, keeping the local serve.env values that are site-specific (paths, port).
3. Restart through ops/hold.sh, so the watchdog does not race the restart.
4. Check that /health says ok and that VERSION is the new one, then send a short request.

Run the gate and the client smoke set before step 1 (Chapter 15). A deploy that fails step 4 goes back to the backup. The usage ledger and the config database migrate forward in place and keep every row, and an older release ignores the columns a newer one added, so a rollback needs no data step. The disk tier sweeps files of another code fingerprint at the start; a rollback therefore starts that tier empty, and the session spill keeps its conversations when only the files that keep state changed.