28Watching several engines

With more than one backend, the dashboard answers three questions for all of them at once: what each request is doing right now, how each engine is doing inside, and how the machine behind it is doing. Each answer has its own source, and each source is optional. All of it is measured or null, never estimated: a figure a backend does not report reads “not reported by this backend”.

28.1 Every request at once

Performance, Parallel (#/performance?tab=parallel) shows every request in flight on every backend, the engine included. It reads GET /v1/dashboard/parallel once a second; the server builds the document at most every 0.4 s for all viewers, and a poll with since carries only the new samples.

• The aggregate decode rate now, and its last five minutes stacked by backend.
• The requests by state: waiting, prefilling, thinking, tool call, writing.
• The time to first token at p50 and p90, the cache hit and the cost per minute, over the requests finished in the last five minutes.
• One card per backend.
• One lane per request: backend and model, client, session, state, tokens with thinking apart, decode tok/s now and on average, TTFT, prefill rate, cache, cost so far, elapsed. A lane keeps its row until a few seconds after it ends.

A lane’s live token count comes from the best source the backend allows. The engine’s own requests use the engine’s count. A TandemLLM backend behind a proxy is read from that engine’s own board once a second, because the relay tags each request with X-Tandem-Relay-Id; the engine streams blocks of several tokens, so its frames are not a count. Any other backend’s stream frames count only after they were checked: each finished streamed answer’s frames are compared with its usage.completion_tokens (within 2 tokens and 1 %), answers with and without tool calls apart, and two passing answers of a kind with no failure among the last 8 make the frames a count. DeepSeek’s reasoning and content frames are one token each; its tool-call arguments arrive several tokens a frame, so a DeepSeek lane has no live count while it writes a call. Without a count a lane shows its frames and “not reported”, and its tokens arrive with the usage at the end.

tools/parallel_load.py drives the page on your own backends, for example --mix deepseek-flash=10,TandemLLM=2 --duration 90 --budget-usd 0.5 --board. It reports TTFT, decode rate, cache hit, throughput and cost per model, stops starting turns at the budget, and with --board puts the page’s per-second series beside the frames its own clients received.

28.2 Inside vLLM and SGLang: the engine’s page (L1)

For a backend of kind vllm or sglang the proxy reads the engine’s /metrics every 15 s through a name table checked against vLLM v0.30.0 and SGLang v0.5.21 (proxy/engines.py). Window figures come from the first and the newest reading of the last 5 minutes; with one reading, or no traffic in the window, they are null. A family a release renamed maps to nothing and its tile says “not reported by this backend”: a new release can cost a tile, never a failed read. After a failed read the proxy waits 30 s, then 60, then 120 s.

The proxy’s own /metrics carries the mapped families as qse_backend_* with a backend label, for a Prometheus of your own; they never enter the engine’s own families. Mapping one page of about 500 lines takes about 2 ms on the poller’s own thread, and a relay thread never waits for it.

28.3 Inside vLLM and SGLang: the plugins (L2)

The page gives totals. A single request’s queue time, its prefill time and whether the scheduler preempted it are not on it. Two small packages in plugins/ run inside the engine and send those numbers per request:

pip install plugins/vllm        # tandemllm-vllm-plugin, into vLLM's own Python
pip install plugins/sglang      # tandemllm-sglang-plugin, into SGLang's own Python
TANDEM_COLLECT_URL=http://proxy-host:8080/v1/collect/<backend id>
TANDEM_COLLECT_TOKEN=<the proxy's collect token>

vLLM loads its plugin from the vllm.stat_logger_plugins entry point, once per data-parallel engine in its API server process; SGLang loads its plugin from sglang.srt.plugins in every process it starts, with two hooks on its tokenizer manager and its scheduler’s metrics collector. Neither package has a dependency, and neither reads a prompt, an answer or a token id: each item is counts and times. A plugin keeps the last 10,000 finished requests and 300 one-second scheduler samples in memory and posts what is new once a second, or an empty batch every 10 s when the engine is idle. The collect token is TANDEM_COLLECT_TOKEN in the proxy’s environment, or one the proxy makes and stores in its config database at the first need; it lets its holder post numbers for a backend that exists, nothing else.

The proxy joins each item to the request’s ledger row by an id it sends upstream: vLLM gets X-Request-Id: px-<hex> and reports chatcmpl-px-<hex>; SGLang gets rid in the body. A client that sets its own id keeps it, and the join uses it. The row then gets the engine’s own queue_ms, prompt_ms, decode_ms, decode_tps, prefill_tps, cached_tokens, preemptions and engine_finish, and l2_join says exact. A request the proxy cannot tag (the Responses API, SGLang with n above 1) is matched by time and token counts when exactly one row fits, and says approximate. An item waits up to 10 minutes for its row, and a row for its item.

The engine always starts. If the plugin does not fit a release, vLLM’s loader logs the import error and starts without it, SGLang skips a renamed hook target, and a failing hook is logged at most once a minute and switches itself off after 100 failures in a row. The backend’s card says “L2 plugin” while batches arrive and “L1 only” after 60 s without one.

28.4 The machine behind each backend

The System view lists the machines: this one first, with the server’s own process, then the machine behind each backend, down to the process behind the backend’s port. For a vLLM on the same machine that is its API server and its engine core, with their RSS, CPU, threads, uptime and GPU memory. A backend’s host setting says where to read it:

On a remote machine, run the collector next to the backend and give both sides the same token:

# on the backend's machine
TANDEM_COLLECTOR_TOKEN=$(openssl rand -hex 24) tandem-dash hostcollect --host 0.0.0.0 --port 9108
# on the proxy's machine
TANDEM_COLLECTOR_TOKEN=<the same token>
TANDEM_BACKEND_VLLM1_HOST=collector
TANDEM_BACKEND_VLLM1_HOST_URL=http://gpu-host:9108

The collector answers GET /host as JSON and GET /metrics as Prometheus text (tandem_host_*), both behind the bearer token. It refuses to bind beyond loopback without one, because its answer names processes and their command lines. Command lines show the home as ~, and the value after any flag whose name holds key, token, secret, passw or auth is masked. A remote read runs on its own thread, so a slow or dead host never holds the page.

On a GB10 the CPU and the GPU share one memory pool. The driver reports no memory for the GPU itself, so the view shows “unified” there and MemAvailable is the figure to watch. Nothing is sampled while nobody looks: when the System view asks, the collector reads at most every 5 s and scans the processes at most every 15 s. Measured on a DGX Spark against the running engine: one sample with a process scan cost 7.46 ms of CPU, 0.050 % of a core at that cadence; the memory figures equalled /proc/meminfo to the byte, and the engine’s GPU memory equalled nvidia-smi’s within 49,152 bytes of 81.7 GB (measured, the 0.4.0 series).

28.5 Your own Prometheus

The proxy’s /metrics (open while no token is set; else a bearer metrics or admin token, or a request from the machine itself) carries its own families (qse_proxy_requests_total, qse_proxy_tokens_total, qse_proxy_cost_micros_total, qse_proxy_ttft_seconds, qse_backend_up, qse_backend_inflight and more), the mapped engine families as qse_backend_*, and the plugin’s age and joins. The collector’s /metrics carries tandem_host_*. Table 32.1 lists every family registered in the code.