31Metrics and the ledger

31.1 Prometheus metrics

GET /metrics serves the families below in the Prometheus text format, to anyone while no token is set, else behind the metrics or admin token, or to a request from the machine itself. ops/monitoring/ has a scrape configuration, alert rules with their tests, and a Grafana dashboard generated by tools/grafana_dashboard.py. With the metrics token set, a scrape job needs it as a bearer token:

- job_name: tandemllm
  authorization: {type: Bearer, credentials_file: /etc/prometheus/tandem-metrics.token}
  static_configs: [{targets: ['gpu-host:8000']}]

The engine’s families never count a relayed request, and the proxy’s families never see the engine’s own. The table lists every family registered with a literal name in the code; families built at scrape time from a table (the mapped qse_backend_* engine families, the cache families by cache label) are listed by their prefix in the help text of the page itself.

31.2 The usage ledger

server/ledger.py writes one SQLite row per request: counts, timings, the finish reason, the cache source, a client id. Never the prompt, the answer, a tool name or argument, a conversation id, the user field, an IP address or a raw token. Refused requests (503, 429) and rejected ones (400) are rows too, with null token fields, so the request count is honest.

It never slows or fails a request. A row is put on a bounded queue (10,000), a writer thread commits in batches at most every second, and a disk error drops that batch, counts it and logs it at most once a minute. Rows older than --usage-retention-days (at least 400) are deleted once a day, and once an ISO week an online backup goes beside the file, the newest 8 kept. The server takes the path from --usage-ledger only, never from the environment, so a benchmark server started with serve.env exported never writes the operator’s ledger. Migrations are forward only and in place, and keep every row.

The conductor’s journal shares the file: conductor_events has 53 columns (id, ts_ms, request_id, session_key, kind, task, tier, mode, sp, rule, trigger, score, decision, action, cap, tier1_model, question_id, probabilities, tier1_ms, hidden_text_id, hidden_tokens, tokens_at_cut, position, outcome, next_checker, policy_version, client_kind, model, profile, profile_version, router_version, class_id, confidence, source, kept, stages, classifier_ms, effort, budget, budget_source, thinking, decisions, injected, replayed, shadow, digest, label, label_note, labelled_at, acted, changed, detail, extra), and conductor_texts keeps every hidden text once, by its hash.

A row’s speed figures follow the benchmark row’s formula, and a replay or a side-lane request has no decode rate. A relayed request has null queue, prefill, blocks, draft counts and cache source unless a plugin reported them, in which case l2_join says how the row was joined. A quick look:

sqlite3 ~/.qwen38-spark-engine/usage/ledger.sqlite3 \
  "select datetime(ts_ms/1000,'unixepoch','localtime'), finish_reason, prompt_tokens,
          completion_tokens, round(decode_tps,1), cache_source from requests order by id desc limit 10"