7Community weights

A community pack is a set of weights someone else published on Hugging Face that TandemLLM can load: a smaller quantisation, or a fine-tune of a model we serve. Since 0.8.2 the setup guide lists these packs under their model, after our own profiles, and ops/use-profile.sh runs one like any other profile. Listing a pack takes one JSON file in the repository. The first entry is ThinkingCap Qwen3.8-27B Kearuga by 0xWhiteMage, an EXL3 pack at 4.0 bits a weight.

7.1 The packs in the setup guide

The weights step of the setup guide (#/setup) shows a Community weights section below our profiles, one card per pack. A card shows the maker’s GitHub avatar and @login, the networks the entry lists, the fine-tune it comes from, the format, the bits a weight and the download size. It also lists the modalities the pack holds, in grey with the reason when TandemLLM cannot serve one yet, and which of our drafters it runs with. Once a maintainer has run the pack, its card shows the measured speed too. A badge gives the license, another the status, and a link leads to the Hugging Face repository.

You can search over names, authors, handles, tags and the fine-tune’s source, and sort by five keys, newest first by default. “Use” selects a pack the way a radio card selects one of our profiles, as community/<id>. Every later step takes that name, Wren included. A pack that cannot run images turns the Vision tool off, as the server would. “Add yours” opens the issue form of Section 7.4 with the model filled in.

7.2 Where the list comes from

The list is models/community-index.json, one entry per file under models/<model>/community/. A box asks GitHub for that file on the public repository’s main branch at two moments only: when the weights step opens and when you press Refresh (POST /v1/setup/community/sync). Nothing fetches it on a timer. The box sends the ETag of its last copy, so an unchanged list costs one 304 answer.

When the fetch fails, the page shows the box’s last copy (~/.tandemllm/community/index.json). A box that never fetched one shows the list that came with its release. Look at the section’s head to see which source the list came from and which entries it dropped. A 404 from GitHub is not an error: the repository has published no list yet, and the page shows a quiet note, “no published list yet”, beside the bundled list. A real failure is one short line without the URL, such as “GitHub could not be reached”; the server’s log has the URL and the detail. QSE_COMMUNITY_URL points the box at another index, and QSE_COMMUNITY_URL=off never fetches.

A fetched index is untrusted. One with the wrong shape never replaces the cached copy, and an entry reaches the page only after it passes the registry’s checks, those of Section 7.5.

7.3 Running a pack

~/TandemLLM/run/ops/use-profile.sh community/thinkingcap-27b-kearuga-exl3-4.00 --dry-run
~/TandemLLM/run/ops/use-profile.sh community/thinkingcap-27b-kearuga-exl3-4.00

The script writes the pack’s profile from its entry to run/community/<id>.env, which serve.env reads. It downloads the pack at the commit the entry pins into ~/.tandemllm/community/<id>, checks every weight file against the sha256 Hugging Face publishes, and fetches our draft models when they are missing. Then it stops the engine, starts it on the pack and sends one test chat. The checks are those of our own profiles: it refuses a download that would leave less than 60 GB of disk free, and it rolls back to the previous profile when the start or the test chat fails. --dry-run prints every URL and size and writes the profile into a folder of its own.

The memory estimate for a pack comes from our own profile of the same format and the nearest size: exl3-4.0 for a 4.0 bpw EXL3 pack. It stays an estimate until a maintainer has loaded the pack.

7.4 Submitting a pack

Send a pull request that adds one file: models/<model>/community/<id>.json. The file name is the id. The first entry, models/qwen3.8-27b/community/thinkingcap-27b-kearuga-exl3-4.00.json, is a good one to copy, and the JSON Schema is models/schema/community.schema.json. If you would rather not write JSON, open an issue with the “Add a community weight pack” form (.github/ISSUE_TEMPLATE/community-quant.yml) and a maintainer writes the file.

The fields that matter most:

• hf.repo and hf.revision: a public repository, and the full 40-character commit. A branch name is refused, because a branch can change after the review.
• format and bpw: only exl3 passes today. The schema knows three more formats, which the check refuses until TandemLLM can load them.
• load: standalone when the pack is the whole model (Section 9.8), or overlay when it holds EXL3 projections that go over the base checkpoint.
• modalities: what works in TandemLLM today. A standalone pack is text only for now, even with a vision tower inside; list image in modalities_in_pack instead. An overlay keeps the base checkpoint’s vision tower.
• drafters: tandem-exl3 for our EXL3 drafters; docs/community-weights.md lists the other two values.
• author.github, your GitHub user name, whose avatar the card shows. Leave status at submitted.

docs/community-weights.md lists every field. The registry refuses a pack that is private or gated, a branch in place of a commit, a format TandemLLM cannot load, a base model other than qwen3.8-27b, a modality the load path cannot serve, a link that is not https, markup in a text field, and a license that does not let others download and run the weights.

7.5 How verification works

Verification has two parts. On the pull request, the community check runs python -m server.setup.registry --check and python -m server.setup.community --index --check, which hold the file to the schema and to the rules above. Then a maintainer downloads the pack at the pinned commit and runs it on a DGX Spark. If it loads and answers, the maintainer fills verified (the build, the device, the date, tok/s per kind of text, tokens per round, a note) and sets status to verified. A pack that does not load stays submitted, or becomes broken with a note.

After the merge, a maintainer regenerates the index (python -m server.setup.community --index). Every box sees the pack the next time someone opens the weights step or presses Refresh.