What runs here once the GPU lands
Six frozen examples scored live and checked against their audited labels.
The model's frozen 324-example evaluation re-run here, compared to the published bf16 reference (292/324 = 90.1%).
Group-64 affine int8 simulated on sampled weight matrices: rel-RMS error and cosine similarity.
The runner (app.py) is coded for ZeroGPU and switches on with a one-line
README change once the grant is approved. An alternative for immediate runs: run_tests_cuda.py
executes the same suite on Hugging Face Jobs (hf jobs run, needs prepaid credits; the full
324-example pass costs well under a dollar).
Live now: the official Nimble demo
The official demo Space (by hugging-apps), self-hosting the same model.
First load after idle takes a couple of minutes while the GPU boots.
Open in a new tab.
Apple-Silicon 8-bit build — local validation (in progress)
The upstream repository ships no quantization path; this project built one for a 16 GB M5 Mac
(the bf16 weights alone are ~18 GB): the released LoRA fused into
mlx-community/Qwen3.5-9B-MLX-8bit — an 8.9 GB artifact at 9.2 GiB peak active
memory, with weight-level rel-RMS error ~0.7–1.1% and cosine ≥ 0.99994 on sampled tensors, every file
sha-verified against publisher manifests. The model's own frozen holdout is being replayed on the
local machine with the identical scoring protocol — now complete:
| Primitive | 8-bit local (324) | Published bf16 (324) |
|---|---|---|
| All | 289/324 · 89.2% | 292/324 · 90.1% |
| Choice | 122/146 · 83.6% | 124/146 · 84.9% |
| Yes/No | 112/114 · 98.2% | 112/114 · 98.2% |
| Rating | 55/64 · 85.9% | 56/64 · 87.5% |
Complete run, all 324 examples: −3 examples versus the published bf16 numbers, within sampling noise (~0.6σ). Median 1.70 s/example at 9.76 GiB peak memory. The Space re-runs the same suite on GPU once compute is attached.