Which quant should I run?
Pick your GPU or Mac (or type its memory), your context length and what you'll use the model for. We'll show the tested that fit, ranked by measured quality, with real speeds and a link to every run's raw logs. New to this? Start here.
Loading measured results…
How this works
1. Does it fit?
Memory need ≈ file size (the real size of the GGUF / MLX weights on Hugging Face) + KV cache + 0.6 GB of compute buffers and runtime overhead.
- KV cache = context length × the model's bytes per token at f16 (from its attention config: layers × KV heads × head size × 2 for K and V × 2 bytes). Gemma 4 12B: 64 KB/token (0.50 GB at 8k); Granite 4.2 8B: 160 KB/token (1.25 GB at 8k); Ornith 1.5 9B: 32 KB/token (0.25 GB at 8k); Qwen3.5 4B: 32 KB/token (0.25 GB at 8k); Qwen3.5 9B: 32 KB/token (0.25 GB at 8k); Qwen3.8 27B: 64 KB/token (0.50 GB at 8k). Hybrid models (Qwen3.5) only count their full-attention layers; the fixed-size state of their linear-attention layers doesn't grow with context and is not counted separately.
- GPU: usable = VRAM − 0.5 GB kept free for your desktop.
- Mac: macOS lets the GPU wire only part of unified memory by default: about ⅔ on Macs with up to 32 GB and ¾ above 32 GB (so 16 GB → ~10.7 GB, 36 GB → 27 GB, 64 GB → 48 GB, 512 GB → 384 GB). Apple doesn't document this; the rule comes from the decompiled GPU driver (reserve 33.3%, or 25% above 32 GB) quoted in llama.cpp discussion #2182, and matches reported values such as a 36 GB MacBook showing ~29 GB (ollama #2370). llama.cpp prints the real figure as
recommendedMaxWorkingSetSize; MLX reports it inmx.metal.device_info(). You can raise it until the next reboot withsudo sysctl iogpu.wired_limit_mb=<MB>(MLX docs); leave several GB for macOS or the system can stall. - This is an estimate. Real use varies with the runtime, batch size and flash attention. Quantizing the KV cache (
-ctk q8_0 -ctv q8_0) roughly halves the KV part. If you're right at the limit, step down one quant.
2. Which is best?
- Each use case has one measured score that's comparable across models: general chat = MMLU + ARC-Challenge (350 questions pooled), math = GSM8K (150), coding / tool calls = our 200 BFCL tool-call cases. We compute a 95% interval (Wilson) for each.
- Quants whose interval overlaps the best one's are tied. The rest are marked ▼ below and never picked.
- Within each tied model, we take the quant with the smallest diff from the original (lowest mean KLD vs BF16). If a smaller file's KLD is within its 95% margin, we take the smaller one.
- The top pick is the tied option with the highest score. Because the margins are wide at these sample sizes, we also show the smaller & faster tied option: we can't measure it as worse on this test.
- Speeds are real measurements on our machines: RTX 5070 12GB (llama-bench) for GGUF, Apple M2 Pro (mlx-lm) for MLX. On any other hardware you'll see a clearly labelled estimate (see 3). Untested quality numbers say "not tested yet", never a guess.
3. Speed on your hardware (estimate)
Generating a token means reading the model's weights from memory once, so decode speed is mostly set by memory bandwidth. For hardware we haven't run, we estimate:
tokens/s ≈ 1 ÷ ( GB read per token ÷ (eff × bandwidth GB/s) + t0 )
- GB read per token = the quant's file size (for MoE models, scaled by active ÷ total parameters).
- eff = 1.062 and t0 = 2.33 ms per token (fixed per-token overhead) are fitted to our 36 full-GPU llama-bench runs on the RTX 5070 12GB (672 GB/s). eff above 1 just means fewer bytes are read per token than the file holds (for example, unused rows of the token-embedding table); treat eff and t0 as fitted constants, not physics.
- Error on those runs: 5.6% mean, 24.9% worst. Holding out each model in turn and fitting on the others: 8.3% mean, 28.9% worst. The plain version, tokens/s ≈ eff × bandwidth ÷ file size with one median eff (0.832), is off by 12% on average and up to 54% (it overestimates small models badly), so we add t0.
- Apple Silicon: not calibrated yet. Until our M2 Pro MLX runs are in, Mac estimates reuse the RTX 5070 fit and are less certain.
- All calibration runs come from one machine per backend, so the error on other GPUs (different cache sizes, compute, drivers, ROCm / Vulkan / SYCL) is not measured and is probably larger. Prompt processing (prefill) is compute-bound and is not estimated. Bandwidths are the vendors' peak figures (sources); edit the field if you know better.
Calibration data: measured vs formula on the RTX 5070 12GB
| Quant | GB / token | Measured t/s | Formula t/s | Error |
|---|---|---|---|---|
| gemma-4-12b Q6_K unsloth | 9.79 | 56.9 | 62.4 | 9.6% |
| gemma-4-12b Q5_K_M unsloth | 8.41 | 62.9 | 70.9 | 12.7% |
| gemma-4-12b Q4_K_M unsloth | 7.12 | 71.9 | 81.3 | 13.1% |
| gemma-4-12b Q3_K_M unsloth | 5.69 | 77.7 | 97.1 | 24.9% |
| gemma-4-12b UD-Q2_K_XL unsloth | 4.66 | 98.4 | 112.9 | 14.8% |
| granite-4.2-8b Q8_0 bartowski | 9.35 | 64.8 | 64.9 | 0.0% |
| granite-4.2-8b Q6_K bartowski | 7.52 | 78.5 | 77.8 | -0.9% |
| granite-4.2-8b Q5_K_M mradermacher | 6.25 | 92.3 | 90.2 | -2.2% |
| granite-4.2-8b Q4_K_M bartowski | 5.54 | 101.6 | 99.2 | -2.4% |
| granite-4.2-8b i1-Q4_K_M mradermacher | 5.35 | 104.8 | 101.9 | -2.8% |
| granite-4.2-8b IQ4_XS bartowski | 4.94 | 112.5 | 108.2 | -3.9% |
| granite-4.2-8b IQ3_M bartowski | 4.24 | 119.6 | 121.0 | 1.1% |
| granite-4.2-8b Q2_K bartowski | 3.58 | 145.7 | 136.3 | -6.4% |
| ornith-1.5-9b Q8_0 ornith-ai | 9.79 | 65.2 | 62.4 | -4.3% |
| ornith-1.5-9b Q6_K ornith-ai | 7.56 | 83.0 | 77.4 | -6.7% |
| ornith-1.5-9b Q5_K_M ornith-ai | 6.64 | 89.7 | 86.0 | -4.1% |
| ornith-1.5-9b Q4_K_M ornith-ai | 5.78 | 99.4 | 96.0 | -3.5% |
| qwen3.5-4b Q8_0 unsloth | 4.48 | 114.0 | 116.2 | 2.0% |
| qwen3.5-4b Q6_K bartowski | 3.81 | 129.9 | 130.6 | 0.6% |
| qwen3.5-4b Q4_K_M bartowski | 3.01 | 151.8 | 152.8 | 0.7% |
| qwen3.5-4b UD-Q4_K_XL unsloth | 2.91 | 150.6 | 156.2 | 3.7% |
| qwen3.5-4b Q4_K_M unsloth | 2.74 | 156.3 | 162.2 | 3.8% |
| qwen3.5-4b i1-Q4_K_M mradermacher | 2.71 | 157.9 | 163.4 | 3.5% |
| qwen3.5-4b IQ3_M bartowski | 2.48 | 161.4 | 172.3 | 6.8% |
| qwen3.5-4b UD-Q2_K_XL unsloth | 1.94 | 186.5 | 198.3 | 6.3% |
| qwen3.5-9b Q8_0 unsloth | 9.53 | 66.4 | 63.8 | -3.9% |
| qwen3.5-9b Q6_K unsloth | 7.46 | 82.0 | 78.3 | -4.5% |
| qwen3.5-9b Q5_K_M bartowski | 7.11 | 87.5 | 81.4 | -7.0% |
| qwen3.5-9b Q4_K_M bartowski | 6.17 | 97.8 | 91.2 | -6.8% |
| qwen3.5-9b UD-Q4_K_XL unsloth | 5.97 | 97.3 | 93.6 | -3.7% |
| qwen3.5-9b Q4_K_M unsloth | 5.68 | 101.0 | 97.3 | -3.7% |
| qwen3.5-9b i1-Q4_K_M mradermacher | 5.63 | 101.4 | 98.0 | -3.4% |
| qwen3.5-9b IQ4_XS bartowski | 5.50 | 110.6 | 99.7 | -9.9% |
| qwen3.5-9b i1-IQ4_XS mradermacher | 5.07 | 113.6 | 106.1 | -6.6% |
| qwen3.5-9b IQ3_M bartowski | 4.98 | 112.0 | 107.5 | -4.0% |
| qwen3.5-9b UD-Q2_K_XL unsloth | 4.12 | 133.0 | 123.5 | -7.1% |
Only quants we have measured appear here. Missing your model or quant? Request it. Full details: VRAM fit and ranking in the methodology.