Which quant should I run?

1. Does it fit?

Memory need ≈ file size (the real size of the GGUF / MLX weights on Hugging Face) + KV cache + 0.6 GB of compute buffers and runtime overhead.

• KV cache = context length × the model's bytes per token at f16 (from its attention config: layers × KV heads × head size × 2 for K and V × 2 bytes). Granite 4.2 8B: 160 KB/token (1.25 GB at 8k); Qwen3.5 4B: 32 KB/token (0.25 GB at 8k); Qwen3.5 9B: 32 KB/token (0.25 GB at 8k). Hybrid models (Qwen3.5) only count their full-attention layers; the fixed-size state of their linear-attention layers doesn't grow with context and is not counted separately.
• GPU: usable = VRAM − 0.5 GB kept free for your desktop. Mac: usable ≈ ⅔ of unified memory up to 36 GB, ¾ above (macOS's default GPU memory limit; it can be raised with sudo sysctl iogpu.wired_limit_mb=…, at your own risk).
• This is an estimate. Real use varies with the runtime, batch size and flash attention. Quantizing the KV cache (-ctk q8_0 -ctv q8_0) roughly halves the KV part. If you're right at the limit, step down one quant.

2. Which is best?

• Each use case has one measured score that's comparable across models: general chat = MMLU + ARC-Challenge (350 questions pooled), math = GSM8K (150), coding / tool calls = our 200 BFCL tool-call cases. We compute a 95% interval (Wilson) for each.
• Quants whose interval overlaps the best one's are tied. The rest are marked ▼ below and never picked.
• Within each tied model, we take the quant with the smallest diff from the original (lowest mean KLD vs BF16). If a smaller file's KLD is within its 95% margin, we take the smaller one.
• The top pick is the tied option with the highest score. Because the margins are wide at these sample sizes, we also show the smaller & faster tied option: we can't measure it as worse on this test.
• Speeds are real measurements: RTX 5070 12GB (llama-bench) for GGUF, Apple M2 Pro (mlx-lm) for MLX. Untested numbers say "not tested yet", never a guess.