Best Gemma 4 12B quant for a 16GB GPU
Short answer: Q8_0 from unsloth. Of the 6 Gemma 4 12B files we tested that fit fully in 16GB with a 8192-token context, it has the highest measured accuracy (89.2% on GSM8K + MMLU + ARC-C, 95% CI 86.2–91.6). KLD pending: ranked by accuracy for now.
- File size
- 11.80 GB
- KLD
- pending
- Accuracy (500 items)
- 89.2% (86.2–91.6)
- Tool calls
- 87.0%
- Accuracy
- tied with best
- Generation
- 14 t/s GPU+RAM split
Speed was measured split on our 12GB test GPU. This file fits fully in 16GB, so expect it to run noticeably faster on your card.
gemma-4-12b-it-Q8_0.gguf- File size
- 9.11 GB
- KLD
- pending
- Accuracy (500 items)
- 88.2% (85.1–90.7)
- Tool calls
- 86.0%
- Accuracy
- tied with best
- Generation
- 57 t/s full GPU
- File size
- 4.34 GB
- KLD
- pending
- Accuracy (500 items)
- 67.2% (63.0–71.2)
- Tool calls
- 58.5%
- Accuracy
- ▼ below best on GSM8K, MMLU, ARC-C
- Generation
- 98 t/s full GPU
Other budgets: 8GB12GB24GB · all 6 Gemma 4 12B quants
How we picked
We estimate VRAM need as file size + f16 KV cache for 8192 tokens + about 0.6GB of compute buffers, and keep 0.5GB free for your desktop. Among files that fit, we rank by mean KL divergence against the BF16 model's logits, then by tool-call score. If you need a longer context, step down one quant level or quantize the KV cache (-ctk q8_0 -ctv q8_0). Methodology.
GPU too small? Rent one.
Run a 32B model at Q6/Q8, or a 70B, on a cloud GPU for an hour or two instead of buying hardware.
Cloud GPU pods with 24-80GB cards (RTX 4090 / 5090 / A100 / H100) for big quants or BF16 reference runs.
Rent on RunPodGPU marketplace for renting a 24-48GB card for an evening of testing.
Rent on Vast.aiReferral links: QuantDiff may earn credit if you sign up. Referral partners have no say in scores or rankings.