Best Gemma 4 12B quant for a 8GB GPU

Short answer: Q3_K_M from unsloth. Of the 2 Gemma 4 12B files we tested that fit fully in 8GB with a 8192-token context, it has the highest measured accuracy (85.0% on GSM8K + MMLU + ARC-C, 95% CI 81.6–87.9). KLD pending: ranked by accuracy for now.

Smallest diff that fits 8GB

Q3_K_M by unsloth

File size: 5.30 GB
KLD: pending
Accuracy (500 items): 85.0% (81.6–87.9)
Tool calls: 83.5%
Accuracy: tied with best
Generation: 78 t/s full GPU

gemma-4-12b-it-Q3_K_M.gguf

Runner-up

UD-Q2_K_XL by unsloth

File size: 4.34 GB
KLD: pending
Accuracy (500 items): 67.2% (63.0–71.2)
Tool calls: 58.5%
Accuracy: ▼ below best on GSM8K, MMLU, ARC-C
Generation: 98 t/s full GPU

gemma-4-12b-it-UD-Q2_K_XL.gguf

Fastest that fits

UD-Q2_K_XL by unsloth

File size: 4.34 GB
KLD: pending
Accuracy (500 items): 67.2% (63.0–71.2)
Tool calls: 58.5%
Accuracy: ▼ below best on GSM8K, MMLU, ARC-C
Generation: 98 t/s full GPU

gemma-4-12b-it-UD-Q2_K_XL.gguf

Other budgets: 12GB 16GB 24GB · all 6 Gemma 4 12B quants

How we picked

We estimate VRAM need as file size + f16 KV cache for 8192 tokens + about 0.6GB of compute buffers, and keep 0.5GB free for your desktop. Among files that fit, we rank by mean KL divergence against the BF16 model's logits, then by tool-call score. If you need a longer context, step down one quant level or quantize the KV cache (-ctk q8_0 -ctv q8_0). Methodology.

GPU too small? Rent one.

Run a 32B model at Q6/Q8, or a 70B, on a cloud GPU for an hour or two instead of buying hardware.

RunPod

Cloud GPU pods with 24-80GB cards (RTX 4090 / 5090 / A100 / H100) for big quants or BF16 reference runs.

Rent on RunPod

Vast.ai

GPU marketplace for renting a 24-48GB card for an evening of testing.

Rent on Vast.ai

Your GPU cloud here

Sponsor slot. Sponsors never influence scores or rankings.

Sponsor QuantDiff

Referral links: QuantDiff may earn credit if you sign up. Referral partners have no say in scores or rankings.