Best Gemma 4 12B quant for your GPU

6 quantized files of google/gemma-4-12B-it (11.91B): accuracy, tool calls and speed measured; KLD vs the BF16 original pending.

google/gemma-4-12B-it (Apache-2.0). 48 layers: 40 sliding-window layers (1024-token window, 8 KV heads x 256) + 8 global attention layers (1 KV head x 512), so the KV cache is small. Text-only GGUF quants from unsloth (vision projector not included). Gemma 4 often bolds its answer letter (**A**); since Oct 22 our letter scorer ignores markdown emphasis for every model (see methodology).

KLD pending. KL divergence vs the BF16 reference is queued on our Apple M2 Pro test Mac and will be added here when measured. Until then, picks for this model are ranked by measured accuracy (GSM8K + MMLU + ARC-C, 500 items, 95% CI) with tool calls as tie-break.

Quality vs file size

Left: KLD pending. Right: accuracy on all 500 items (GSM8K + MMLU + ARC-C). Error bars = 95% intervals. Download PNG to share.

Gemma 4 12B: KLD and accuracy vs file size for 6 GGUF quants, with 95% error bars

Not sure which fits? Use the quant picker →

Recommendation by VRAM

"Best" = highest measured accuracy (500 items pooled) among files that fit fully on the GPU (file + 8192-token KV cache), tool-call score as tie-breaker. KLD pending.

8GB GPU

Best accuracy that fits

Q3_K_M by unsloth

File size: 5.30 GB
KLD: pending
Accuracy (500 items): 85.0% (81.6–87.9)
Tool calls: 83.5%
Accuracy: tied with best
Generation: 78 t/s full GPU

gemma-4-12b-it-Q3_K_M.gguf

Fastest that fits: UD-Q2_K_XL (unsloth, 98 t/s, acc 67.2%, KLD pending)

12GB GPU

Best accuracy that fits

Q6_K by unsloth

File size: 9.11 GB
KLD: pending
Accuracy (500 items): 88.2% (85.1–90.7)
Tool calls: 86.0%
Accuracy: tied with best
Generation: 57 t/s full GPU

gemma-4-12b-it-Q6_K.gguf

Fastest that fits: UD-Q2_K_XL (unsloth, 98 t/s, acc 67.2%, KLD pending)

16GB GPU

Best accuracy that fits

Q8_0 by unsloth

File size: 11.80 GB
KLD: pending
Accuracy (500 items): 89.2% (86.2–91.6)
Tool calls: 87.0%
Accuracy: tied with best
Generation: 14 t/s GPU+RAM split

Speed was measured split on our 12GB test GPU. This file fits fully in 16GB, so expect it to run noticeably faster on your card.

gemma-4-12b-it-Q8_0.gguf

Fastest that fits: UD-Q2_K_XL (unsloth, 98 t/s, acc 67.2%, KLD pending)

All Gemma 4 12B quants

Quant	Publisher	Size	Fits	KLD mean	KLD p99	Same top	Tool calls	GSM8K	MMLU	ARC-C	PP t/s	Gen t/s	Run
UD-Q2_K_XL	unsloth imatrix	4.34 GB	8GB+	pending	—	—	58.5%	71.3 ▼ below 64–78	63.0 ▼ below 56–69	68.7 ▼ below 61–76	2509	98	settings & logs
Q3_K_M	unsloth imatrix	5.30 GB	8GB+	pending	—	—	83.5%	94.0 ≈ 89–97	76.0 ≈ 70–81	88.0 ≈ 82–92	2940	78	settings & logs
Q4_K_M	unsloth imatrix	6.63 GB	12GB+	pending	—	—	85.5%	96.0 92–98	80.0 74–85	88.7 ≈ 83–93	3174	72	settings & logs
Q5_K_M	unsloth imatrix	7.84 GB	12GB+	pending	—	—	87.5%	94.7 ≈ 90–97	78.5 ≈ 72–84	90.7 ≈ 85–94	3082	63	settings & logs
Q6_K	unsloth imatrix	9.11 GB	12GB+	pending	—	—	86.0%	94.7 ≈ 90–97	78.5 ≈ 72–84	94.7 ≈ 90–97	2668	57	settings & logs
Q8_0	unsloth imatrix	11.80 GB	16GB+	pending	—	—	87.0%	96.0 92–98	79.0 ≈ 73–84	96.0 92–98	1479	14 GPU+RAM split	settings & logs

Accuracy cells: % correct, with the 95% confidence interval (Wilson) underneath. ≈ = statistically tied with the best quant of the same model (intervals overlap). ▼ below = clearly worse than that model's best quant (intervals do not overlap). Ties are not ranked: sorting by an accuracy column puts tied quants together and orders them by KLD. How accuracy is measured.

GB = GiB (1024³ bytes). KLD, tool calls and speeds from Ghost (RTX 5070 12GB, CUDA, Ryzen 7 5800X, 64GB DDR4). On a different GPU, expect generation speed to scale roughly with memory bandwidth. Task accuracy (GSM8K / MMLU / ARC-C, % correct on fixed subsets of 150 / 200 / 150 items, frozen suite quantdiff-lmeval-v1) was measured with the same GGUF files and llama.cpp build on Ghost (CUDA, NVIDIA GeForce RTX 5070 12 GB (64 GB RAM)), minipro (Metal, Apple M2 Pro (Mac mini), 12-core CPU, 19-core GPU, 32 GB unified memory). Accuracy depends on the weights, not on the GPU, but small numeric differences between backends can flip individual answers. With these sample sizes, differences under about 5 points are within noise (± shown on run pages). Methodology.

GPU too small? Rent one.

Run a 32B model at Q6/Q8, or a 70B, on a cloud GPU for an hour or two instead of buying hardware.

RunPod

Cloud GPU pods with 24-80GB cards (RTX 4090 / 5090 / A100 / H100) for big quants or BF16 reference runs.

Rent on RunPod

Vast.ai

GPU marketplace for renting a 24-48GB card for an evening of testing.

Rent on Vast.ai

Your GPU cloud here

Sponsor slot. Sponsors never influence scores or rankings.

Sponsor QuantDiff

Referral links: QuantDiff may earn credit if you sign up. Referral partners have no say in scores or rankings.