QuantDiff

Best Granite 4.2 8B quant for your GPU

8 quantized files of ibm-granite/granite-4.2-8b (8.79B), each diffed against the BF16 original.

ibm-granite/granite-4.2-8b (Apache-2.0). Dense 40-layer transformer with grouped-query attention (8 KV heads). Measured here: on the 60 parallel and parallel_multiple tool-call cases it emits only one call per turn, so all 8 quants scored 0 to 2 of 60 there (Q8_0: 0/60, bartowski Q4_K_M: 1/60), even with parallel_tool_calls=true. Its total tool-call score is 60.5 to 62.5% vs 82 to 92.5% for Qwen3.5 9B.

Recommendation by VRAM

"Best" = smallest diff (lowest mean KLD) among files that fit fully on the GPU (file + 8192-token KV cache), with tool-call score as tie-breaker.

8GB GPU

Smallest diff that fits
Q4_K_M by bartowski
File size
5.16 GB
KLD (mean)
0.0729
Same top token
91.0%
Tool calls
62.5%
Generation
102 t/s full GPU
granite-4.2-8b-Q4_K_M.gguf

Fastest that fits: Q2_K (bartowski, 146 t/s, KLD 0.4598)

12GB GPU

Smallest diff that fits
Q8_0 by bartowski
File size
8.70 GB
KLD (mean)
0.0082
Same top token
97.8%
Tool calls
61.0%
Generation
65 t/s full GPU
granite-4.2-8b-Q8_0.gguf

Fastest that fits: Q2_K (bartowski, 146 t/s, KLD 0.4598)

16GB GPU

Smallest diff that fits
Q8_0 by bartowski
File size
8.70 GB
KLD (mean)
0.0082
Same top token
97.8%
Tool calls
61.0%
Generation
65 t/s full GPU
granite-4.2-8b-Q8_0.gguf

Fastest that fits: Q2_K (bartowski, 146 t/s, KLD 0.4598)

All Granite 4.2 8B quants

QuantPublisherSizeFitsKLD meanKLD p99Same topTool callsPP t/sGen t/sRun
Q8_0 bartowski imatrix 8.70 GB 12GB+ 0.0082 0.063 97.8% 61.0% 4574 65 settings & logs
Q6_K bartowski imatrix 7.00 GB 12GB+ 0.0136 0.140 96.2% 60.5% 3679 78 settings & logs
Q5_K_M mradermacher 5.82 GB 12GB+ 0.0579 0.746 92.1% 62.5% 4220 92 settings & logs
Q4_K_M bartowski imatrix 5.16 GB 8GB+ 0.0729 0.928 91.0% 62.5% 4236 102 settings & logs
IQ4_XS bartowski imatrix 4.60 GB 8GB+ 0.0846 1.025 89.5% 62.5% 4808 113 settings & logs
i1-Q4_K_M mradermacher imatrix 4.98 GB 8GB+ 0.0856 1.188 90.2% 60.5% 4248 105 settings & logs
IQ3_M bartowski imatrix 3.95 GB 8GB+ 0.2233 2.836 82.9% 60.5% 4402 120 settings & logs
Q2_K bartowski imatrix 3.33 GB 8GB+ 0.4598 5.202 75.0% 61.0% 3315 146 settings & logs

GB = GiB (1024³ bytes). Speeds from llama-bench on Ghost (RTX 5070 12GB, Ryzen 7 5800X, 64GB DDR4). On a different GPU, expect generation speed to scale roughly with memory bandwidth.

GPU too small? Rent one.

Run a 32B model at Q6/Q8, or a 70B, on a cloud GPU for an hour or two instead of buying hardware.

RunPod

Per-second GPU pods. RTX 4090 / 5090 / A100 / H100 for big quants or BF16 reference runs.

Link coming soon
Vast.ai

Marketplace GPUs, often the cheapest way to try a 24-48GB card for an evening.

Link coming soon
Your GPU cloud here

Sponsor slot. Sponsors never influence scores or rankings.

Sponsor QuantDiff