QuantDiff

Best Spark-X2.6 4B quant for a 12GB GPU

Short answer: Q8_0 from bartowski . Of the 10 Spark-X2.6 4B files we tested that fit fully in 12GB with a 8192-token context, it has the smallest diff from the BF16 model (mean KLD 0.0027, same top token 98.0%).

Smallest diff that fits 12GB
Q8_0 by bartowski
File size
4.07 GB
(mean)
0.0027
98.0%
79.5%
Accuracy
Generation ()
39 t/s on Apple M2 Pro 32GB full GPU
Spark-X2.6-4B-Q8_0.gguf
How to run this file

From bartowski/Spark-X2.6-4B-GGUF, file Spark-X2.6-4B-Q8_0.gguf.

(downloads it, then starts a chat; type /bye to quit):

ollama run hf.co/bartowski/Spark-X2.6-4B-GGUF:Spark-X2.6-4B-Q8_0.gguf

: open the Discover tab, search bartowski/Spark-X2.6-4B-GGUF, pick the Q8_0 download, then load it from the Chat tab.

Step-by-step guide

Runner-up
Q6_K by bartowski
File size
3.36 GB
(mean)
0.0099
95.7%
79.5%
Accuracy
Generation ()
44 t/s on Apple M2 Pro 32GB full GPU
Spark-X2.6-4B-Q6_K.gguf
How to run this file

From bartowski/Spark-X2.6-4B-GGUF, file Spark-X2.6-4B-Q6_K.gguf.

(downloads it, then starts a chat; type /bye to quit):

ollama run hf.co/bartowski/Spark-X2.6-4B-GGUF:Spark-X2.6-4B-Q6_K.gguf

: open the Discover tab, search bartowski/Spark-X2.6-4B-GGUF, pick the Q6_K download, then load it from the Chat tab.

Step-by-step guide

Fastest that fits
IQ4_XS by bartowski
File size
2.18 GB
(mean)
0.1288
84.6%
76.0%
Accuracy
Generation ()
62 t/s on Apple M2 Pro 32GB full GPU
Spark-X2.6-4B-IQ4_XS.gguf
How to run this file

From bartowski/Spark-X2.6-4B-GGUF, file Spark-X2.6-4B-IQ4_XS.gguf.

(downloads it, then starts a chat; type /bye to quit):

ollama run hf.co/bartowski/Spark-X2.6-4B-GGUF:Spark-X2.6-4B-IQ4_XS.gguf

: open the Discover tab, search bartowski/Spark-X2.6-4B-GGUF, pick the IQ4_XS download, then load it from the Chat tab.

Step-by-step guide

Other budgets: 8GB16GB24GB · all 10 Spark-X2.6 4B quants

How we picked

We estimate VRAM need as file size + f16 KV cache for 8192 tokens + about 0.6GB of compute buffers, and keep 0.5GB free for your desktop. Among files that fit, we rank by mean KL divergence against the BF16 model's logits, then by tool-call score. If you need a longer context, step down one quant level or quantize the KV cache (-ctk q8_0 -ctv q8_0). Methodology.

GPU too small? Rent one.

Run a 32B model at Q6/Q8, or a 70B, on a cloud GPU for an hour or two instead of buying hardware.

RunPod

Cloud GPU pods with 24-80GB cards (RTX 4090 / 5090 / A100 / H100) for big quants or BF16 reference runs.

Rent on RunPod
Vast.ai

GPU marketplace for renting a 24-48GB card for an evening of testing.

Rent on Vast.ai
Your GPU cloud here

Sponsor slot. Sponsors never influence scores or rankings.

Sponsor QuantDiff

Referral links: QuantDiff may earn credit if you sign up. Referral partners have no say in scores or rankings.