QuantDiff

Best Qwen3.8 27B quant for a 24GB GPU

Short answer: UD-Q4_K_M from unsloth. Of the 4 Qwen3.8 27B files we tested that fit fully in 24GB with a 8192-token context, it has the highest measured accuracy (91.8% on GSM8K + MMLU + ARC-C, 95% CI 89.1–93.9). Accuracy and speed only, no KLD: reference too large.

Smallest diff that fits 24GB
UD-Q4_K_M by unsloth
File size
15.33 GB
no KLD: reference too large
Accuracy (500 items)
91.8% (89.1–93.9)
88.5%
Accuracy
Generation ()
9 t/s on Apple M2 Pro 32GB full GPU
Qwen3.8-27B-UD-Q4_K_M.gguf
How to run this file

From unsloth/Qwen3.8-27B-GGUF, file Qwen3.8-27B-UD-Q4_K_M.gguf.

(downloads it, then starts a chat; type /bye to quit):

ollama run hf.co/unsloth/Qwen3.8-27B-GGUF:Qwen3.8-27B-UD-Q4_K_M.gguf

: open the Discover tab, search unsloth/Qwen3.8-27B-GGUF, pick the UD-Q4_K_M download, then load it from the Chat tab.

Step-by-step guide

Runner-up
UD-Q5_K_M by unsloth
File size
18.41 GB
no KLD: reference too large
Accuracy (500 items)
91.8% (89.1–93.9)
86.5%
Accuracy
Generation ()
8 t/s on Apple M2 Pro 32GB full GPU
Qwen3.8-27B-UD-Q5_K_M.gguf
How to run this file

From unsloth/Qwen3.8-27B-GGUF, file Qwen3.8-27B-UD-Q5_K_M.gguf.

(downloads it, then starts a chat; type /bye to quit):

ollama run hf.co/unsloth/Qwen3.8-27B-GGUF:Qwen3.8-27B-UD-Q5_K_M.gguf

: open the Discover tab, search unsloth/Qwen3.8-27B-GGUF, pick the UD-Q5_K_M download, then load it from the Chat tab.

Step-by-step guide

Fastest that fits
UD-Q2_K_XL by unsloth
File size
9.15 GB
no KLD: reference too large
Accuracy (500 items)
90.4% (87.5–92.7)
87.5%
Accuracy
Generation ()
10 t/s on Apple M2 Pro 32GB full GPU
Qwen3.8-27B-UD-Q2_K_XL.gguf
How to run this file

From unsloth/Qwen3.8-27B-GGUF, file Qwen3.8-27B-UD-Q2_K_XL.gguf.

(downloads it, then starts a chat; type /bye to quit):

ollama run hf.co/unsloth/Qwen3.8-27B-GGUF:Qwen3.8-27B-UD-Q2_K_XL.gguf

: open the Discover tab, search unsloth/Qwen3.8-27B-GGUF, pick the UD-Q2_K_XL download, then load it from the Chat tab.

Step-by-step guide

Other budgets: 8GB12GB16GB · all 4 Qwen3.8 27B quants

How we picked

We estimate VRAM need as file size + f16 KV cache for 8192 tokens + about 0.6GB of compute buffers, and keep 0.5GB free for your desktop. Among files that fit, we rank by mean KL divergence against the BF16 model's logits, then by tool-call score. If you need a longer context, step down one quant level or quantize the KV cache (-ctk q8_0 -ctv q8_0). Methodology.

GPU too small? Rent one.

Run a 32B model at Q6/Q8, or a 70B, on a cloud GPU for an hour or two instead of buying hardware.

RunPod

Cloud GPU pods with 24-80GB cards (RTX 4090 / 5090 / A100 / H100) for big quants or BF16 reference runs.

Rent on RunPod
Vast.ai

GPU marketplace for renting a 24-48GB card for an evening of testing.

Rent on Vast.ai
Your GPU cloud here

Sponsor slot. Sponsors never influence scores or rankings.

Sponsor QuantDiff

Referral links: QuantDiff may earn credit if you sign up. Referral partners have no say in scores or rankings.