Methodology

Each quant is diffed against the original full-precision model on the token distribution (KL divergence and top-token agreement), tool calling and speed, plus an optional task-accuracy check. One machine, fixed settings, everything published. With the same file and the same llama.cpp build, you should be able to reproduce our numbers.

1. The core diff: KL divergence and top-token agreement vs the full-precision model

Benchmarks like MMLU are too noisy to separate a Q4_K_M from a Q5_K_M. KL divergence is not. For every token of a fixed text set, we compare the quant's full next-token probability distribution with the BF16 model's. Zero means identical.

• Reference: the original Hugging Face safetensors, converted by us to a BF16 GGUF with llama.cpp's convert_hf_to_gguf.py (same llama.cpp tag as the binaries). Its logits are generated once per model and cached. The BF16 model may run partly from system RAM, which is slower but exact.
• Text set quantdiff-kld-v1 (401,701 characters, sha256 ff19568d…): wikitext-2 test prose, CPython 3.12.7 standard-library code, and UltraChat test conversations, interleaved in ~2,000-character segments.
• Settings: context 512, 100 chunks, batch 512. The KLD runs use the tokens stored with the reference logits, so every quant sees exactly the same input.
• Reported: mean KLD (headline), median, 99th percentile, max, same top token % (how often the quant's most likely token matches BF16's), RMS Δp and perplexity of quant vs reference.

# once per model
llama-perplexity -m MODEL-BF16.gguf -f quantdiff-kld-v1.txt -c 512 -b 512 --chunks 100 \
  -ngl <auto> --kl-divergence-base MODEL.kld

# per quant
llama-perplexity -m QUANT.gguf --kl-divergence-base MODEL.kld --kl-divergence -c 512 -b 512 -ngl 999

Rough guide: mean KLD < 0.01 is near-lossless, 0.01-0.05 is good, and > 0.1 is noticeable drift. Compare quants within one model. Absolute values across different models and tokenizers aren't directly comparable.

2. Tool-calling reliability (BFCL subset)

Quantization often breaks structured output before it hurts prose. We use a fixed, seeded sample of 200 test cases from the Berkeley Function Calling Leaderboard (BFCL v4 data, gorilla commit f7cf735; set quantdiff-bfcl-v1, sha256 4d0e1667…): 60 simple_python, 40 multiple, 30 parallel, 30 parallel_multiple and 40 irrelevance (where the right answer is not calling a tool).

• Served with llama-server --jinja -c 4096 -np 1 (the model's own chat template) via the OpenAI-compatible tools API, temperature 0, seed 1234, max 1024 tokens, thinking disabled (chat_template_kwargs: {"enable_thinking": false}).
• Parallel calls: the first batch was sent with llama-server's default (parallel_tool_calls unset). A control re-run of one quant per model with parallel_tool_calls: true gave identical results on all 200 cases. Granite 4.2 makes one call per turn on the parallel categories either way, which is a real model/template behavior that its scores reflect.
• Scored with BFCL-style AST matching: correct function name(s) and call count (parallel calls order-insensitive), all required arguments present, every argument value in BFCL's list of acceptable answers, no hallucinated arguments. Irrelevance cases pass only if no call is made. Server or parse errors count as failures.
• Reported: % of 200 cases passed, with a per-category breakdown and the full transcript on each run page.

3. Task accuracy (small subsets)

An optional sanity check with EleutherAI lm-evaluation-harness against llama-server: MMLU-Pro (50 questions × 14 subjects, 5-shot CoT) and GSM8K (first 250, 5-shot, strict match). Not run for the current results. The first batch measured the diff, tool calls and speed only, and accuracy columns appear once these runs exist.

lm_eval --model local-completions \
  --model_args model=quant,base_url=http://127.0.0.1:8089/v1/completions,num_concurrent=1,tokenized_requests=False \
  --tasks mmlu_pro,gsm8k --limit <N> --gen_kwargs temperature=0 --seed 1234 --log_samples

4. Speed

llama-bench -m QUANT.gguf -ngl 999 -fa 1 -p 512 -n 128 -r 5 -t 8 -o json

• PP t/s: prompt processing, 512 tokens. Gen t/s: token generation, 128 tokens. Mean of 5 repetitions.
• full GPU: every layer on the RTX 5070. GPU+RAM split: the file doesn't fit in 12GB, so as many layers as fit go to the GPU (MoE models move expert tensors to CPU with --n-cpu-moe). Split speeds reflect this machine's DDR4 and CPU.
• Before every stage the harness checks with nvidia-smi that the GPU is idle (no other GPU workload, utilization under 10%). It samples again every 20 s during the run. A run that sees another workload appear is discarded, not published.

How "fits in X GB" is estimated

VRAM need ≈ file size + f16 KV cache for 8192 tokens (from each model's attention layer/head config; hybrid models only count their attention layers) + 0.6GB compute buffers. A quant "fits" a budget if that is at most the budget minus 0.5GB (kept free for your desktop).

Benchmark machine

"Ghost": RTX 5070 12GB (Blackwell), Ryzen 7 5800X (8C/16T), 64GB DDR4, Windows 11, NVIDIA driver 616.92, official llama.cpp CUDA 13.4 Windows build (b11770 (39da130fc)). The exact llama.cpp build, driver and CUDA version are recorded in every run.

Reproducibility promise

1. Every run page lists the exact file (repo, filename, HF revision, sha256), llama.cpp build, every command line and every setting.
2. Test sets are versioned (quantdiff-kld-v1, quantdiff-bfcl-v1), and their sha256 is recorded per run. When a test set changes, the version bumps and old results are never mixed with new ones.
3. Raw logs (llama-perplexity, llama-bench JSON, llama-server, tool-call transcripts) are published for every run.
4. The full dataset is downloadable as JSON and CSV.
5. We never publish invented numbers.