Glossary

Short, plain-English meanings for the words used on this site. New to all of this? Read Start here first.

Basics

Local model

An AI chat model that runs on your own computer instead of on a company's server. Your chats stay on your machine.

A large language model (LLM) that you download and run on your own GPU or Mac with an app such as LM Studio or Ollama. It works offline and your prompts never leave your computer. How big a model you can run depends on how much memory your GPU or Mac has.

Quant (quantized model)

A smaller, compressed copy of a model. Smaller files run faster and fit more hardware, but squeezing too far makes the model dumber.

Models are trained with 16 bits per number (weight). A quant stores each weight with fewer bits, for example about 4 or 8, so the file is much smaller and faster to run. The trade-off: the more you squeeze, the more the model drifts from the original. QuantDiff measures how much each quant drifts and whether it still answers questions and calls tools correctly.

Formats

GGUF

The model file format used by llama.cpp, LM Studio and Ollama. It runs on NVIDIA, AMD and Intel GPUs, on plain CPUs and on Macs.

A single-file format for quantized models made by the llama.cpp project. If you have a Windows or Linux PC with an NVIDIA, AMD or Intel GPU, GGUF is the format you want. It also runs on Macs. One Hugging Face repo usually holds many GGUF files, one per quant level.

MLX

Apple's machine-learning format for Apple Silicon Macs (M1 and newer). It does not run on Windows or NVIDIA GPUs.

MLX is Apple's open-source framework for Apple Silicon. MLX models are a folder of files (one Hugging Face repo per quant, for example …-4bit). You can run them in LM Studio on a Mac or with the mlx-lm Python package. On a Mac you can use either MLX or GGUF.

BF16

16-bit "brain float": the full-size original model weights. We compare every quant against it.

BF16 (bfloat16) is the 16-bit number format most models are released in. It is the full-quality reference: QuantDiff runs the BF16 model once and measures how far each quant drifts from it.

Quant names

Q4_K_M, Q8_0, Q6_K… (quant names)

The name of a GGUF quant level. The number is roughly the bits per weight (Q8 ≈ 8 bits, Q4 ≈ 4 bits). Lower = smaller file, more drift.

llama.cpp names its quant types like Q4_K_M: Q4 = about 4 bits per weight, K = the "k-quant" method, and S / M / L = small, medium or large mix (bigger mixes keep more important parts at higher precision). Q8_0 is the near-lossless 8-bit type; Q2_K is the smallest and loses the most. On the leaderboard we group quants by this number.

IQ quants (IQ3_M, IQ4_XS…)

"I-quants": a newer llama.cpp packing that keeps more quality at small sizes. On some hardware they run a little slower than K-quants.

I-quants such as IQ3_M or IQ4_XS use a smarter encoding than the classic K-quants, so they often lose less quality at the same file size. They are usually made with an importance matrix. Speed depends on your hardware; check the measured speeds.

UD quants (UD-Q4_K_XL…)

Unsloth "Dynamic" quants: Unsloth chooses a different precision for each layer, keeping the most sensitive layers bigger.

Files from Unsloth with a UD- prefix are Unsloth Dynamic quants. Instead of one quant type for the whole model, they keep sensitive layers at higher precision and squeeze the rest more. The number still tells you the rough size class (UD-Q4_K_XL is a 4-bit-class file).

i1 / imatrix

An "i1-" file (from mradermacher) was quantized with an importance matrix: sample text shows which weights matter most, so those keep more detail.

An importance matrix (imatrix) is measured by running sample text through the model. The quantizer then spends its limited bits where they matter most. mradermacher puts these files in separate -i1-GGUF repos with an i1- prefix; other makers often use one too (model pages mark imatrix files). Same quant type, different maker can still give different results: see makers compared.

Hardware

VRAM

The memory on your graphics card. The model file plus its context memory must fit in it to run at full speed.

Video RAM: the memory on a GPU, for example 8, 12, 16 or 24 GB. If the model file plus the KV cache does not fit, apps put part of the model in normal system RAM, which is much slower (we label that GPU+RAM split). The picker estimates the fit for you.

Unified memory

On Apple Silicon Macs, the CPU and GPU share one pool of memory. macOS lets the GPU use only part of it by default.

A Mac's memory is shared between the CPU and the GPU, so a 32 GB Mac can run bigger models than a GPU with less VRAM. macOS keeps some of it for the system; the picker uses the default GPU limit (see how the picker counts Mac memory).

Full GPU vs GPU+RAM split

Full GPU: the whole model is on the graphics card (fast). GPU+RAM split: part of it lives in system RAM, which is much slower.

When a model does not fit in VRAM, llama.cpp-based apps can keep some layers in system RAM. It still works, but generation slows down a lot. Speeds measured that way are labelled split.

Context (context length)

How much text the model can keep in mind at once: your messages, its replies and any documents, counted in tokens. Longer context needs more memory.

The context window is the model's working memory for one conversation, measured in tokens. Our picks assume 8k (8,192) tokens unless you choose another length in the picker. Longer context needs a bigger KV cache.

KV cache

Memory the model uses to remember the conversation so far. It grows with context length and comes on top of the file size.

While it reads and writes, the model stores "keys" and "values" for every token in the context. That KV cache needs memory on top of the model file, and it grows with the context length. Apps can quantize the KV cache to save memory.

Speed

Token

A word or piece of a word. Models read and write text as tokens.

Models split text into tokens: common words are one token, rare words are several pieces. Context length and speed are counted in tokens.

tok/s (tokens per second)

How fast the model writes its answer. Higher is faster.

Generation speed: how many tokens per second the model produces while answering ("Gen t/s"). "PP t/s" (prompt processing) is how fast it reads your input. We measure GGUF speed with llama-bench on our test GPU and MLX speed with mlx-lm on a Mac; other hardware gets a clearly labelled estimate.

Measurements

KLD (KL divergence)

How much a quant's word predictions differ from the original model's. 0 means identical; lower is better.

Kullback-Leibler divergence compares, word by word, the quant's full set of next-token probabilities with the BF16 original's on the same text. It is our most sensitive measure of quality loss. Compare KLD only between quants of the same model. How we measure it.

Same top token

How often the quant's single most likely next word is the same as the original model's.

A simpler companion to KLD: the share of positions where the quant and the original agree on the most likely next token.

95% interval

The range where the true score most likely lies. We test a sample of questions, so every score has some luck in it.

We test each file on a fixed sample of questions, not every possible question. The 95% interval (Wilson) shows how much the score could move by chance. Wider interval = less certain.

Statistically tied

Two scores whose 95% intervals overlap. The difference is small enough that it could be luck, so we can't call one better.

If two files' 95% intervals overlap, we call them tied: on this test we cannot say which is better. A file is marked ▼ below only when its whole interval sits under the best file's interval.

Tool calls (BFCL)

Whether the model can correctly ask to use a tool or app (a "function call"). Tested with cases from the Berkeley Function Calling Leaderboard.

Tool calling is how a model asks an app to do something, like search or run code, by writing a structured request. We use cases from the Berkeley Function Calling Leaderboard (BFCL) and check the function name and every argument. Some cases test that the model does not call a tool when none fits. Details.

MMLU

Multiple-choice questions across many school and professional subjects. Tests general knowledge.

Massive Multitask Language Understanding: multiple-choice questions from subjects such as history, law, medicine and math. We use a fixed sample. Details.

ARC-Challenge

Hard grade-school science multiple-choice questions. Tests reasoning.

The AI2 Reasoning Challenge, Challenge set: science questions that simple methods get wrong. We use a fixed sample. Details.

GSM8K

Grade-school math word problems. The model writes out its steps, then the answer.

Grade School Math 8K: word problems that need a few steps of arithmetic. We use a fixed sample and check the final number. Details.

Apps

llama.cpp

The open-source engine that runs GGUF files. LM Studio and Ollama use it (or its ggml library) under the hood.

llama.cpp is the C/C++ inference engine behind GGUF. We run all GGUF tests with one pinned llama.cpp build so every quant gets identical settings.

LM Studio

A free desktop app (Windows, Mac, Linux) to search, download and chat with local models. Runs GGUF everywhere and MLX on Macs.

LM Studio has a built-in model search (the Discover tab) that downloads straight from Hugging Face. See how to run a pick in LM Studio.

Ollama

A free app that runs GGUF models from the command line with one command, and serves them to other apps.

Ollama can pull any GGUF file from Hugging Face with ollama run hf.co/user/repo:tag. See how to run a pick in Ollama.

Hugging Face

The website where model files are shared. Each pick links to its exact repo and file there.

Hugging Face (huggingface.co) hosts model repos. A repo name looks like unsloth/Qwen3.5-9B-GGUF: the maker, then the model.