GPU too small? Rent one.
If our "fits" column says a quant needs more VRAM than you have, you have three options: a lower-bit quant, a GPU+RAM split (works, but generation often drops below 10 t/s), or a rented GPU by the hour.
GPU too small? Rent one.
Run a 32B model at Q6/Q8, or a 70B, on a cloud GPU for an hour or two instead of buying hardware.
RunPod referral
Per-second GPU pods. RTX 4090 / 5090 / A100 / H100 for big quants or BF16 reference runs.
Link coming soonVast.ai referral
Marketplace GPUs, often the cheapest way to try a 24-48GB card for an evening.
Link coming soonRough sizing
- 24GB (RTX 3090/4090): any ≤32B model at Q4-Q5 with a useful context.
- 32GB (RTX 5090): 32B at Q6_K, or a 24B at Q8_0.
- 48-80GB (A6000, L40S, A100, H100): 32B at Q8/BF16, 70B at Q4-Q6.
Disclosure: links on this page are referral links. QuantDiff may earn a commission at no extra cost to you. Referral partners and sponsors have no input on scores or rankings.