Before you start: hardware, honestly
Last reviewed:
The question everyone asks first: what do I need? Here is what we measured on our own machine, two 16 GB NVIDIA cards (an RTX 5080 and an RTX 4070 Ti SUPER).
How these figures were obtained. Measured on our own machines (see the protocol), usually a single run, with speeds as reported by the server itself; we do not publish raw logs. Treat them as orders of magnitude on this hardware, not as guarantees on yours. Figures from outside are labelled with their source.
The rule: video memory decides
A model runs fast when it fits entirely in the card's video memory (VRAM). The file size of the model is a good first estimate of what it needs, plus room for the context (the "KV cache") and a margin.
| Model | File / VRAM | Fits on |
|---|---|---|
| Qwen3 8B, Q4_K_M | 5.0 GB file | an 8 GB card |
| A 9B model, Q8, long context | 9.4 GB used, measured | a 12 GB card |
| A 27B model, IQ3_XXS, 64k context | 14.2 GB used, measured | a 16 GB card, just |
| Qwen3.6 35B-A3B (mixture of experts), Q4_K_M | 22.1 GB file | 16 GB card + the experts in system RAM |
System RAM matters too
"Mixture of experts" models only use a small part of their weights for each word, so their experts can live in system RAM while the rest stays on the card. That trades VRAM for RAM: count 20 GB of free RAM for the 35B above.
What to buy first
- An NVIDIA card with as much VRAM as you can afford. VRAM matters more than raw speed. 12 GB is a comfortable start, 16 GB opens the 27B class.
- 32 GB of RAM or more if you want mixture-of-experts models.
- A fast SSD with room: a few models take 50 to 100 GB.
- Linux. Everything in this series runs on Linux; it is where the tools are first.
Prices change every month, so we do not quote them. Compare VRAM per euro, not benchmark scores.
Found this useful? Tell us what to measure next.