A 177-billion-parameter model on two consumer cards: placement beats bandwidth

Level AdvancedEstimated time an eveningRead first Several models, two graphics cards: a grid instead of a juggling act, RAM, zram and swap on an AI machine: the three thieves that cost us a factor of tenA word you don't know? Glossary

A mixture-of-experts (MoE) model only uses a small part of its weights for each token. That is what lets a 177-billion-parameter model, compressed to 2 bits, run on two 16 GB cards and system RAM: most experts wait in RAM, the busy ones sit on the cards. It works, at about 55 tokens per second for us, once three things are right.

How these figures were obtained. Measured on our own machines (see the protocol), usually a single run, with speeds as reported by the server itself; we do not publish raw logs. Treat them as orders of magnitude on this hardware, not as guarantees on yours. Figures from outside are labelled with their source.

Three measurements, in the order they were taken. Widening the PCIe link alone changed nothing, because most of the model still lived behind the slow card. Putting the model on one card doubled the speed.

A bit of history

The mixture-of-experts idea is older than deep learning: Jacobs, Jordan, Nowlan and Hinton described it in 1991, a set of specialists and a gate that chooses which to ask. In 2017 a Google team made it sparse and huge: thousands of experts, only a few active per token. That is why a model of 177 billion parameters can run at home: most of it rests while each token is computed.

1. Placement first, bandwidth second

Our model was spread over both cards, and the big card ran its PCIe link at x4 instead of x16. Fixing the link alone gave nothing: 26.1 before, 22.1 after. Most of the model still lived on the second card, behind its own four-lane link. Confining the model to the big card, at x16, gave 55.6. Measure one change at a time, in order, or you will credit the wrong one.

2. Map the weights, never pin them

Loading options decide whether the model's weights can leave RAM when memory runs short. Memory-mapped loading (mmap) lets the system drop pages it can read again from the file; "resident" loading pins them. With resident loading, 47.6 GB of our 60 became memory nothing could reclaim, and the machine froze twice in one afternoon. Load big MoE models with mmap, and keep the RAM thieves away (see RAM, zram and swap).

3. Keep the prompt cache

An agent sends the same long prefix (instructions and tool definitions) with every message. llama.cpp keeps the state of prompts it has already read in a RAM cache, --cache-ram, 8,192 MB by default. We set it to 0 after the freeze, confusing it with the weights: every turn then re-read 11,233 tokens, 112 seconds. With the cache back, the second turn read 4 tokens in 2 seconds: fifty times faster, for no loss.

Traps

The prompt cache holds a prompt's state, not the model's weights: it was not the cause of the freeze. Two memories with similar names; turn off only the one you have measured.

The first turn after loading still pays the full prompt. After switching to a big model, play one turn yourself before an agent or a person uses it.

Make the dangerous settings impossible to come back: our deployment test refuses a configuration with the prompt cache at 0, and we checked that the test turns red when someone tries.

Sources

Found this useful? Tell us what to measure next.