RAM, zram and swap on an AI machine: the three thieves that cost us a factor of ten

Level IntermediateEstimated time 45 minRead first Before you start: hardware, honestlyA word you don't know? Glossary

A model that does not fit entirely in VRAM keeps part of itself in system RAM (see Hardware, honestly). Then RAM speed is model speed. Ours crawled at 1.9 tokens per second; the card, the PCIe link and the quantisation were all fine. Three things were stealing the RAM.

How these figures were obtained. Measured on our own machines (see the protocol), usually a single run, with speeds as reported by the server itself; we do not publish raw logs. Treat them as orders of magnitude on this hardware, not as guarantees on yours. Figures from outside are labelled with their source.

Same model, same card, same settings: generation went from 1.9 to about 20 tokens per second once three things stopped eating the RAM that held the model's experts.

A bit of history

Swap, writing memory pages to disk when RAM runs out, is as old as virtual memory. zram, merged into the Linux kernel in 2014, offers another way: a compressed block of RAM used as swap, so pages are squeezed rather than written to a slow disk. It is fast and free, but it is still RAM: sized too large, it steals from the very models it was meant to help.

Thief 1: zram sized to all the RAM

zram is swap compressed into RAM. Some distributions size it to the whole RAM by default. On a normal desktop it compresses about three to one and it is a good deal. On an AI machine, what gets swapped are the pages of the model file, already compressed by quantisation: ours compressed at 1.14 to one. zram held 35.8 GB of real RAM to save 5, and those pages were exactly the cache that kept the model fast.

Check it, then cap it in /etc/systemd/zram-generator.conf (which overrides the distribution's file):

cat /sys/block/zram0/mm_stat     # stored bytes vs bytes really used
# /etc/systemd/zram-generator.conf
[zram0]
zram-size = 16384

sudo swapoff /dev/zram0
sudo systemctl restart systemd-zram-setup@zram0.service

Thief 2: an application that leaks

One of our image-generation apps grew by about 30 GB in 25 hours with an empty queue. That is not a useful cache, it is a leak: watch the resident memory of your long-running apps as a counter that rises, and restart the ones that leak on a schedule.

ps -eo rss,etime,comm --sort=-rss | head

Thief 3: the out-of-memory killer aimed at the model

Tools like earlyoom kill a process before the system freezes. Ours had been configured to prefer killing the model server, and to protect everything else. Read your settings (/etc/default/earlyoom): the model should be the last thing to die, not the first.

Two more things worth knowing

Stopping a model container that loaded its file with mmap does not give the RAM back: the file stays in the page cache (buff/cache in free). sync; echo 3 | sudo tee /proc/sys/vm/drop_caches releases it, which we measured at 46 GB.

Four RAM sticks from two different kits ran at 3,600 MT/s instead of 6,000 on our board: about 40 % less bandwidth for a model that reads its experts from RAM at every token. Two matched sticks are often faster than four mixed ones.

Swap is not one rule for every machine. A database VM wants a low swappiness (keep its hot pages in RAM, swap is only a safety valve); a small VM with bursts wants the default. And swapon --show run without privileges prints nothing instead of an error: read /proc/swaps, which tells the truth. Test an fstab change with swapoff -a && swapon -a instead of rebooting.

Sources

zram, Linux kernel documentation

Found this useful? Tell us what to measure next.