Long context on a 16 GB card: KVMem, and the build we run

Level AdvancedEstimated time an eveningRead first Slow for no reason? It is re-reading: the KV cache trap, A 177-billion-parameter model on two consumer cards: placement beats bandwidthA word you don't know? Glossary

A coding agent's conversation grows to 100,000 tokens and more. A normal server keeps the attention cache of all of it on the card, and slows down as it grows. KVMem, an open-source extension of llama.cpp, keeps the history in system RAM and brings back to the card only the blocks relevant to the current step. We measured it against plain llama.cpp on our own cards, then built our own version for our machine.

How these figures were obtained. Measured on our own machines (see the protocol), usually a single run, with speeds as reported by the server itself; we do not publish raw logs. Treat them as orders of magnitude on this hardware, not as guarantees on yours. Figures from outside are labelled with their source.

What you get. Measured here: 48 recall checks out of 48 at 44k and 130k tokens; at 103k tokens, 150.9 tokens per second against 132.1 for plain llama.cpp; 4.9 GB of VRAM freed. And a 177B mixture-of-experts model served from a single 16 GB card.

1. The idea

Finished blocks of the attention cache go to system RAM. At each step, KVMem picks the blocks that matter for the current question and places them, in order, in a bounded working set on the card. The paper behind it reports that a 32k resident window is essentially lossless against the full 256k history.

llama-kvmem-server -m model.gguf -c 262144 --kvmem-budget 8192 \
  --kvmem-gen-reserve 16384 --spec-type draft-mtp

One generation cannot exceed --kvmem-gen-reserve tokens, thinking included. Size it for your longest answer.

2. Measure before switching

Recall: 48/48 for llama.cpp, KVMem unbounded and KVMem bounded to 8,192, with questions about the start and the end of the conversation.
Speed: plain llama.cpp is faster on short contexts (178.9 against 151.0 at 39k); KVMem stays flat and overtakes it around 70k tokens on our faster card, 53k on the other.
Switching engines mid-conversation cost 17.9 s, mostly re-reading the context: the crossover is not the right switching point.

Our rule: KVMem for agents (long contexts), plain llama.cpp for chat and short calls. Two presets, no automatic switch.

3. What our build changes, and why

One binary for two card generations. Upstream compiles for the newest architecture only; a host with an older second card loads the model and fails there, silently. We build for both (89-real;120a-real), so we can compare cards changing nothing else.
Experts in host memory: --cpu-moe, --n-cpu-moe, -ot and --no-repack in the KVMem server, so the experts of a huge MoE model stay in evictable page cache instead of 21 GB of pinned memory.
A crash fixed for the Flash-Next architecture: the memory context handed to the graph was of the wrong type and the server asserted at start.
Token probabilities (logprobs, top_logprobs) on the chat endpoint: the model's hesitation, per token, for an agent that decides when to ask for help.
A GPU expert cache, by rebasing on a llama.cpp fork that has one: the busy experts stay on the card. That fork's newer tip decoded at 28.6 tokens per second, the commit we pinned at 36.9: we kept the faster one.

Traps

Skip scripts/apply-patches.sh after the submodule update and the build fails deep inside, on missing members, which looks like a wrong submodule version. It is not. Running it twice is safe.

A comment placed between two lines of a \ continuation silently cuts the command: our CMake call lost its CUDA flags and bash -n said the file was fine. Only running the script caught it; the test now checks the arguments CMake really receives.

Some model families are not supported (sliding-window attention, for instance): they stay on plain llama.cpp.

Sources

Found this useful? Tell us what to measure next.