Slow for no reason? It is re-reading: the KV cache trap

Your model writes at 54 tokens per second, the benchmark is happy, and yet a coding agent waits minutes before its first word. The culprit is almost never the model. It is the first phase of every answer, reading the prompt, and one cache setting that quietly breaks it.

How these figures were obtained. Measured on our own machines (see the protocol), usually a single run, with speeds as reported by the server itself; we do not publish raw logs. Treat them as orders of magnitude on this hardware, not as guarantees on yours. Figures from outside are labelled with their source.

What you get. The answer to "why is my local agent so slow" in two minutes of measurement, and a fix of one flag: on our card, a 23,400-token agent prompt went from 18 minutes to 17 seconds.

1. Two speeds, not one

A model answers in two phases. First it reads the whole prompt (prompt processing, or prefill), then it writes the answer token by token (generation). Most benchmarks and most chat demos only measure the second. An agent sends a long prompt at every turn: the first phase is the one you wait for.

2. Measure the reading, on a long prompt

llama.cpp prints both speeds at the end of each request. Send a long prompt (a few thousand tokens: paste a big file and ask for a summary), then read the server's log:

prompt eval time = 5830 ms / 7758 tokens (1330 tokens per second)
       eval time = 3700 ms /  200 tokens (  54 tokens per second)

The first line is the reading. On a recent card with flash attention, expect hundreds to thousands of tokens per second. Tens is a problem.

3. The flag that breaks it

To save memory, llama.cpp can store its key/value cache in compressed form: -ctk for the keys, -ctv for the values. Compressing the values to q4_0 looks like a free win. It is not: flash attention is silently disarmed. The server accepts -fa on, says nothing in its log, even at the highest verbosity, and reads 21 times slower.

The saving was about 33 KB per token instead of 50: 1.8 GB on a 106k window. Paid with twenty-one times the reading time, it is a bad deal in every case.

llama-server -m model.gguf -fa on -ctk q8_0 -ctv q8_0   # this, and nothing lower for V

4. Recognize the signature

Without flash attention, reading slows down as it goes: in our case 212 tokens per second over the first 2,048 tokens, then 82, 51, 37. A speed that falls with the position in the prompt is unoptimized attention. A speed that is flat but low is something else (memory, offloading to the processor).

Traps

The other cause of re-reading is a prompt cache turned off. With --cache-ram 0, the server forgets the start of the conversation and re-reads it at each turn (see the 177B on two cards).

Running Claude Code on a local model? For a model name it does not know, it assumes a 200,000-token window and compacts too late; and automatic compaction only arms above 100,000. Set CLAUDE_CODE_MAX_CONTEXT_TOKENS to your server's real window (see Claude Code, two ways).

Sources

• Vaswani et al. (2017), Attention Is All You Need
• llama.cpp server documentation

Found this useful? Tell us what to measure next.