Speculative decoding (MTP): up to twice as fast on the same card

Level IntermediateEstimated time 1 hRead first Your first local model with llama.cppA word you don't know? Glossary

A model writes one token at a time, and each token costs a full pass through the model. Speculative decoding guesses several tokens ahead with something cheap, then checks them all in a single pass: every correct guess is a token for free. Some recent models ship their own guessing head (multi-token prediction, MTP) inside the model file.

How these figures were obtained. Measured on our own machines (see the protocol), usually a single run, with speeds as reported by the server itself; we do not publish raw logs. Treat them as orders of magnitude on this hardware, not as guarantees on yours. Figures from outside are labelled with their source.

Tokens per second on our setup as the number of drafted tokens grows. The gain stops at 5 and falls after: the best value depends on your hardware, so measure yours instead of copying someone else's.

A bit of history

Speculative decoding was published in 2022 and 2023 by teams at Google and DeepMind: a small, fast model guesses several tokens ahead, and the large model checks them all in one pass, keeping the ones it agrees with. The output is exactly what the large model would have written, only faster. Multi-token prediction (MTP) goes further: the model itself is trained to propose the next few tokens.

1. The right way to turn it on

With a llama.cpp build that supports it, and a model file that contains MTP tensors:

--spec-type draft-mtp --spec-draft-n-max 5

Passing the MTP head as an ordinary draft model (--model-draft alone) crashed our server, and we first concluded that MTP did not work. And an older llama.cpp build loads such a file and silently ignores the MTP part: on an old build, no gain proves nothing.

2. Measure your own n_max

--spec-draft-n-max is how many tokens are guessed ahead. Too few wastes the idea; too many wastes passes on wrong guesses. On our setup the gain rose until 5 and fell after 8. Published tables measured on data-centre GPUs stop much earlier: a slower setup gains from speculation for longer. Run the same prompt three times per value and keep yours.

3. Expect less in real use than on a benchmark

On a code benchmark with reasoning off, we measured almost three times the speed and 84 % of guesses accepted. In production, with reasoning on, it settled between 84 and 98 tokens per second: about twice the speed, with half the guesses accepted. Reasoning is less predictable than code. Announce the real number, not the benchmark one.

Traps we paid for

It costs VRAM: about 1.7 GB on our card. Check that the model still has room for its computation buffers (see the GPU grid).

On some requests, the speculative path crashed our server. What looked like several unrelated failures (agents dying after one answer, a dashboard showing the backend down, mode switches that "failed" after succeeding) was this one crash. If your containers do not restart by themselves on purpose, something must notice and reload them (see route and repair with guardrails).

Sources

Found this useful? Tell us what to measure next.