Which local model for which job: a bench that measures real work
Every week a new model tops a leaderboard. On your card, with your agent, it may write nothing usable. We stopped trusting scores we did not produce, and built a bench of our own: short tests judged by code, then real work judged by the tests that run afterwards. The second half changed our choices.
How these figures were obtained. Measured on our own machines (see the protocol), usually a single run, with speeds as reported by the server itself; we do not publish raw logs. Treat them as orders of magnitude on this hardware, not as guarantees on yours. Figures from outside are labelled with their source.
What you get. A way to pick the model for each job from facts, on your own hardware, and four settings that turned "useless" models into reliable ones. Our bench restores the machine as it found it and never interrupts a game.
1. Two halves: tests, then real work
- Capabilities judged by code: a tool really called, a JSON that validates, an answer that matches, a program whose tests pass, a needle found from 4k to 128k tokens, a colour read from an image. No model judges another.
- Real work: the model gets a small repository and a task in an agent session, and the tests that run afterwards decide.
Generated code runs in a throwaway interpreter, with no environment and capped processor and memory. A bench must not be the thing that breaks your machine.
2. What it showed
| Model | Tests | Speed | Real work |
|---|---|---|---|
| 27B, Q6 | 8/8 | 28.6 tokens/s | 3 timeouts, zero lines |
| 9B, Q8 | 7/8 | 61.7 tokens/s | 2 clean pull requests |
| 4B, Q8 | 8/8 | 139.8 tokens/s | 1 pull request, 342 lines, 2 bugs |
The tests favoured the 4B; real work favoured the 9B. On long files the 4B lost track of its own function names. The 27B produced nothing usable. Small and well-framed beats big and vague.
3. Four settings that matter more than the model
- Turn thinking off on distilled models (
enable_thinking: false). Left on, they burned 2,500 tokens without writing a line; off, the same correct code came out in 2.5 s. Their template ignores "low" or "medium": it is on or off. - An f16 attention cache and a 128k window: the 9B then found a needle in 101,050 tokens, as well as the 27B, in 13.1 GB.
- Declare tools in the model's own style. Some models pick the right function from a JSON schema but never fill the values; declared as Python functions, the same model scores 5/5.
- Measure with the exact settings the agent will use, not the demo's.
4. Who was set aside, and why
- A code model that never emitted a tool call in the agent's format, writing it as prose in a code block.
- A faster variant that lost a reasoning test: 40 % faster is not worth a wrong answer.
- Quantizations that never loaded: deleted, 26 GB recovered.
Traps
Test each model in its own language. Tested with JSON tools, a small model looked unable to call anything; with Python tools, it was perfect. A verdict in the wrong language is a verdict on your harness.
Never evict a game to run a bench: our bench skips the card a game is using, and puts every model back where it was.
Found this useful? Tell us what to measure next.