Route requests with a small trained classifier, not keywords
Sending every request to the biggest model is slow; sending every request to the smallest is wrong. A router decides, and the usual first router is a list of keywords. We measured ours against something simpler and against a small trained classifier. The numbers changed the design.
How these figures were obtained. Measured on our own machines (see the protocol), usually a single run, with speeds as reported by the server itself; we do not publish raw logs. Treat them as orders of magnitude on this hardware, not as guarantees on yours. Figures from outside are labelled with their source.
What you get. A router right 84 times out of 100 instead of 64, answering in about 20 ms on the processor, and a method to know whether yours beats the simplest possible rule.
1. Measure against the dumb baseline first
On 237 real requests, each labelled easy, standard or hard, in cross-validation:
| Router | Correct |
|---|---|
| random, respecting class frequencies | 59.5 % |
| keyword matching | 63.7 % |
| always "standard" | 72.6 % |
| trained classifier | 84.0 % |
The real bar was not random: it was "always answer the most common class". Our keyword router was below it. Measure that baseline before celebrating anything.
2. Where the gain came from
The "easy" requests: keywords found 2 out of 35, the classifier 34 out of 35, with no false positive. That is where small models can take over, and where the speed comes from. Latency: 18 to 27 ms for one request, 3.8 ms each in a batch.
3. Three decisions the numbers imposed
- No confidence fallback. The classifier beat keywords at every confidence level, so falling back to keywords when unsure made things worse. The fallback only serves when the classifier is down.
- A calibration was rejected although it scored 86.9 %: it never predicted "hard" again. Three points bought by abandoning the class whose mistakes cost quality.
- "Hard" stays weak (13 out of 30): thirty examples are not enough. Collect more before trusting it.
4. The bigger lesson: the tool catalogue
Measuring the requests revealed something else: our tool catalogue weighed 19,157 tokens for 58 tools, paid at every turn. Loading tools by intent saves about 73 %. And our coding agents deliver merged pull requests with six tools (read, write, edit, search, list, shell): 2,182 tokens. The question stopped being "how to reduce" and became "why 58".
Traps
llama-server ignores enable_thinking at the root of the request: the model thought for 844 characters and answered nothing. Only chat_template_kwargs: {"enable_thinking": false} turns it off (1.7 s instead of 9.6). Through some gateways the root form works, directly it does not.
A router with no logging never improves: record each request and the model that answered, or your training set never grows.
Found this useful? Tell us what to measure next.