A tiny local judge for the small decisions

Level IntermediateEstimated time 45 minRead first One gateway for all your models, and three ways a local model freezes your agentsA word you don't know? Glossary

Most decisions in an agent network are small: is this e-mail a bill or a newsletter, which of these four tutorials answers that question, is this change risky. Asking a large model each time is slow and wasteful. A tiny model trained to answer typed questions does it in a twentieth of a second, on the processor, and says how sure it is.

The code, not the prompt, sends a decision to the judge. The judge answers a typed question with a probability. Above the threshold the code acts; below it, the question goes up the cascade instead of being guessed.

1. Typed questions, not free text

A judge does not chat. It answers one of three shapes, with probabilities:

yes/no: a probability that the answer is yes;
choice: which option, among a few, each with its probability;
score: a level on a scale you define.

The format matters to the letter. With the API we use, a choice takes its options as an object ({"a": "the certificate", "b": "the typo"}), and a yes/no takes a plain instructions sentence. The wrong shape returns an HTTP error, not a diagnosis.

2. What a 0.8B judge did, measured

On 77 real e-mails to sort, a 0.8-billion-parameter judge on the processor reached 0.87 accuracy at its best setting, in 0.05 s per e-mail, with zero failure in 924 calls. Its confidence is honest: above 0.8, it settled 38 % of the e-mails alone, all correctly. Sending the rest up to a larger model gave 0.96, better than the larger model alone.

3. The code routes, not the prompt

We first told our 9B agent, in its instructions, to ask the judge for its decisions. It did so 0 times out of 3. With a worked example, it copied the example and missed every new case. Deterministic routing must be in the code: the program decides that a question goes to the judge, builds the typed question, and reads the answer.

If you need a model to fill in the question for you, force the format: llama.cpp ignores tool_choice, but it enforces response_format with a JSON schema, as a grammar. The model then cannot answer in any other shape.

Traps

Too many options lose the small judge: asked to pick one tutorial among thirty, it doubted; among the four best pre-selected by a word-overlap rule, it chose confidently. Pre-filter with rules, then let the judge choose.

The small judge reads about 2,000 tokens at most. Give it the facts that bear on the question, not the whole context: padding makes the verdict worse, not safer.

A doubt is an answer. Never turn a 0.55 into a yes: send it up the cascade (see Homelab 14).

Found this useful? Tell us what to measure next.