A fully local judge in two passes: Tev1 first, Nimble when in doubt

Level IntermediateEstimated time 45 minRead first A tiny local judge for the small decisionsA word you don't know? Glossary

A small judge is fast and honest about its doubts. A larger one is surer but costs memory and time. Rather than choose, chain them, and let code decide who answers. Our judge service speaks the same API as a hosted judge: callers changed one address and stopped sending their questions outside.

How these figures were obtained. Measured on our own machines (see the protocol), usually a single run, with speeds as reported by the server itself; we do not publish raw logs. Treat them as orders of magnitude on this hardware, not as guarantees on yours. Figures from outside are labelled with their source.

What you get. Decisions on your own data with nothing sent out: 0.87 to 0.91 on 77 real e-mails, against 0.86 to 0.95 for the hosted service. At rest the memory is empty: each model loads on demand and leaves.

1. The two passes, decided by code

Every question goes to the fast model: Tev1, 0.8B, about 1 GB, 0.05 s per decision on the processor.
Its answer is kept when its confidence reaches 0.8. At that level it was never wrong on our e-mails, and it settled 38 % of them alone.
The rest go to the slow model: Nimble, 9B, about 10 GB, 0.8 s per decision, loaded on demand and freed 60 s later, so a burst shares one load.
If the slow model cannot answer (its machine asleep), the fast answer comes back marked "confirmed": false. The caller decides. Nothing is guessed.

Every answer says who decided (decided_by) and whether it is confirmed.

2. Serve both models with Ollama, on the processor

OLLAMA_HOST=127.0.0.1:11435 OLLAMA_KEEP_ALIVE=0 ollama serve
fh-judge serve      # http://127.0.0.1:8475/v1/systemone, GET /health
fh-judge ask < question.json

No graphics card involved: the cards stay for the agents and the images.

3. Measured

Setup	Lexical criteria	Conceptual criteria
Tev1 alone, best setting	0.87	0.81
Nimble alone	0.91	0.84
Tev1 then Nimble	0.91	0.87
Hosted judge, for reference	0.95	0.86

The local pair matches the hosted judge on conceptual questions and stays four points behind on lexical ones. That was our condition before letting a judge read personal documents.

Traps

The question goes in instructions, not question: the wrong field returns an error 400. And the small model reads about 2,000 tokens at most: send the facts, not the file.

Place the small judge where it never sleeps. The large one can stay on the big machine; the small one must answer at night too, for the automations.

Found this useful? Tell us what to measure next.