Why is it failing? Give the model facts, and a machine that remembers

"Why does this service fail?" A model with a shell will run commands at random, quote log lines that do not exist and blame files it made up. The problem is not its intelligence: it lacks the facts, and gathering them is not its strength. So we gather them with code, and hand them over.

How these figures were obtained. Measured on our own machines (see the protocol), usually a single run, with speeds as reported by the server itself; we do not publish raw logs. Treat them as orders of magnitude on this hardware, not as guarantees on yours. Figures from outside are labelled with their source.

What you get. On four faults planted on purpose, our 27B model found 5 out of 5 in 72 seconds with the facts tool, against 2.75 out of 5 in 9 minutes without it. Same model, same faults: only the facts changed.

1. A facts tool: fh-why

fh-why unit nginx.service            # its last words, since when, what changed just before
fh-why unit --failed                 # every failed unit, system and user
fh-why container my-model            # health checks, restarts, out-of-memory kills, last errors
fh-why run 4182                      # an agent run: end state and the lines that failed
fh-why unit --failed --json          # one object per line, for a model or a script

One file, Python standard library only, no model inside. Its rule: what cannot be read is said, never skipped. "Could not read the journal: permission denied" is a fact; silence is not.

2. A machine that remembers: fh-probes

An agent run fails at 3 a.m. Was it the code, or was the machine drowning? Without a record, you cannot know. A small sampler writes, once per second, one line about the cards, memory, memory pressure and the model servers. A second command joins that series to each run's time window:

fh-history 4182
# "this run failed while memory pressure climbed to 91 and prompt reading dropped to 20 tokens/s"

It keeps extremes, not averages, and says when two runs overlapped and therefore share the same numbers.

3. Rules learned from a freeze

The day our server froze, the old collector had slowed from 3 to 20 seconds per pass, and ten copies ran side by side, each launching its own subprocesses: the monitoring helped cause the freeze it should have seen. The sampler was rewritten with three rules:

1. No subprocess: the card's library called directly, /proc, two local HTTP requests.
2. One instance: an exclusive lock; a second copy exits.
3. Skip, never catch up: a late round is recorded as skipped, never repaid in a burst.

Watch memory pressure (/proc/pressure/memory), not "available memory". Three minutes before the freeze, available memory said all was well; pressure was at 99. Model weights held resident in shared memory were what filled the machine.

Traps

A failure seen often enough deserves a written page, not a new diagnosis each time. Keep a list of recurring failure signatures and their fix.

A search index that still answers but is forty hours behind is worse than one that is down. Check its freshness, not just its status code.

Found this useful? Tell us what to measure next.