fh-probes: measure a node without becoming the cause of what you measure

Last reviewed: 2026-10-22

Every monitoring setup ends up showing itself: an nvidia-smi every second, docker inspect in a loop, and soon the measurement tool tops the consumer list. fh-probes starts from the opposite principle: the measurement must never appear in what it measures. One JSON line per second, zero subprocesses, and an admission when a round is late instead of catching up in a burst.

What you get. The machine's truth, second by second, in a plain .jsonl file — and with fh-history, the cross-reference that changes everything: "this run failed while memory pressure climbed to 91 and prefill dropped to 20 tokens/s". Python 3.10+, standard library only, user systemd unit included (low priority, 128 MiB memory cap).

1. Zero subprocesses is the whole point

GPUs are read through NVML (libnvidia-ml.so.1 via ctypes), everything else comes from /proc and two local HTTP requests (/metrics and /slots, llama.cpp format). No nvidia-smi, no docker inspect, no ssh: spawning a process costs more than the measurement itself. Without a GPU or driver, it still works — the gpu field is simply empty.

2. Install

git clone https://github.com/bengua/fh-probes
cd fh-probes
mkdir -p ~/.local/bin ~/.config/systemd/user
cp fh-probes fh-history ~/.local/bin/
cp fh-probes.service ~/.config/systemd/user/

3. First run: three rounds, in the foreground

FH_PROBES_ROUNDS=3 ~/.local/bin/fh-probes
tail -1 ~/.local/state/fh-probes/$(date +%F).jsonl

Each line is a complete JSON object: GPU (utilization, memory, temperature), RAM, PSI pressure, model-server counters. Cumulative counters are stored raw — the derivative is computed at read time, so a server restart doesn't produce a lying spike.

4. As a continuous service

systemctl --user daemon-reload
systemctl --user enable --now fh-probes

The included unit runs with low CPU/IO priority and a 128 MiB memory cap: if the machine suffers, the probe steps aside first. Single instance only — a second copy exits immediately, cleanly.

5. fh-history: cross the machine with your runs

The probe alone says what the machine was doing. fh-history lays your executions (agent runs, jobs, benchmarks) on the same timeline, and keeps the extremes, not the averages — a run that passed through a pressure dip and a run that died in a spike don't tell the same story. It also flags when two runs overlapped: their machine numbers are then shared, and you need to know that before concluding anything.

Pitfalls

The skipped field is not a bug: a late round is admitted, never caught up in a burst. A catch-up burst would distort precisely the pressure measurements you came for.

One file per day in ~/.local/state/fh-probes/ — plan rotation if the machine measures for months (lines are small, but days add up).

Found this useful? Tell us what to measure next.