Failures that call you: every service reports, every deploy can roll back

Monitoring suites are heavy and mostly look at what is up. What you need first is simpler: when a service fails, you hear about it, once, with its name. systemd already knows when a unit fails; you only have to give it somewhere to say so.

What you get. Every failed service sends a signed event with its name; every machine says once a day that it is alive; a delivery that breaks a service puts the previous version back and tells you. The rollback and the skip were proven here by forcing them on purpose.

1. One template for every failure

# ~/.config/systemd/user/notify-failure@.service
[Unit]
Description=Report the failure of %i

[Service]
Type=oneshot
ExecStart=%h/.local/bin/notify-failure %i

Then attach it to any unit with a drop-in, without editing the unit itself:

# ~/.config/systemd/user/my-tunnel.service.d/notify.conf
[Unit]
OnFailure=notify-failure@%n.service

notify-failure is a few lines that call a sender with a source, a type, a severity and a small JSON detail. The sender signs the body with HMAC and posts it to an automation, which writes the event and pushes a notification to the phone. Its exit codes say what happened: accepted, refused (bad signature), or collector unreachable.

2. A daily "I am alive", with patience

Each machine sends a health event once a day. The watcher alerts when one goes silent too long, with a tolerance that fits the machine: a laptop may stay closed for three days, a server much less. An "info" event never rings.

3. Skip, don't fail, when the target is away

A service that reads the server every ten minutes failed 86 times per boot of the laptop, 65 of them because the server was asleep, and each failure triggered a failure report that failed in turn. A condition skips the run instead:

[Service]
ExecCondition=bash -c 'ssh -o BatchMode=yes -o ConnectTimeout=3 server true || exit 1'

The || exit 1 is not decoration. For ExecCondition, exit codes 1 to 254 mean "skip", but 255 means "fail", and 255 is exactly what ssh returns when it cannot connect.

4. Deliveries that roll back

Our delivery script does, for each component: back up, install, restart, check health, and if the check fails, put the previous version back and restart again. It remembers the commit it delivered and sends deploy.delivered or deploy.rolled_back. We proved the rollback by forcing a failure with a --simulate-failure flag.

Traps

A user service does not have ~/.local/bin in its PATH: write full paths, or the failure report itself fails.

A notification storm teaches you to ignore notifications. Report a failure once, and the recovery once.

Found this useful? Tell us what to measure next.