Homelab 5: backups that survive an attack on your server

A backup protects you from a dead disk. A good backup also protects you from your own server turning against you: a mistake run as root, ransomware, an attacker. The usual advice is "keep a copy somewhere else". The detail that makes it work is the direction: the second copy must be pulled by another machine, not pushed by the server.

1. Copy 1: every night, on another disk

In Proxmox (Homelab 1), Datacenter → Backup → Add: every guest, every night, to a storage on a separate disk. Our retention keeps 7 daily and 4 weekly points:

keep-daily=7, keep-weekly=4

We used to keep three points in total: a problem noticed three days late and every point would have held it. Count how many days you would need to notice a problem, then double it.

An external disk survives a dead motherboard. But it stays mounted and written by the server: anything that destroys through the server reaches it too. That is what copy 2 is for.

2. Copy 2: pulled with a key that can only read

On the server, create an unprivileged user whose SSH key can do one thing only: read the backup folder. In its ~/.ssh/authorized_keys, one line:

command="/usr/bin/rrsync -ro /path/to/backups",restrict ssh-ed25519 AAAA… copy2

rrsync ships with rsync; -ro makes it read-only and confines the key to that folder; restrict removes everything else (shell, forwarding). Then, on the other machine, every night after copy 1:

rsync -a --delete \
  --backup --backup-dir=../quarantine/$(date +%F) \
  backup@your-server:./ /path/to/copy2/

--delete keeps an exact mirror; --backup-dir moves anything deleted or replaced into a dated quarantine instead of losing it. Keep a week of quarantine.

Check the key the way an attacker would, four tests: opening a shell must be refused, reading must work, writing must be refused, escaping with .. must be refused.

3. Restore, for real

Once a month, restore one guest from copy 2 into a new, unused ID on a test machine, and start it. A backup that has never been restored is a hope, not a backup.

Traps

The backup user must be able to read the archives: check their permissions after the first night, or copy 2 stays empty without an error.

Give the pulling machine a floor: if free space drops under a threshold, refuse to run and say so. A full disk in the middle of a copy leaves a half-written mirror.

A copy is only as good as the alarm that says it failed. Make the nightly job report its result somewhere you actually look.

Found this useful? Tell us what to measure next.