Backups and Restores

Three nightly layers, two alert paths, and one restore I've actually done

January 2026, Active

Project Overview

Everything my family uses on this server has to survive a dead drive, a bad upgrade, or me deleting the wrong folder. The ZFS mirror covers a dead drive. The rest is three backup layers that run every night, two separate alerts that tell me when one of them stops, and a restore I've done for real instead of assuming it works.

The Three Layers

1:00 AM Database dumps
1:30 AM Config backups
3:00 AM Backblaze B2 sync
  • Database Dumps: A script runs pg_dump against 12 Postgres databases and keeps 30 days of compressed dumps. Every dump uses --no-owner --no-acl, so it restores cleanly into a scratch database that doesn't have the production roles.
  • Config Backups: A second script archives the things that don't live in the synced folders or would be awkward to rebuild: the TrueNAS system config, the cron job definitions (they sit on the boot drive, which the off-site sync never sees), Frigate's config, and Mosquitto's config and data. It also keeps 30 days.
  • Off-site Sync: A TrueNAS Cloud Sync task (rclone underneath) pushes the pool's important folders to Backblaze B2, including app data, the Compose stacks, scripts, system docs, logs, the photo library, and Nextcloud files. The dumps and archives from the first two layers sit inside app data, so they go off-site too. Cloud Sync encrypts everything on the server with a password and salt before upload, so Backblaze only ever stores encrypted files.

What Stays Off Backblaze

Leaving things out is a decision too, so each exclusion has a reason:

  • The Media Library: Movies and shows can be downloaded again by the *arr apps. Paying to store them twice isn't worth it.
  • Camera Footage: Frigate keeps a rolling four-day window and trims itself, so there's nothing long-lived to keep.
  • Live Database Files: A Postgres data folder copied mid-write isn't a usable backup. The nightly dump is the real copy, so the live folders are excluded.

The Live-Write Lesson

For a while the B2 sync kept failing with hash and size mismatches on the same handful of files, run after run. The first time it showed up I treated it as an Immich photo problem. It wasn't. rclone was reading files while the app that owned them was still writing to them.

  • The Fix: Exclude the whole class of live-write files at any depth instead of chasing one app at a time: SQLite write-ahead and shared-memory files, Postgres WAL folders, other WAL directories, and Redis state.
  • Why It Matters: A new stack is covered before it ever fails, because the rule matches the file type, not the app.
  • Written Down: All sixteen exclude rules are recorded with the reason for each and what still covers that data.

How I Know It Worked

In July 2026 the config backups stopped for 25 days and nothing told me. The script had lost its execute permission after an edit, so the nightly job failed every night in silence. That's the failure I design against now: a backup that quietly stops.

  • Database Check (2:15 AM): Every database's newest dump has to be less than 36 hours old. The result goes to Healthchecks.io, which alerts me by email and Signal, and a failure also emails me the full report straight from the server.
  • Config Check (3:00 AM): If the config backup's log is missing or more than 36 hours old, I get an email.
  • Two Paths on Purpose: Healthchecks.io runs outside my network, so it still alerts me if the check never reports at all, even with the whole server down. The server's own email is the second path, so one dead channel can't hide a failed backup.

The Restore Drill

A backup you've never restored is a guess. On July 30, 2026 I pulled the World Cup Pool database dump down from Backblaze B2, restored it into a scratch Postgres container, and compared it to the live database. The row counts matched exactly, and the whole thing took a few minutes.

The next day I spot-checked B2 for the file side: real, recent photos from the Immich library and real documents from Nextcloud were there.

What's Not Proven Yet

  • Watch-History Database: One stack runs TimescaleDB, which needs special handling to restore. The nightly dump is there, but I haven't restored it yet. It holds Plex watch history, so it's a gap I can live with for now.
  • Files Coming Back Down: I've confirmed photos and documents are in B2, but I haven't pulled one back down and opened it. The database half of the chain is proven; the file half is only checked.

Setting Up a New Stack

One new app ran for 12 days with no database backup because I skipped adding it to the backup script. Nothing failed, so nothing alerted. Now two steps happen in the same sitting as the deploy:

  • Add the Database: Put it in the dump script and the freshness check together, then run both once to see it work.
  • Check for Live Files: Look for a database or cache folder under a name the exclude rules won't catch, before the B2 sync finds it the hard way.

Related Projects

Back to Projects