0038. Backups are judged by coverage, freshness and drills
Status: accepted, 2026-10-30
Context
backups judged itself by its runs: a run that failed sent a critical event, and a source that never ran sent
nothing. The 2026-10-21 audit found an app database that had never been backed up for ten nights, sources that had
left their plan without a word, a heartbeat nobody pinged, and whole classes of data no plan could take: every
volume on every node, Caddy's certificates, PostgreSQL's roles, databases no service owns, the forge's own backups.
Drills restored each platform database into a scratch database and passed when it had a table; the whole instance
had been restored once, by hand, when it had five services. Nobody could say what wasn't backed up, how old the
newest copy of anything was, or how long a restore would take.
Decision
- A map of every stateful thing. backupsd keeps an inventory of sources from xvii-node on the control plane
(every database in the cluster, the roles,
/etc/xvii, every entry of the platform's data folder, the release
store, Caddy's binary and certificates, the name server's state), from apps (every database app and volume on
every node) and from external sources that report their own backups with a ping (a forge, a NAS). Each source is
covered, stale, degraded, failing, pending, uncovered, excluded (with a person's reason) or gone. What is not
backed up by design is listed with why; an entry nobody classified is uncovered.
- Freshness per source is what pages. A source's target is a person's value or its plans' interval plus two
hours. backupsd detects and sends keyed problems (ADR 0023): its freshest copy past target (critical for the
platform's data and protected environments), one destination's copy past target, a source no plan takes. A
failed run is a warning; the page comes when the target is missed. They all page as one group,
backups.
- A posture of eight checks (fresh, protected, off-site, point in time, drilled, recovery kit, verified,
classified) heads the map, home's instance page and the hub's attention items.
- Drills restore the real thing, in isolation, every month. An instance node a superuser gives the
drill
role restores the whole instance from a destination into a throwaway systemd-nspawn container, the way a real
restore would. Its network reaches only that destination and the distribution's mirrors, and closes before the
restored services start; every service honours drill mode (kit/drill) and does nothing outside. RTO and RPO
are measured against targets and reported to notify; a failed drill pages. Database drills and drills of app
databases and volumes on their nodes keep running beside it.
Consequences
- apps reports what data it holds and how it may be copied, so the map shows what no plan takes.
- Every service's spec gains a "Drill mode" line; kit gains
kit/drill. A service that still reaches out in a
drill is caught by the closed network and named in the drill's report.
- A drill node needs about the instance's memory and twice its data, and holds every secret of the instance for
the drill's length: only instance nodes may have the role.
- The observability design's
backup.failed becomes a warning; its seeded backup-stale rule stays as the
backstop for backupsd itself stopping.
- Design:
superpowers/specs/2026-10-30-backups-dr-design.md in this repository.