0040. Workload nodes back up their own data, into repositories of their own
Status: accepted, 2026-10-30. Refines 0014, which foresaw backups of apps going
through their nodes' agents.
Context
App databases were backed up by streaming each dump from its node through appsd into restic on the control plane,
so every byte crossed the 1 vCPU server twice and was compressed and encrypted there. Volumes weren't backed up at
all. A move between nodes copies volumes from the old node, so it needs that node alive: a dead node lost every
volume on it. Workload nodes run other people's code; giving one the platform repository's password would let a
container escape read every secret of the instance. A node's owner already sees what runs on it (ADR 0008).
Decision
- The node does the work. restic runs on the node that holds the data: database apps' dumps and volumes.
backupsd sends jobs through appsd over the agent's connection (ADR 0008); appsd never stores their credentials;
the node keeps a finished job's result until appsd has it.
- appsd owns what the data means: which databases and volumes exist, each volume's consistency (
dump for
database apps, live, stop, hook with commands run in the container, skip with a reason), the engines'
commands, stopping and starting apps. A team's admins set their volumes' modes and restore their own app's data
from apps, which asks backups as a service.
- A repository per node and destination. Its password is derived from a secret in the recovery kit, so nodes
joining don't change the kit. A node gets only its own repositories' passwords; the target of an evacuation from
backups gets the dead node's for that job, recorded in the journals. A node reaches the storage with its own
client (a vault), its own key (S3, when given; SFTP keys are derived per node), or the destination's key where a
person allowed it.
- A node's workloads leave it with one command.
xvii apps nodes evacuate moves every project environment
off a live node through the existing moves, and from a dead one restores each environment on another node from
the newest snapshots, saying how much is lost. The old node is fenced by its desired state; its volumes stay,
labelled, until a person copies from them or removes them.
Consequences
- No app data passes through the control plane; nodes spend their own CPU at night, and need restic 0.17 or newer.
- Where nodes share a storage account with the platform's repository (one SFTP user, one S3 key), a compromised
node can delete what that account holds; the posture says so, and the protected copy (ADR 0041) is the backstop.
- Volumes of database images default to being copied with the app stopped, a few seconds to minutes at night.
- An image built on a node that died is lost until builds push to the instance's registry; an evacuation lists
such apps as blocked until they are deployed again.
- Design:
superpowers/specs/2026-10-30-backups-dr-design.md in this repository.