0046. A warm standby control plane: a streaming replica, mirrored files and releases, promoted by a person
Status: accepted, 2026-10-30. Amends 0004 (the one cluster on the control-plane server may
have one replica) and 0010; the vision's one control plane per instance
holds. Beside 0042: restoring stays the way back when no
standby is there.
Context
Everything the platform needs lives on one server: PostgreSQL, every service's data and seal key, Caddy's
certificates and the release store. Losing it means restoring the instance on a fresh server from backups
(ADR 0042): with point-in-time recovery the database loses minutes (ADR 0039), but the restore itself, a new
server and DNS take long enough that an outage is measured in an hour or more. The owner decided to prepare a
second control plane and buy its server later.
Decision
- A second server can be paired as a warm standby, by a superuser with a passkey. It runs PostgreSQL as an
asynchronous streaming replica of the control plane's cluster (TLS, a replication-only role, one address, a
slot capped so a lost standby can't fill the primary's disk, the WAL archive behind it when there is one); it
receives
/etc/xvii, the services' data folders and Caddy's storage from the primary's xvii-node every two
minutes over mutual TLS pinned at pairing; and it mirrors the release store from its signed index. Its
services are installed and never started.
- The standby is passive: no service runs there, nothing reads from it, and it never promotes itself. A person
runs
xvii-node failover on it: fence the old primary when it answers, promote, start the services and Caddy,
move the instance's DNS records the way a restore does, and print what else to change.
- Epochs keep a returning primary from serving: one that finds its peer promoted fences itself until it is made
the new standby.
- The mirror installs the standby's own services and serves releases to nodes' self-updates when the control
plane can't.
Consequences
- Recovery takes minutes once a person decides; the database loses the replication lag, files at most two
minutes.
- The standby costs a second server at least the primary's size, a WAL stream and a small file feed; the
primary's PostgreSQL listens on its public address for one other address.
- It is not a backup: a dropped table replicates. Backups and point-in-time recovery stay the backups service's.
- Split brain is prevented by procedure and epochs, not consensus: the runbook says to power the old primary off.
- Own name servers and relay nodes' firewalls learn the standby's address in advance; mail edges follow the
control plane's name (ADR 0042's design).
- Design:
superpowers/specs/2026-10-30-autopilot-and-thin-control-plane-design.md in this repository.