Backups and restores
backups backs up everything the instance needs to come back: every service's
database and data folder, /etc/xvii, the release store, and every database apps manages.
Destinations
| Destination |
Notes |
| S3-compatible |
any provider; a bucket and a key with access to it |
| SFTP |
another server; backups makes an Ed25519 key pair, you add the public half there |
| Local folder |
a second disk; not a protection against losing the server |
| Telegram channel |
AES-256-GCM encrypted 19.9 MB parts and a manifest; full snapshots, no deduplication |
S3, SFTP and local folders are restic repositories: encrypted, deduplicated, incremental. Use at least
one destination outside the control plane's provider.
Plans
A plan has a schedule, its sources, its destinations and a retention per source. A plan with no sources
takes everything, including services installed later. Each run makes one snapshot per source.
| Source |
Is |
/etc/xvii |
domain, desired services and versions, settings and generated secrets |
| releases |
the release store |
db:xvii_<service> |
a platform service's database |
| data folders |
each platform service's folder under /var/lib/xvii (keys, files) |
app:<id> |
a database apps manages, streamed from its node |
The recovery kit
When a destination is created, backups shows a recovery kit: the repository passwords and storage keys
needed to restore without a working instance. Download it and keep it somewhere that isn't the instance.
Later it can be shown again only in a logged panel session.
Verification and drills
- Weekly, each repository is checked with
restic check --read-data-subset 5%.
- Monthly, a drill restores every platform database into a scratch database and checks it.
- Failures send
backup.failed, backup.verify_failed to notify. Subscribe to them.
A backup you haven't restored is a hope, not a backup: restore one into a test server once.
Restore part of it
From the backups panel:
- a database, into a new name or in place;
- a folder;
- one file, as a download;
- an app's database, in place.
A restore replaces data. Stop writers, confirm the target, keep a fresh backup of what you are about to
replace, and check the application afterwards with real reads and writes.
Restore the whole instance
On a fresh server, with the recovery kit:
xvii server restore root@203.0.113.7 --kit xvii-recovery.txt
Over SSH it installs restic, PostgreSQL, Caddy and xvii-node, then restores from the newest snapshots:
/etc/xvii (the domain, services, settings and secrets), the releases, and every service's database and
data folder. The services start with their data. Point the instance's DNS at the new server for
certificates and sign-in to work.
Then check: sign in, open every panel, check that nodes reconnect (xvii apps nodes) and name servers
pull. Nodes keep running their apps while the control plane is away and reconnect on their own.
Keys
Several services seal secrets with a key in their data folder (env's master key, apps' seal key, domains'
DNSSEC keys). Those folders are part of the backup. A database restored without its folder has secrets
nobody can open, and a lost DNSSEC key with a DS still published is a DNS outage.