Backups and recovery

What matters, in order

Item	Why	Loss means
Master key (SLIPWAY_MASTER_KEY, or your KMS key)	Decrypts every stored secret	Permanent loss of all stored secrets. There is no recovery
Postgres	All state: orgs, pipelines, runs, releases, deployments, audit log, encrypted secrets	Everything except logs and artifacts
Object storage	Logs, artifacts, caches	History of logs and artifacts. Caches rebuild themselves
NATS JetStream	In-flight queues and events	Mostly transient (see below)
Session secret	Signs cookies	Everyone is signed out. Nothing else

The master key

Store it separately from the database backup, in a secret manager or sealed offline copy. A database backup without the key restores structure but not usable secrets; the key without the backup is useless. If you use a KMS-backed keyring (AWS KMS, GCP KMS, Azure Key Vault, Vault transit), protect that key with deletion protection and key-policy review instead, and enable key rotation.

Postgres

• Enable point-in-time recovery (managed services do this; for self-managed use WAL archiving with pgBackRest or WAL-G).
• Take regular logical dumps too, so a restore is possible into any version:

pg_dump --format=custom --no-owner "$SLIPWAY_DATABASE_URL" > slipway-$(date +%F).dump

• Test restores on a schedule. An untested backup is a hope.
• Audit-log and run-event tables are partitioned by month; plan retention to match your plan’s limits.

Object storage

Enable versioning and a lifecycle policy matching your retention (log 14 days to 1 year and artifact 7 to 180 days depending on plan). Replicate cross-region if artifacts are release inputs you cannot rebuild. Releases reference artifacts by digest, so a restored database with missing artifacts will show them as unavailable rather than corrupt.

NATS

JetStream holds work queues (job dispatch, notifications, webhooks) and event streams. Loss during a failover is tolerated: the scheduler re-derives job state from Postgres leases, and undelivered notifications can be missed. Use 3 replicas with persistent volumes rather than backing NATS up.

Restore procedure

1. Stop server, scheduler and worker (scale to zero).
2. Restore Postgres to the desired point in time.
3. Ensure the same master key (or KMS key) is configured.
4. Start the control plane of the same version as the backup, or the version immediately above it so migrations apply forward.
5. Verify /readyz, sign in, run a pipeline, read a secret-using job’s output.
6. Re-register runners if their state files were lost (runners keep their own credential in state_file).

Disaster recovery targets

Tier	Suggested
Small team	Daily dumps, 7-day PITR, master key escrowed. RPO 24h, RTO hours
Production	Multi-AZ database with PITR, cross-region replica or backups, replicated object storage. RPO minutes, RTO under an hour with a tested runbook