Backup & Recovery (Litestream)

Continuous, off-server backups of the production/staging SQLite databases are provided by Litestream via the litestream Ruby gem. The Litestream binary runs as a Puma plugin inside the app container and streams the WAL to S3-compatible object storage in near real time.

What is replicated

Litestream config lives in config/litestream.yml. Two databases are replicated:

Database	File	Replica path	Why
primary	`storage/<env>.sqlite3`	`<env>/primary`	All application data (votes, polls, protokoll, orgs). Losing it is unrecoverable.
queue	`storage/<env>_queue.sqlite3`	`<env>/queue`	Solid Queue jobs — in-flight/scheduled work (protokoll exports, magic-link sends) we don’t want to lose on host death.

Deliberately not replicated: cache (Solid Cache — fully regenerable) and cable (Solid Cable — transient Action Cable messages). Both would be pure noise. storage/ is a persisted Kamal volume (voteringsapp_storage:/rails/storage in production, /volume1/docker/votering:/rails/storage on staging), so Litestream’s generation/temp files survive container restarts.

Enabling backups (operator, one-time per environment)

Litestream is ENV-gated and inert until a replica bucket is configured — the Puma plugin (config/puma.rb) only loads when LITESTREAM_REPLICA_BUCKET is set, mirroring the Sentry DSN-gating philosophy. Un-provisioned deploys, dev, and test are completely unaffected. To turn backups on, set these secrets in Doppler (project votering) and redeploy:

Secret	Required	Notes
`LITESTREAM_REPLICA_BUCKET`	Yes	Bucket name. Presence of this is what enables Litestream.
`LITESTREAM_ACCESS_KEY_ID`	Yes	S3-compatible access key id.
`LITESTREAM_SECRET_ACCESS_KEY`	Yes	S3-compatible secret access key.
`LITESTREAM_REPLICA_ENDPOINT`	Optional	Only for non-AWS providers (Backblaze B2, Cloudflare R2, MinIO).
`LITESTREAM_REPLICA_REGION`	Optional	Region for AWS S3 / Backblaze B2.

doppler secrets set LITESTREAM_REPLICA_BUCKET=my-bucket --config prd
doppler secrets set LITESTREAM_ACCESS_KEY_ID=... --config prd
doppler secrets set LITESTREAM_SECRET_ACCESS_KEY=... --config prd
# optional, non-AWS:
doppler secrets set LITESTREAM_REPLICA_ENDPOINT=s3.eu-central-003.backblazeb2.com --config prd
kamal deploy   # or: KAMAL_DESTINATION=staging kamal deploy --destination staging

Set the equivalent secrets on the stg config for staging.

How it is wired

The Puma plugin (plugin :litestream in config/puma.rb) runs the Litestream binary inside the app container, once, from the Puma master.
config/litestream.yml is read raw by the Go binary, so it must not use ERB; it relies on Litestream’s own environment expansion (${RAILS_ENV}, $LITESTREAM_*).
config/initializers/litestream.rb feeds the LITESTREAM_* variables into the gem’s config.
The secrets reach the container through Kamal’s env.secret and Doppler (prd / stg). .kamal/secrets* read them with || echo "" fallbacks, so a deploy succeeds before the bucket is provisioned.

Checking backup status

# Are replicas being written? (lists configured dbs + generations)
kamal app exec "bin/rails litestream:databases"

# End-to-end proof a db is restorable (writes a sentinel, waits, restores, checks)
kamal app exec "bin/rails litestream:verify -- storage/production.sqlite3"

Automated freshness monitoring

LitestreamVerificationJob runs daily at 04:00 UTC — 06:00 Oslo time in summer, 05:00 in winter; the schedule names UTC, so the app zone does not move it (config/recurring.yml) — and calls Litestream.verify! against each replicated database. On failure or staleness it reports to Sentry (Sentry.capture_exception) and the Rails log (Rails.logger.error), so a broken/stale backup is observable. The job no-ops when Litestream isn’t configured.

Restore runbook

Restoring rebuilds a database file from its replica. Do this on the server (the config + credentials live there).

# 1. Stop the app so nothing writes to the db during restore.
kamal app stop

# 2. Restore each replicated database. --database must match a `path:` in
#    config/litestream.yml (after ${RAILS_ENV} expansion).
kamal app exec "bin/rails litestream:restore -- --database=/rails/storage/production.sqlite3 -o /rails/storage/production.sqlite3"
kamal app exec "bin/rails litestream:restore -- --database=/rails/storage/production_queue.sqlite3 -o /rails/storage/production_queue.sqlite3"

# 3. Bring the app back up.
kamal app boot

Point-in-time recovery is supported via -timestamp on the restore command; see the Litestream restore reference.

Restore rehearsal is mandatory before relying on backups. A backup you have never restored is not a backup. After enabling Litestream in an environment, perform at least one restore rehearsal against a scratch environment (never production): provision a throwaway bucket + deploy, let replication run, then restore to a temp path and diff against the live db. This rehearsal requires a real bucket, credentials, and a deploy, so it is an operator step that cannot be automated from the app codebase.

​What is replicated

​Enabling backups (operator, one-time per environment)

​How it is wired

​Checking backup status

​Automated freshness monitoring

​Restore runbook