> ## Documentation Index
> Fetch the complete documentation index at: https://skaum.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Backup & Recovery (Litestream)

> Continuous SQLite replication and the restore runbook (VOT-372)

Continuous, off-server backups of the production/staging SQLite databases are
provided by [Litestream](https://litestream.io) via the
[`litestream`](https://github.com/fractaledmind/litestream-ruby) Ruby gem. The
Litestream binary runs as a Puma plugin inside the app container and streams the
WAL to S3-compatible object storage in near real time.

## What is replicated

Litestream config lives in `config/litestream.yml`. Two databases are replicated:

| Database | File | Replica path | Why |
| - | - | - | - |
| **primary** | `storage/<env>.sqlite3` | `<env>/primary` | All application data (votes, polls, protokoll, orgs). Losing it is unrecoverable. |
| **queue** | `storage/<env>_queue.sqlite3` | `<env>/queue` | Solid Queue jobs — in-flight/scheduled work (protokoll exports, magic-link sends) we don't want to lose on host death. |

Deliberately **not** replicated: `cache` (Solid Cache — fully regenerable) and
`cable` (Solid Cable — transient Action Cable messages). Both would be pure noise.

`storage/` is a persisted Kamal volume (`voteringsapp_storage:/rails/storage` in
production, `/volume1/docker/votering:/rails/storage` on staging), so Litestream's
generation/temp files survive container restarts.

## Enabling backups (operator, one-time per environment)

Litestream is **ENV-gated** and inert until a replica bucket is configured — the
Puma plugin (`config/puma.rb`) only loads when `LITESTREAM_REPLICA_BUCKET` is set,
mirroring the Sentry DSN-gating philosophy. Un-provisioned deploys, dev, and test
are completely unaffected.

To turn backups on, set these secrets in Doppler (project `votering`) and redeploy:

| Secret | Required | Notes |
| - | - | - |
| `LITESTREAM_REPLICA_BUCKET` | Yes | Bucket name. **Presence of this is what enables Litestream.** |
| `LITESTREAM_ACCESS_KEY_ID` | Yes | S3-compatible access key id. |
| `LITESTREAM_SECRET_ACCESS_KEY` | Yes | S3-compatible secret access key. |
| `LITESTREAM_REPLICA_ENDPOINT` | Optional | Only for non-AWS providers (Backblaze B2, Cloudflare R2, MinIO). |
| `LITESTREAM_REPLICA_REGION` | Optional | Region for AWS S3 / Backblaze B2. |

```bash theme={null}
doppler secrets set LITESTREAM_REPLICA_BUCKET=my-bucket --config prd
doppler secrets set LITESTREAM_ACCESS_KEY_ID=... --config prd
doppler secrets set LITESTREAM_SECRET_ACCESS_KEY=... --config prd
# optional, non-AWS:
doppler secrets set LITESTREAM_REPLICA_ENDPOINT=s3.eu-central-003.backblazeb2.com --config prd
kamal deploy   # or: KAMAL_DESTINATION=staging kamal deploy --destination staging
```

Set the equivalent secrets on the `stg` config for staging.

### How it is wired

* The Puma plugin (`plugin :litestream` in `config/puma.rb`) runs the Litestream
  binary inside the app container, once, from the Puma master.
* `config/litestream.yml` is read raw by the Go binary, so it must not use ERB;
  it relies on Litestream's own environment expansion (`${RAILS_ENV}`,
  `$LITESTREAM_*`).
* `config/initializers/litestream.rb` feeds the `LITESTREAM_*` variables into the
  gem's config.
* The secrets reach the container through Kamal's `env.secret` and Doppler
  (`prd` / `stg`). `.kamal/secrets*` read them with `|| echo ""` fallbacks, so a
  deploy succeeds before the bucket is provisioned.

## Checking backup status

```bash theme={null}
# Are replicas being written? (lists configured dbs + generations)
kamal app exec "bin/rails litestream:databases"

# End-to-end proof a db is restorable (writes a sentinel, waits, restores, checks)
kamal app exec "bin/rails litestream:verify -- storage/production.sqlite3"
```

### Automated freshness monitoring

`LitestreamVerificationJob` runs daily at 04:00 UTC — 06:00 Oslo time in summer,
05:00 in winter; the schedule names UTC, so the app zone does not move it
(`config/recurring.yml`) — and
calls `Litestream.verify!` against each replicated database. On failure or
staleness it reports to **Sentry** (`Sentry.capture_exception`) **and** the Rails
log (`Rails.logger.error`), so a broken/stale backup is observable. The job
no-ops when Litestream isn't configured.

## Restore runbook

Restoring rebuilds a database file from its replica. Do this on the server (the
config + credentials live there).

```bash theme={null}
# 1. Stop the app so nothing writes to the db during restore.
kamal app stop

# 2. Restore each replicated database. --database must match a `path:` in
#    config/litestream.yml (after ${RAILS_ENV} expansion).
kamal app exec "bin/rails litestream:restore -- --database=/rails/storage/production.sqlite3 -o /rails/storage/production.sqlite3"
kamal app exec "bin/rails litestream:restore -- --database=/rails/storage/production_queue.sqlite3 -o /rails/storage/production_queue.sqlite3"

# 3. Bring the app back up.
kamal app boot
```

Point-in-time recovery is supported via `-timestamp` on the restore command; see
the [Litestream restore reference](https://litestream.io/reference/restore/).

> [!WARNING]
>
> **Restore rehearsal is mandatory before relying on backups.** A backup you have
> never restored is not a backup. After enabling Litestream in an environment,
> perform at least one restore rehearsal against a **scratch** environment (never
> production): provision a throwaway bucket + deploy, let replication run, then
> restore to a temp path and diff against the live db. This rehearsal requires a
> real bucket, credentials, and a deploy, so it is an **operator step** that cannot
> be automated from the app codebase.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.