Self-Hosted Database Restore

The RunWhen self-hosted platform uses Zalando’s Spilo image (Postgres + Patroni + WAL-G) for its database. Backups are written continuously to S3 via WAL-G. This guide covers restoring from the latest backup when you have direct pod access — the minimal path.

Before you start

Confirm you actually need a restore:

helm test <release> -n <namespace> --filter name=<release>-postgresql-health-test

If the health check passes, you don’t need to restore.

When to restore

Patroni DCS (ConfigMap) lost or corrupted — cluster won’t bootstrap
PGDATA PVC deleted or data corrupted
Schema-level corruption that application migrations cannot fix
Cross-environment refresh (e.g., prod → staging)

When NOT to restore

A single table is corrupted — use pg_dump -t from a logical backup instead
Patroni reports (uninitialized) but Postgres is healthy — this is a chart label bug fixed in chart ≥0.2.48; upgrade instead
You just need to reset app data — run db-init or use psql to nuke specific tables

Prerequisites

kubectl access to the cluster namespace
Pod-level exec access to the Spilo container
The Spilo pod is running and wal-g is configured (no additional setup needed)

Listing available restore points

NS=<namespace>         # e.g. runwhen-env-staging
RELEASE=<release>      # e.g. rw

# List base backups
kubectl -n "$NS" exec "${RELEASE}-postgresql-0" -c postgres -- \
  envdir /run/etc/wal-e.d/env wal-g backup-list

The latest backup is the default restore target. Each line is a full snapshot:

backup_name                         modified             wal_file_name
base_000000780000010D00000088       2026-08-07T02:01:01Z  000000780000010D00000088

Simple restore procedure (preferred path)

This is the minimal procedure — exec into the running pod, restore directly, promote. No Helm upgrades, no DCS ConfigMap deletions, no scaling contortions.

Step 1 — Scale down the replica

Kill the replica so it doesn’t get confused when the primary’s timeline changes:

kubectl -n "$NS" scale sts "${RELEASE}-postgresql" --replicas=1
kubectl -n "$NS" wait --for=delete pod/"${RELEASE}-postgresql-1" --timeout=120s
kubectl -n "$NS" delete pvc "pgdata-${RELEASE}-postgresql-1" --ignore-not-found

The replica’s PVC is deleted so it gets a fresh one on scale-up and does a clean pg_basebackup from the restored primary.

Step 2 — Restore inside the primary pod

kubectl -n "$NS" exec "${RELEASE}-postgresql-0" -c postgres -- bash -c '
  set -e

  echo "=== Pausing Patroni ==="
  patronictl -c /run/postgres.yml pause --wait 2>/dev/null || true

  echo "=== Stopping postgres ==="
  su - postgres -c "pg_ctl -D /home/postgres/pgdata/pgroot/data -m fast stop" || true

  echo "=== Clearing old data ==="
  rm -rf /home/postgres/pgdata/pgroot/data
  mkdir /home/postgres/pgdata/pgroot/data

  echo "=== Restoring LATEST backup ==="
  envdir /run/etc/wal-e.d/env wal-g backup-fetch /home/postgres/pgdata/pgroot/data LATEST

  echo "=== Signaling archive recovery ==="
  touch /home/postgres/pgdata/pgroot/data/recovery.signal

  echo "=== Starting postgres ==="
  su - postgres -c "pg_ctl -D /home/postgres/pgdata/pgroot/data start"

  echo "=== Waiting for recovery ==="
  until su - postgres -c "psql -tAc \"SELECT pg_is_in_recovery()\"" 2>/dev/null | grep -q "^f$"; do
    sleep 5
  done
  echo "=== Recovery complete, primary accepting writes ==="

  echo "=== Resuming Patroni ==="
  patronictl -c /run/postgres.yml resume
'

If wal-g backup-fetch fails with Directory must be empty, the data directory wasn’t fully cleared. Restore to a temp location instead:
Terminal window
envdir /run/etc/wal-e.d/env wal-g backup-fetch /tmp/pg_restore LATEST
cp -a /tmp/pg_restore/* /home/postgres/pgdata/pgroot/data/
chown -R postgres:postgres /home/postgres/pgdata/pgroot/data

Step 3 — Verify

kubectl -n "$NS" exec "${RELEASE}-postgresql-0" -c postgres -- \
  psql -U postgres -tAc "SELECT pg_is_in_recovery()"
# → f

kubectl -n "$NS" exec "${RELEASE}-postgresql-0" -c postgres -- \
  psql -U postgres -d core -c 'select count(*) from workspaces;'
# → pre-backup count

helm test "$RELEASE" -n "$NS" --filter name=${RELEASE}-postgresql-health-test

Step 4 — Scale the replica back

kubectl -n "$NS" scale sts "${RELEASE}-postgresql" --replicas=2

The replica will automatically pg_basebackup from the primary.

Manual recovery steps

If the automated procedure above doesn’t work, or you prefer to run each step individually:

Inside the pod

# 1. Stop postgres
su - postgres -c "pg_ctl -D /home/postgres/pgdata/pgroot/data -m fast stop" || true

# 2. Clear old data
rm -rf /home/postgres/pgdata/pgroot/data/*
ls /home/postgres/pgdata/pgroot/data   # confirm empty

# 3. Restore latest backup
envdir /run/etc/wal-e.d/env wal-g backup-fetch /home/postgres/pgdata/pgroot/data LATEST

# 4. Signal archive recovery
touch /home/postgres/pgdata/pgroot/data/recovery.signal

# 5. Start postgres
su - postgres -c "pg_ctl -D /home/postgres/pgdata/pgroot/data start"

# 6. Promote to primary
pg_ctl -D /home/postgres/pgdata/pgroot/data promote

# 7. Verify
psql -U postgres -tAc "SELECT pg_is_in_recovery()"    # → f
psql -U postgres -d core -c 'select count(*) from workspaces;'

Notes

The rm -rf removes contents of the data directory, not the directory itself
recovery.signal tells Postgres to replay WAL from the archive on next start
pg_ctl promote ends recovery and makes the server accept writes
Restoring to /tmp first avoids WAL-G’s delta-mode check that requires an empty directory

Cross-environment restore

To restore one environment’s database from another environment’s backup (e.g., prod → staging):

# In your values overlay, enable the chart-level clone mechanism:
postgresql:
  spilo:
    restore:
      enabled: true
      scope: "rw-prod-postgres"                                    # source scope
      s3Prefix: "s3://postgres-backups/spilo/rw-prod-postgres"     # source S3 path
      targetTimeline: "current"

Then scale down, wipe DCS, and scale up — see the full chart RESTORE.md for the procedure.

Verification checklist

Run all of these before declaring the restore complete:

kubectl get pods -l app.kubernetes.io/component=postgresql — both pods Ready 1/1
patronictl -c /run/postgres.yml list — exactly one Leader, one Replica in streaming
SELECT pg_is_in_recovery() — f on the primary
SELECT count(*) FROM workspaces — pre-backup count
helm test <release> --filter name=<release>-postgresql-health-test — pass
kubectl rollout restart deploy <release>-papi — app login works end-to-end

Known failure modes

Symptom	Cause	Fix
`Directory for delta base must be empty`	Data dir wasn’t fully cleared before `wal-g backup-fetch LATEST`	Restore to `/tmp/pg_restore` and copy into place
`NoCredentialProviders` / `X-Amz-Credential mal-formed`	Spilo’s clone env `AWS_ACCESS_KEY_ID` receives the S3 prefix instead of the key ID	Copy credentials from `/run/etc/wal-e.d/env/` into `/run/etc/wal-e.d/env-clone-<scope>/`
`requested timeline N is not a child`	Poison `.history` file in S3 archive (from prior `pg_resetwal`)	Audit and delete `.history` files with timeline higher than the backup from `wal_005/` in the bucket
`WAL ends before end of online backup`	Newest backup’s tail WAL wasn’t archived at backup time	Retry after 30s or restore the previous backup
`following a different leader` loop in Patroni	DCS state is stale — Patroni thinks it’s a replica with no leader	`patronictl remove <scope> --force` to reset DCS
Patroni shows `(uninitialized)` but Postgres is healthy	Chart label bug < 0.2.48	Upgrade the chart

Rollback

There is no clean undo once PGDATA is deleted. The pre-restore data is recoverable only from:

The S3 base backup + WAL just restored from (repeat this procedure)
An out-of-band pg_dump taken before starting

Always take a safety dump before a restore:

kubectl -n "$NS" exec "${RELEASE}-postgresql-0" -c postgres -- \
  pg_dumpall -U postgres --clean --if-exists \
  | gzip > /tmp/${RELEASE}-preRestore-$(date +%Y%m%dT%H%M%SZ).sql.gz

Store it outside the cluster.