Self-Hosted Database Troubleshooting

This guide covers common database problems in the RunWhen self-hosted platform and their fixes. It assumes you’re using the built-in Spilo (Postgres + Patroni + WAL-G) backend.

Quick reference

Symptom	Most likely fix
`helm test` health check fails	Check required databases/roles exist; run `db-init`
Patroni shows `(uninitialized)`	Upgrade chart to ≥0.2.48 (label bug)
Postgres won’t start after restore	Check for poison timeline history files in S3
`wal-g backup-list` hangs	S3 endpoint unreachable or auth is wrong
Replica stuck in `starting`	WAL-G timeout too low or S3 is slow

Health check failures

The platform ships a read-only health check Job that runs after every install/upgrade:

helm test <release> -n <namespace> --filter name=<release>-postgresql-health-test

The check verifies:

Required databases exist (core, usearch, agentfarm, litellm)
Required roles exist
Required tables exist with minimum row counts
Primary is not in recovery

Missing databases or roles

If the health check reports missing databases, the db-init Job may not have run:

# Check if the db-init Job exists
kubectl -n <ns> get job <release>-db-init

# Re-run it
helm -n <ns> upgrade <release> ./charts/runwhen-platform -f your-values.yaml \
  --set postgresql.dbInit.force=true

Missing tables or data

If databases and roles exist but tables or rows are missing, the application migrations may not have completed:

# Check migration status
kubectl -n <ns> logs deployment/<release>-papi | grep -i migration

# Restart PAPI to trigger migrations
kubectl -n <ns> rollout restart deployment/<release>-papi

Patroni cluster issues

`(uninitialized)` in `patronictl list`

Symptom: patronictl list shows (uninitialized) for all members but Postgres is healthy and the REST API reports role: leader.

Cause: A chart bug before 0.2.48 — Patroni’s Kubernetes DCS ConfigMap labels are asymmetric. The daemon writes ConfigMaps with application=spilo + version=<scope> but the read selector includes extra labels from PATRONI_KUBERNETES_LABELS.

Fix: Upgrade the chart to ≥0.2.48. No data loss — it’s a label mismatch, not a database problem.

`following a different leader` loop

Symptom: Patroni repeatedly logs My wal position exceeds maximum replication lag and following a different leader because i am not the healthiest node.

Cause: DCS (ConfigMap) state is stale — Patroni thinks this node is a replica but no leader exists. Common after a failed restore or when DCS outlives the cluster.

Fix: Reset the DCS state:

kubectl -n <ns> delete cm <release>-postgres-config <release>-postgres-leader <release>-postgres-sync --ignore-not-found
kubectl -n <ns> delete pod <release>-postgresql-0

Patroni restarts, sees local data with empty DCS, and promotes itself.

Replica won’t join the cluster

Symptom: The replica pod is running but patronictl list shows it as starting or creating replica indefinitely.

Causes and fixes:

Cause	Check	Fix
Replica’s PVC has old data on wrong timeline	`kubectl logs <release>-postgresql-1 -c postgres` shows replication errors	`kubectl delete pvc pgdata-<release>-postgresql-1` and scale replica back up
`wal-g backup-list` hangs in replica bootstrap	S3 endpoint unreachable from replica node	Set `createReplicaMethods: [basebackup, wal_e]` so `pg_basebackup` from the leader is tried first
Startup probe timeout	Large DB takes >10min for `pg_basebackup`	Increase `postgresql.spilo.probes.startup.failureThreshold`

WAL-G / backup issues

`wal-g backup-list` hangs

Symptom: Running wal-g backup-list from inside the pod produces no output and never returns.

Causes:

S3 endpoint is unreachable from the pod
S3 credentials are wrong or expired
WALG_NETWORK_TIMEOUT is too high (default 60s in chart ≥0.2.35)

Fix:

# Check S3 connectivity from inside the pod
kubectl exec -n <ns> <release>-postgresql-0 -c postgres -- \
  curl -s -o /dev/null -w "%{http_code}" <s3-endpoint>

# Verify env vars are set correctly
kubectl exec -n <ns> <release>-postgresql-0 -c postgres -- \
  envdir /run/etc/wal-e.d/env env | grep -E 'AWS_|WALG_|S3_'

`wal-g wal-fetch` hangs during recovery

Symptom: Postgres recovery hangs with no progress. Log shows repeated restarting after failure in progress and strace shows wait4 on a wal-g wal-fetch child.

Cause: WAL_RESTORE_TIMEOUT defaults to 0 (no timeout) in Spilo. A single missing WAL segment or timeline history file hangs recovery forever.

Fix:

# In values.yaml:
postgresql:
  spilo:
    walg:
      restoreTimeoutSeconds: 20   # Never set to 0

This bounds every wal-g wal-fetch call. A missing segment fails fast instead of hanging.

Restore-specific failures

Poison timeline history (`.history`) files

Symptom: Restore fails with requested timeline N is not a child of this server's history.

Cause: Someone ran pg_resetwal (or Patroni did), which incremented the timeline and pushed a new .history file into the S3 archive. Every subsequent restore fetches that file and dies.

Fix: Audit and delete poison history files from the S3 bucket:

# Inside the pod — find the backup's timeline from the backup name
# base_00000078... → timeline 0x78 = 120
BACKUP_TL=120

# List history files in wal_005/
envdir /run/etc/wal-e.d/env wal-g wal-show | grep "\.history"

# Delete any with hex prefix > backup timeline
# e.g., 00000079.history.lz4 → timeline 121 > 120 → poison

Clone-env credential bug

Symptom: After enabling restore.enabled: true and scaling down/up, the pod logs show NoCredentialProviders or X-Amz-Credential parameter is mal-formed.

Cause: Spilo’s configure_spilo.py writes the clone environment directory with AWS_ACCESS_KEY_ID containing the S3 prefix value instead of the access key ID.

Fix: Copy real credentials from the working env into the clone env dir:

kubectl exec -n <ns> <release>-postgresql-0 -- sh -c '
  cd /run/etc/wal-e.d
  SCOPE=$(grep -m1 "^scope:" /run/postgres.yml | awk "{print \$2}")
  CLONE_DIR="env-clone-${SCOPE}"
  for f in AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY AWS_ENDPOINT \
           WALG_S3_ENDPOINT AWS_S3_FORCE_PATH_STYLE WALG_DISABLE_S3_SSE \
           WALG_DOWNLOAD_CONCURRENCY WALG_UPLOAD_CONCURRENCY PGPORT; do
    cp "env/$f" "${CLONE_DIR}/$f"
  done
  envdir ${CLONE_DIR} wal-g backup-list | tail -3
'

Connectivity and operational issues

Can’t connect to the database

# Via the primary service
kubectl -n <ns> exec deploy/<release>-papi -- \
  psql "$(kubectl -n <ns> get secret <release>-platform-secrets \
    -o jsonpath='{.data.DATABASE_URL}' | base64 -d)"

# Via direct pod exec
kubectl -n <ns> exec <release>-postgresql-0 -c postgres -- \
  psql -U postgres -d core -c 'SELECT 1'

If neither works, check:

Is the Spilo pod running? kubectl -n <ns> get pods -l app.kubernetes.io/component=postgresql
Is Patroni healthy? kubectl -n <ns> exec <release>-postgresql-0 -c postgres -- patronictl -c /run/postgres.yml list
Is postgres accepting connections? kubectl -n <ns> exec <release>-postgresql-0 -c postgres -- pg_isready

PgBouncer connection issues

If applications connect via PgBouncer (port 6432) instead of directly to Postgres (5432):

# Check PgBouncer status
kubectl -n <ns> exec deploy/<release>-pgbouncer -- psql -U pgbouncer -d pgbouncer -c "SHOW STATS;"

# Restart PgBouncer if connections are stuck
kubectl -n <ns> rollout restart deployment/<release>-pgbouncer

Vault-specific database issues

Vault stores platform secrets. If Vault is unhealthy, database credentials may not be available.

# Check Vault status
kubectl -n <ns> exec <release>-vault-0 -- vault status

# Check Vault backup ran recently
kubectl -n <ns> get cronjob <release>-vault-backup
kubectl -n <ns> get jobs -l app.kubernetes.io/component=vault-backup

See Self-Hosted Database Restore for the full restore procedure.

Getting help

If you encounter a database issue not covered here:

Check the Install Friction Log for known chart-level issues
Check the chart RESTORE.md for the full operator restore runbook
Collect logs: kubectl -n <ns> logs <release>-postgresql-0 -c postgres --tail=100
Collect Patroni state: kubectl -n <ns> exec <release>-postgresql-0 -c postgres -- patronictl -c /run/postgres.yml list
Contact RunWhen support with the above information