Skip to content
RunWhen

External S3

Use objectStorage.kind: external when object storage is provided by your cloud or platform team — AWS S3, Google Cloud Storage (S3-compatible HMAC keys), Azure Blob (S3 gateway), Backblaze B2, Wasabi, an external SeaweedFS cluster, MinIO, JFrog Artifactory S3 backend, etc.

The chart stops rendering SeaweedFS (seaweedfs.deploy: false) and wires every consumer to your endpoint via S3_INTERNAL_HOST, S3_HOST, and S3_ACCESS_KEY / S3_SECRET_KEY in the platform ConfigMap/Secret.

When to use which backend

objectStorage.kindChart rendersBest for
seaweedfs (default)SeaweedFS subchart + bucket-init JobSelf-contained installs, air-gap friendly, namespace-scoped
externalNothing — env vars + credentials onlyEnterprise managed S3, existing object-store fleet, multi-env shared bucket account

The legacy minio kind was removed — helm template fail-fasts if an old overlay still sets it.

Requirements

S3 API compatibility

RequirementDetail
APIS3-compatible REST API (SigV4 signing). All platform consumers use boto3 / aws-sdk-go paths.
Path-style vs virtual-hostedSome backends (SeaweedFS, MinIO) prefer path-style addressing. If you see redirect loops or BucketRegionError, set AWS_S3_FORCE_PATH_STYLE=true in consumer overlays or front the bucket with a path-based gateway.
Region labelobjectStorage.region / S3_REGION must be non-empty (many backends ignore the value but SDKs require it). Use the provider’s region string (e.g. us-east-1, auto).
TLSIn-cluster traffic should use TLS where policy requires it. Set internalHost to https://… when the endpoint expects HTTPS.
Multipart uploadsWorkspace artifacts and Mimir blocks use multipart PUTs — ensure load balancers allow large bodies (no accidental client-body-buffer-size: "0" on nginx).

Network

PathPurpose
objectStorage.external.internalHostHost:port (or URL) pods use for S3 API calls (e.g. s3.us-east-1.amazonaws.com, 10.0.0.5:9000, VPC endpoint DNS)
objectStorage.external.hostPublic hostname baked into presigned URLs for browser/runner uploads (e.g. s3.example.com, CloudFront distribution, or same as internal if runners reach it)

At least one of host or internalHost must be set — the chart fail-fasts otherwise.

Credentials

Access key and secret are rendered into <release>-platform-secrets as S3_ACCESS_KEY and S3_SECRET_KEY. Supply them via:

  • objectStorage.accessKey / objectStorage.secretKey in values (lab only), or
  • secrets.values.s3AccessKey / secrets.values.s3SecretKey, or
  • Pre-populate the platform Secret out-of-band (ExternalSecrets, Vault CSI)

The IAM (or equivalent) principal needs read/write/list on every bucket below.

Buckets (operator-provisioned)

Unlike bundled SeaweedFS, the chart does not create buckets when kind=external. Pre-create each bucket before install or first upload:

Bucket (default name)Purpose
shared-workspaceWorkspace file uploads, presigned URL target (SHARED_WORKSPACE_BUCKET)
mimir-blocksMimir TSDB block storage
mimir-rulerMimir ruler state
mimir-alertmanagerMimir Alertmanager state
agentfarm-artifactsAgentFarm ADK chat artifacts (when agentfarm.artifacts.serviceType=s3)
postgres-backupsSpilo WAL-G (only if postgresql.kind=spilo and WAL-G enabled)
vault-backupsVault backup CronJob (when vault.backup.enabled)

Names are configurable via objectStorage.buckets[] and related sub-keys — the table lists chart defaults. Keep names DNS-compatible (lowercase, hyphens).

Helm configuration example

Turn off SeaweedFS and point at external S3:

objectStorage:
  kind: external
  region: us-east-1
  accessKey: ""          # prefer secrets.values or BYO Secret
  secretKey: ""
  sharedWorkspaceBucket: shared-workspace
  buckets:
    - shared-workspace
    - mimir-blocks
    - mimir-ruler
    - mimir-alertmanager
    - agentfarm-artifacts
    - postgres-backups
    - vault-backups
  external:
    backend: s3          # logical label; app enum accepts gcs|minio — external S3 uses minio client path
    host: s3.us-east-1.amazonaws.com
    internalHost: s3.us-east-1.amazonaws.com
    region: us-east-1
    bucket: ""             # optional; defaults to sharedWorkspaceBucket


seaweedfs:
  deploy: false


secrets:
  method: k8s-secret
  values:
    s3AccessKey: "AKIA..."
    s3SecretKey: "..."

Example: AWS S3 (private cluster egress)

objectStorage:
  kind: external
  region: us-east-1
  external:
    backend: s3
    # Runners/browsers hit the regional endpoint in presigned URLs
    host: s3.us-east-1.amazonaws.com
    # Pods use the same endpoint (or a VPC interface endpoint hostname)
    internalHost: s3.us-east-1.amazonaws.com
    region: us-east-1


seaweedfs:
  deploy: false

IAM policy sketch (adjust ARNs):

{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Action": ["s3:GetObject", "s3:PutObject", "s3:DeleteObject", "s3:ListBucket"],
    "Resource": [
      "arn:aws:s3:::shared-workspace",
      "arn:aws:s3:::shared-workspace/*",
      "arn:aws:s3:::mimir-blocks",
      "arn:aws:s3:::mimir-blocks/*"
    ]
  }]
}

Repeat Resource entries for every bucket in objectStorage.buckets[].

Example: External SeaweedFS or MinIO (in another namespace)

objectStorage:
  kind: external
  region: us-east-1
  external:
    backend: s3
    host: s3.storage.example.com          # public DNS for presigned URLs
    internalHost: seaweedfs-s3.storage.svc.cluster.local:8333
    region: us-east-1


seaweedfs:
  deploy: false


ingress:
  enabled: true
  services:
    s3:
      # Chart S3 Ingress renders only when seaweedfs.deploy=true.
      # For external backends, expose `host` yourself (corporate ingress, LB, or DNS to the external gateway).
      proxyBodySize: "0"
      clientBodyBufferSize: "1m"

Ensure the external gateway’s credentials match S3_ACCESS_KEY / S3_SECRET_KEY in platform secrets.

Public presigned URLs

PAPI generates presigned URLs using S3_HOST (from objectStorage.external.host). When seaweedfs.deploy: false, the chart does not render the s3.<domain> Ingress — you must:

  1. Set external.host to a hostname runners and browsers can reach, and
  2. Route that hostname to your S3 gateway (corporate ingress, cloud front door, or provider public endpoint)

If runners only reach the internal endpoint, presigned URLs that embed the public host will fail unless network policy allows the public path.

Post-upgrade rollout

Changing objectStorage.kind updates ConfigMaps/Secrets but does not automatically restart Deployments. After helm upgrade, restart workloads that mount platform config:

Terminal window
kubectl -n runwhen-platform rollout restart deployment -l app.kubernetes.io/instance=rw-platform
# Include StatefulSets (mimir, etc.) as needed

Verify after install

  1. ConfigMap — kubectl get cm rw-platform-platform-config -o yaml | rg S3_
  2. Secret — S3_ACCESS_KEY present in rw-platform-platform-secrets
  3. Smoke PUT — from a debug pod: aws --endpoint-url https://$INTERNAL s3 ls s3://shared-workspace/ (install aws-cli image)
  4. UI upload — workspace file upload via presigned URL (validates S3_HOST)
  5. Mimir — blocks appearing in mimir-blocks after metrics flow

Troubleshooting

SymptomLikely cause
objectStorage.kind=external requires … host or internalHostMissing external.host / external.internalHost
Presigned URL 403/404Wrong host, IAM policy, or bucket missing
Upload timeout via ingressnginx client-body-buffer-size: "0" or body size limit — use positive buffer + proxy-body-size: "0"
Mimir empty after restartUnrelated local-path bug if on old chart — ensure Mimir PVC paths pinned (see chart release notes)
STORAGE_BACKEND validation errorsChart maps external S3 to minio client path; use STORAGE_BACKEND_KIND for observability
TopicLink
Self-hosted prerequisitesSelf-Hosted Deployment Requirements
External PostgreSQL (often paired)External PostgreSQL
JFrog / air-gap registryJFrog Registry Setup
GAR setupGAR Registry Setup