Self-Hosted LLM Configuration

Self-hosted installs configure LLMs in your Helm values overlay for the runwhen-platform chart. Two blocks work together:

Block	Purpose
llmGateway	In-cluster LiteLLM proxy — models[] routes to your vendor endpoints
llmBootstrap	Post-install Job — seeds PAPI’s provider/model catalog and mints a scoped virtual key

Prerequisites: Self-Hosted Deployment Requirements (cluster, registry, DNS, TLS).

---

Minimal shape

Enable bootstrap and define three models whose names match on both sides:

llmGateway:
  deploy: true
  providersExistingSecret: rw-vendor-llm   # recommended — see below
  models:
    - model_name: my-chat
      litellm_params:
        model: openai/gpt-4o
        api_base: https://your-endpoint.example/v1
        api_key: os.environ/OPENAI_API_KEY
    - model_name: my-bg
      litellm_params:
        model: openai/gpt-4o-mini
        api_base: https://your-endpoint.example/v1
        api_key: os.environ/OPENAI_API_KEY
    - model_name: my-embedding
      litellm_params:
        model: openai/text-embedding-3-small
        api_base: https://your-endpoint.example/v1
        api_key: os.environ/OPENAI_API_KEY
      model_info:
        mode: embedding

llmBootstrap:
  enabled: true
  staffUser:
    email: "platform-sa@yourcompany.example"
  provider:
    name: "company-litellm"
  models:
    chat:
      name: my-chat
      maxTokens: 0
    bg:
      name: my-bg
      maxTokens: 0
    embedding:
      name: my-embedding
      maxTokens: 0
      dimension: 1536   # MUST match your embedding model output size

The onboarding pack includes a fuller air-gap example (values-example-airgap-jcr.yaml in the chart repo) you can copy and adapt.

---

Provider credentials (two patterns)

Pattern B — BYO Secret (recommended for production)

Create a Secret in the release namespace before helm upgrade. The chart references it by name and never writes key material into values:

kubectl -n <platform-namespace> create secret generic rw-vendor-llm \
  --from-literal=OPENAI_API_KEY="sk-..."

llmGateway:
  providersExistingSecret: rw-vendor-llm

Reference keys in litellm_params as api_key: os.environ/OPENAI_API_KEY. Works with SealedSecrets, ExternalSecrets, SOPS, or Vault CSI.

Pattern A — inline providers map

Fine for smoke tests only — keys land in your values file:

llmGateway:
  providers:
    OPENAI_API_KEY: "sk-..."

---

Azure OpenAI / AI Foundry (direct)

Point LiteLLM at your Azure deployment names. Authentication is usually via Service Principal env vars on the gateway pod or keys in your BYO Secret.

llmGateway:
  providersExistingSecret: rw-azure-llm
  models:
    - model_name: company-gpt41-chat
      litellm_params:
        model: azure/my-gpt41-deployment
        api_base: https://my-account.openai.azure.com/
        api_version: "2024-02-01"
        api_key: os.environ/AZURE_API_KEY
    - model_name: company-gpt41-bg
      litellm_params:
        model: azure/my-gpt41-mini-deployment
        api_base: https://my-account.openai.azure.com/
        api_version: "2024-02-01"
        api_key: os.environ/AZURE_API_KEY
    - model_name: company-embedding
      litellm_params:
        model: azure/my-embedding-deployment
        api_base: https://my-account.openai.azure.com/
        api_version: "2024-02-01"
        api_key: os.environ/AZURE_API_KEY
      model_info:
        mode: embedding

For Claude on Foundry, use the azure_ai/claude-* LiteLLM route and the Foundry Anthropic base URL — see Azure OpenAI BYO LLM Setup.

---

Google Vertex AI (Model Garden)

Partner models (Claude, Gemini, Google embeddings) are managed APIs — you enable them in Model Garden, grant roles/aiplatform.user, and call publisher routes. No per-model Vertex Endpoint deploy is required for MaaS.

LiteLLM calls Vertex through the Google client libraries. How you authenticate is separate from the model list — pick one of the patterns below, then apply the same llmGateway.models[] / llmBootstrap shape in every case.

Authentication options

Pattern	When to use	Credential lands on
A. GKE Workload Identity (recommended on GKE)	Platform runs on GKE with WI enabled; you want no long-lived JSON keys	Dedicated serviceAccount.llmGateway only (chart ≥ 0.2.63)
B. Service account JSON key (BYO Secret)	Non-GKE clusters, WI not available, or policy requires explicit key rotation via Secret	llmGateway.providersExistingSecret → gateway pod env / file path
C. External LiteLLM proxy	You already operate a central LiteLLM that holds Vertex creds	Your external proxy — set llmGateway.deploy: false (see External LiteLLM gateway)

Do not annotate the shared platform ServiceAccount with Vertex WI unless you intentionally want every platform pod (PAPI, AgentFarm, …) able to impersonate the Vertex GSA. Pattern A exists to avoid that.

---

Pattern A — GKE Workload Identity (step-by-step)

Requires runwhen-platform chart ≥ 0.2.63 (serviceAccount.llmGateway). No API keys in Helm values or Secrets — Application Default Credentials on the llm-gateway pod resolve through WI.

A.1 Enable models in Vertex Model Garden for your GCP project (accept partner terms where prompted).

A.2 GCP service account — create a Google SA with roles/aiplatform.user on the project (or a narrower scope if your policy allows):

export PROJECT=my-project
export GSA=platform-vertex-llm@${PROJECT}.iam.gserviceaccount.com

gcloud iam service-accounts create platform-vertex-llm \
  --project="${PROJECT}" \
  --display-name="RunWhen platform Vertex LLM"

gcloud projects add-iam-policy-binding "${PROJECT}" \
  --member="serviceAccount:${GSA}" \
  --role="roles/aiplatform.user"

A.3 Dedicated llm-gateway ServiceAccount (Helm) — default create: false keeps legacy behaviour (gateway runs as <release>-platform). For Vertex on GKE, set create: true and annotate the llm-gateway identity:

serviceAccount:
  llmGateway:
    create: true
    annotations:
      iam.gke.io/gcp-service-account: platform-vertex-llm@my-project.iam.gserviceaccount.com

The chart renders Kubernetes SA <release>-llm-gateway (for example rw-platform-llm-gateway) and only the llm-gateway Deployment uses it. The llm-gateway DB reset Job still uses the platform SA — it does not call Vertex.

A.4 Workload Identity binding — allow that KSA to impersonate the GSA. Replace <namespace>, <release>, and project IDs:

export PROJECT=my-project
export NS=runwhen-platform
export RELEASE=rw-platform   # Helm release name → KSA rw-platform-llm-gateway
export GSA=platform-vertex-llm@${PROJECT}.iam.gserviceaccount.com

gcloud iam service-accounts add-iam-policy-binding "${GSA}" \
  --project="${PROJECT}" \
  --role="roles/iam.workloadIdentityUser" \
  --member="serviceAccount:${PROJECT}.svc.id.goog[${NS}/${RELEASE}-llm-gateway]"

GitOps / Crossplane: bind serviceAccount:${PROJECT}.svc.id.goog[<namespace>/<release>-llm-gateway], not <release>-platform.

A.5 LiteLLM model list — omit api_key; WI supplies ADC:

llmGateway:
  models:
    - model_name: company-claude-opus
      litellm_params:
        model: vertex_ai/claude-opus-4-8
        vertex_project: my-project
        vertex_location: us
      model_info:
        mode: chat
    - model_name: company-claude-haiku
      litellm_params:
        model: vertex_ai/claude-haiku-4-5
        vertex_project: my-project
        vertex_location: us
      model_info:
        mode: chat
    - model_name: company-embedding
      litellm_params:
        model: vertex_ai/text-embedding-004
        vertex_project: my-project
        vertex_location: us-central1
      model_info:
        mode: embedding

llmBootstrap:
  enabled: true
  models:
    chat:
      name: company-claude-opus
      maxTokens: 0
    bg:
      name: company-claude-haiku
      maxTokens: 0
    embedding:
      name: company-embedding
      maxTokens: 0
      dimension: 768    # text-embedding-004 — not 1536

Use a cheap model for bg (Haiku, Gemini Flash) — not a flagship chat model.

---

Pattern B — Service account JSON key (any Kubernetes)

Use when the cluster is not GKE, Workload Identity is unavailable, or your security team requires key material in a rotatable Secret rather than metadata-server ADC.

B.1 Create a GCP SA with roles/aiplatform.user (same as Pattern A.2).

B.2 Store the key in a BYO Secret. LiteLLM accepts the JSON inline via vertex_credentials — no file mount required when you use an env var from providersExistingSecret:

gcloud iam service-accounts keys create vertex-llm-key.json \
  --iam-account=platform-vertex-llm@my-project.iam.gserviceaccount.com

kubectl -n runwhen-platform create secret generic rw-vertex-llm \
  --from-file=VERTEX_CREDENTIALS_JSON=vertex-llm-key.json
rm vertex-llm-key.json   # do not leave keys on disk

B.3 Wire the Secret into the gateway and reference it from each Vertex model entry:

llmGateway:
  providersExistingSecret: rw-vertex-llm
  models:
    - model_name: company-claude-opus
      litellm_params:
        model: vertex_ai/claude-opus-4-8
        vertex_project: my-project
        vertex_location: us
        vertex_credentials: os.environ/VERTEX_CREDENTIALS_JSON
      model_info:
        mode: chat
    # … bg + embedding entries with the same vertex_credentials line

Keep serviceAccount.llmGateway.create: false unless you also use Workload Identity — the JSON key is scoped to the gateway pod via providersExistingSecret, not the Kubernetes ServiceAccount.

Alternative (file path): set GOOGLE_APPLICATION_CREDENTIALS in the same Secret to a path inside the pod and mount credentials.json there (CSI, Secret volume, or post-render). LiteLLM picks up ADC from that env var. Same model list as Pattern A.5; omit vertex_credentials when using ADC from a mounted file.

Operational note: JSON keys are long-lived credentials. Prefer Pattern A on GKE. Rotate keys on a schedule, update the Secret, and restart the llm-gateway pod.

---

External LiteLLM gateway

If you already run LiteLLM centrally, disable the in-chart gateway and give PAPI a pre-issued virtual key:

llmGateway:
  deploy: false
  external:
    url: https://llm.customer.example.com

llmBootstrap:
  enabled: true
  papiApiKeySecretRef:
    name: rw-external-llm-papi
    key: PAPI_LLM_API_KEY
  # models.{chat,bg,embedding}.name must match names published by YOUR proxy

Create the Secret before upgrade:

kubectl -n <platform-namespace> create secret generic rw-external-llm-papi \
  --from-literal=PAPI_LLM_API_KEY="sk-..."

---

Pass-through to a shared upstream

Some RunWhen lab environments proxy to a shared LiteLLM (https://llm-gateway.shared.runwhen.com) using a virtual key in Vault or a BYO Secret:

llmGateway:
  providersExistingSecret: rw-upstream-litellm
  models:
    - model_name: shared-claude-opus
      litellm_params:
        model: openai/shared-claude-opus
        api_base: https://llm-gateway.shared.runwhen.com
        api_key: os.environ/UPSTREAM_LITELLM_KEY

Production self-hosted customers typically call their own endpoints instead.

---

Verify after install

1. Gateway pod — kubectl get pods -l app.kubernetes.io/component=llm-gateway
2. Vertex WI (GKE only) — when using Pattern A (serviceAccount.llmGateway.create: true), confirm the gateway pod uses <release>-llm-gateway, not <release>-platform:

   kubectl -n <namespace> get pod -l app.kubernetes.io/component=llm-gateway \
     -o jsonpath='{.items[0].spec.serviceAccountName}{"\n"}'

3. Model list — port-forward the gateway Service to :4000, GET /v1/models (authorize with LITELLM_MASTER_KEY from <release>-platform-secrets)
4. PAPI catalog — Staff UI → LLM settings, or confirm bootstrap Job completed: kubectl logs -l app.kubernetes.io/component=llm-bootstrap
5. Smoke chat — Open workspace chat; confirm AgentFarm routes to your chat model name
6. Embeddings — After changing embedding model or dimension, plan a workspace re-index (USearch graph)

The chart’s install-checklist.sh verify script includes LLM checks when bootstrap is enabled.

---

Troubleshooting

Symptom	Likely cause
AgentFarm 404 on chat	llmBootstrap.models.*.name ≠ llmGateway.models[].model_name
Gateway 401 to vendor	Missing/wrong key in providersExistingSecret
Vertex 403	Model not enabled in Model Garden; wrong auth pattern (WI bound to <release>-platform, expired JSON key, or missing aiplatform.user); or GOOGLE_APPLICATION_CREDENTIALS unset for Pattern B
Search quality collapse after upgrade	Embedding dimension changed without re-index
Bootstrap Job failed	PAPI not ready, or staffUser.email / provider name empty

---

Related documentation

Topic	Link
BYO LLM overview & provider matrix	Bring Your Own LLM
Self-hosted prerequisites	Self-Hosted Deployment Requirements
Azure step-by-step	Azure OpenAI BYO LLM Setup
OpenShift SCC (platform SA; separate from llm-gateway WI)	OpenShift & OKD