Bring Your Own LLM
RunWhen does not ship production LLM models. You point the platform at endpoints you control (Azure, OpenAI, Vertex AI, Bedrock, on-prem vLLM, or your own LiteLLM proxy). The chart runs an in-cluster LiteLLM gateway that routes chat, background, and embedding traffic to those endpoints.
Enterprise configuration data is scrubbed before it leaves your cluster. Community task source code may be sent to LLMs during indexing — see the Data Security Framework.
What you need (three models)
Every production install registers three logical models in PAPI:
| Role | Typical use | Examples |
|---|---|---|
| Chat | Agentic workspace chat, issue analysis | Claude Opus, GPT-4.1, Gemini Pro |
Background (bg) | Bulk summarization, indexing helpers | Claude Haiku, GPT-4o-mini, Gemini Flash |
| Embedding | Semantic search (USearch) | text-embedding-3-small (1536-dim), Vertex text-embedding-004 (768-dim) |
The model_name strings in your Helm llmGateway.models[] must exactly match llmBootstrap.models.{chat,bg,embedding}.name. A mismatch surfaces as 404s from AgentFarm at runtime, not at deploy time.
Pick an embedding model once and keep its dimension stable — changing dimension requires re-indexing workspace search data.
Supported provider patterns
RunWhen validates against LiteLLM’s provider surface. Common patterns:
| Provider | Auth | Notes |
|---|---|---|
| Azure OpenAI / AI Foundry | Service Principal (recommended) or API key | Strong enterprise default; partner models (e.g. Claude) via Foundry. See Azure BYO LLM guide. |
| OpenAI | API key | Direct or via your own LiteLLM proxy |
| Google Vertex AI (Model Garden) | WI on GKE (recommended), JSON key via BYO Secret, or external proxy | Partner MaaS models (Claude, embeddings). Auth patterns: Self-Hosted LLM Configuration. |
| Amazon Bedrock | IAM / keys | Roadmap validation varies by model |
| On-prem vLLM / Ollama | API key or none (internal network) | OpenAI-compatible api_base |
If Azure is organizationally difficult, RunWhen can provision a single-tenant Azure endpoint for your team — contact support@runwhen.com.
Where configuration lives
| Deployment | Configuration |
|---|---|
Self-hosted Helm (runwhen-platform chart) | Values overlay: llmGateway + llmBootstrap. Start here: Self-Hosted LLM Configuration. |
| RunWhen-hosted SaaS | RunWhen operates the upstream gateway; your contract covers model access. |
| External LiteLLM only | Set llmGateway.deploy: false and point PAPI at your existing proxy — details on the self-hosted page. |
Azure IAM quick reference
For Microsoft Entra ID (keyless) access, assign roles at the resource scope:
| Endpoint type | Role | Scope |
|---|---|---|
| Azure OpenAI | Cognitive Services OpenAI User | OpenAI resource |
| Azure AI Foundry | Cognitive Services User | AI Services resource |
Full walkthrough: Azure OpenAI & AI Foundry — BYO LLM Setup.
Related documentation
| Topic | Link |
|---|---|
Self-hosted Helm values (llmGateway, llmBootstrap) | Self-Hosted LLM Configuration |
| Self-hosted prerequisites (registry, DNS, TLS) | Self-Hosted Deployment Requirements |
| Azure OpenAI worked example | Azure OpenAI BYO LLM Setup |
| Embedding dimension / re-index | Chart doc: byo-embedding-model.md in the onboarding pack |
| Security model | Security Model Overview |
Support
For provider selection, IAM setup, network allowlisting, and end-to-end validation: support@runwhen.com