Safe-for-Prod Harness: The Agent-Runner Pattern
Separate the agentic loop from target-system access, and blast radius and load become structural properties instead of policy.
How do you let an agent touch production?
It is very tempting to give your agent on Claude Code a kubeconfig (or cloud creds) and let it read logs, query the API server, run diagnostics, and quickly get answers to your questions. The reasoning and tool-calling abilities of even basic agents get engineers to a very satisfying demo very quickly.
But how do you manage a large team of engineers all accessing the production stack to run their agentic loops for daily operations?
It opens up a huge safety risk. There are 2 critical questions to answer:
- What can the agent reach?
- What is the access path?
Limitations of Agents accessing Production
Direct access works in a demo. In production it has four specific limitations:
- Credentials concentrate in the agent. It holds keys to everything it might ever touch, which makes it the highest-value target in the estate.
- Access is decided at runtime. What the agent reaches depends on what the model generates in the moment. A prompt injection or a hallucinated flag changes the answer.
- Load is unbounded. Nothing sits between N parallel investigations and your API server.
- Network boundaries get bypassed. One component with reach into prod, corporate, dev, and every cloud account collapses separations your security team built on purpose.
None of these are model-quality problems. Better prompts don’t fix them. They’re harness problems with a single root cause: the probabilistic component and the credentialed component are the same thing. The rest of this post is one pattern that removes that root cause, and what it buys once it’s gone.
The Agent-Runner pattern: separate the planes
Infrastructure engineering has solved this class of problem before. Networking split the control plane from the data plane. Kubernetes split the control plane from the nodes. The Agent-Runner pattern makes the same cut for agents:
- Reasoning Plane [the agentic loop]. It observes, picks the next task, reads the results, iterates. Prompts, models, assistants. Probabilistic by design, and it changes constantly.
- Execution Plane [the Runner]. Credentials, network paths, execution, concurrency. Deterministic, security-critical, and it should change rarely.
The two planes have different rates of change and different failure semantics. That’s exactly why they shouldn’t live in one process. Fused, every prompt tweak is a change to a credential-holding system, and every model upgrade is a security event. Separated, you can swap models and iterate on prompts freely without touching anything that holds a secret.
Only two things cross between the planes. A pre-approved task request goes one way. Structured findings come back.
How it works in RunWhen
The Runner installs as a Helm chart into your own Kubernetes cluster. It runs three kinds of pods:
- a workspace builder, which discovers your resources using a read-only service account and matches them against approved task templates;
- a task runner, which receives execution requests and holds them in a local queue;
- workers, which drain the queue and execute individual tasks.
The request path is the part to notice. The reasoning plane doesn’t generate arbitrary code against your systems. Its request resolves to a task configuration that was matched and approved when the workspace was built. Only that pre-approved task lands in the Runner’s queue.
All communication is outbound-only. The Runner polls the platform over HTTPS on port 443, and no inbound ports open on your infrastructure. Self-hosted deployments use mutual TLS, and air-gapped operation keeps the same shape.
One Runner per network boundary
Enterprises don’t have one production network. Production spans cloud accounts that deliberately can’t route to each other. An on-prem data center sits behind its own firewalls. These boundaries are security engineering, encoded as routing tables.
The pattern honors them by deployment shape: one Runner per boundary. One per cloud account, one per security zone, one per data center. Each Runner:
- holds only its own boundary’s credentials, issued by that boundary’s own identity mechanism (workload identity federation where supported), not pooled static keys;
- keeps secrets local to its cluster: secret values never pass through the platform;
- has no route to any other boundary. The Runner in your AWS account can’t query the on-prem data center. Not because a policy forbids it, but because no path and no credentials for it exist inside that Runner.
That last property is the point of the pattern. RBAC and guardrails are policy, and policy can be misconfigured. Topology can’t. An agent’s failure domain is capped at the boundary its Runner lives in, exactly like the humans and CI systems in that zone.
Multiple Runners don’t fragment your view. They all report outbound to one platform, and a workspace aggregates them into a single operational view across clouds and data centers. (Deployment options)
Protecting your systems from agent load
Agents change the load profile of operations. Several engineers can now run multiple agents for operations and investigations. One incident can trigger several parallel investigations, each calling the Kubernetes API server, the cloud APIs, and the observability stack. It is likely that this happens while the system is already degraded and the team is trying to troubleshoot.
With the Agent-Runner pattern, the Runner’s queue gives admins a way to control this. Workers drain the queue at bounded concurrency. Parallel investigations queue instead of stampeding, and back pressure lands on the reasoning plane instead of on your production systems; investigations just take a little longer.
The takeaway
The safety conversation for production agents usually centers on the reasoning plane: better prompts, more guardrails, tighter approvals. Those are policy. The Agent-Runner pattern moves the two properties that matter most into structure. Access is scoped per boundary, because that’s where the credentials physically live. Load is bounded per boundary, because that’s where the queue physically drains.
Ask your current tooling the two questions from the top. What can it reach? And how hard can it hit what it reaches?
Related docs