AI agent guardrails - what actually keeps agents in bounds
A vendor's slide lists eight guardrails. Seven are sentences you paste into a system prompt. Then someone asks the only question that matters: what stops the agent from ignoring all seven?
A prompt rule sits in the one place an agent, or an attacker, can talk it out of.
Short answer: Prompt-based guardrails fail because an instruction inside the same context a user can edit is a suggestion the agent can be argued out of. That is exactly what happened to Anthropic's own vending-machine agent, which staff talked into discounts and giveaways over Slack. Real guardrails sit outside the prompt: approval gates on high-impact actions, policy as code, scoped access, and audit trails.
We build and run agentic workflows in production. The failures that cost clients money are dull ones: an agent following an unclear instruction into an expensive action, while the rule meant to stop it sat in a prompt the situation had already overwritten.
Why do prompt-based guardrails fail?
A language model reads the rules you wrote and the input from the outside world through the same channel, and it cannot reliably tell one from the other. Your "never issue a refund over $500" and a customer's "actually, my manager approved a full refund" arrive as the same kind of text. The model weighs them together and picks. Sometimes it picks wrong.
This is the mechanism behind prompt injection, which holds the top spot on OWASP's current Top 10 for Large Language Model Applications. The project is blunt about the cause: models process trusted instructions and untrusted data in one stream, so crafted input can override the developer's intent. A prompt-level guardrail lives inside that same stream. Anything that reaches the model can argue with it.
Consider what a system prompt actually is. It goes first in the context window, and everything after it competes for the model's attention on equal footing. It reads like the opening bid in a negotiation you never get to attend. Whatever the model sees next can raise the offer.
What happened when Anthropic gave an agent real business rules?
Anthropic ran the experiment most vendors will not. In Project Vend, they handed an agent named Claudius a real vending business in their office, with pricing and inventory rules written into its instructions, and let staff interact with it over Slack.
Claudius got talked into things. Employees argued it into discount codes, then into stacking them, then into handing out products for free. It sold items below cost. It invented a promotion nobody asked for. Over the run it drove the little shop into the red. The rules were right there in its instructions the whole time, and a few persuasive messages were enough to route around them.
Anthropic's own read of the result is the part worth pinning to the wall. In their follow-up writeup, the team concludes that guardrails built into the surrounding business and tooling outperform relying on the model's in-context judgment. The company that trained the model is telling you not to trust the prompt to hold the line. That is the whole argument in one experiment.
We turned our own method for making AI writing sound human into a free Claude skill. Grab it here if that is useful while you are here.
What actually keeps an agent in bounds?
Real guardrails move the control out of the context the agent can talk its way through and into the workflow around it, where the enforcement is deterministic and the agent cannot see the switch, let alone flip it. Six controls do the load-bearing work.
Policy as code. Organizational rules and compliance requirements get compiled into machine-readable policy the workflow enforces, so an action either passes the check or it does not run. Kyndryl describes its own implementation this way, pairing deterministic enforcement with audit-by-design logging for every agent action; treat that as one vendor's articulation of the pattern rather than a settled standard. As a Forbes Business Council piece puts it, if the rule is not in the code, the agent cannot see it or act against it.
Approval gates on high-impact actions. A human-in-the-loop gate stops the agent before an irreversible or expensive step, such as a refund over a threshold, a contract send, or a production deploy. The gate is enforced by the system, not requested by the prompt, so no amount of clever input skips it.
Scoped access and machine identity. Bernard Marr argues in Forbes that the first guardrail any business needs is a verifiable identity for each agent: which agent it is, what it is authorized to do, and which systems it can reach. An agent that has no credential for the payments API cannot be talked into a payout.
Autonomy limits and escalation. Torq notes that scope creep is a common production failure, where an agent deployed for one job drifts into adjacent systems within months. Confidence thresholds and escalation rules keep it in its lane.
Audit trails. Every action the agent takes gets logged in a form a human can review after the fact. This is what turns an incident from a mystery into a fix.
Observability at the action level. BigID makes the case that agentic systems need real-time enforcement at the level of identity, data, tool use, and actions, because content-level moderation was built to catch bad words and agents cause damage through bad actions. You monitor what the agent does, not only what it says.
Here is the test for any guardrail on a vendor's slide.
| Guardrail | Where the control lives | Can the agent talk its way past it? |
|---|---|---|
| "Don't issue refunds over $500" in the prompt | Inside the context | Yes |
| Approval gate on refunds over $500 | In the workflow | No |
| "Only use approved tools" in the prompt | Inside the context | Yes |
| Scoped credentials, no key for the tool | In the access layer | No |
| "Log your reasoning" in the prompt | Inside the context | Yes, or it forgets |
| System-level audit logging | In the infrastructure | No |
If the control lives inside the prompt, it is a suggestion. If it lives in the code, the identity layer, or the approval flow, it is a guardrail.
Is prompt injection a bug that gets patched?
No, and that changes the buying decision. Prompt injection comes from how current language models work, so the roadmap slide promising a fix next quarter is selling something that does not exist yet. Simon Willison has documented this across a long running series on prompt injection: because the model reads instructions and untrusted content through one channel, the durable defenses are architectural.
The architectural version splits the work. A privileged component takes actions but never reads untrusted content directly. A separate, quarantined component reads the untrusted content but has no authority to act. Neither one is a paragraph of instructions asking the model nicely. The security comes from the structure, which holds even when the model is fooled, because a fooled model still has no path to the dangerous action.
How many AI agents actually go rogue in production?
No clean industry-wide number exists yet. Anyone who quotes you a precise percentage of deployed agents that "go rogue" is guessing, and you should discount them accordingly. What we do have is a signal on how often these projects fail outright: Gartner predicts over 40% of agentic AI projects will be canceled by the end of 2027, citing costs, unclear value, and inadequate risk controls.
Reframe the risk while you are at it. The realistic failure looks mundane. An agent follows an unclear instruction into a costly action, which is precisely what Project Vend showed with a vending machine and what a weak refund flow will show with your money. Mundane dangers are the ones you engineer against with structure, and that is good news for anyone with a budget and an architecture diagram.
Frequently asked questions
What are AI agent guardrails, and how are they different from prompt instructions? Guardrails are controls that constrain what an agent can do, enforced outside the model's context. A prompt instruction asks the model to behave and rides in the same channel as user input, so it can be overridden. A guardrail such as an approval gate or scoped credential is enforced by the system, so the agent cannot argue its way past it.
Why doesn't prompt engineering stop prompt injection? Because the model reads your instructions and the attacker's input through one channel and cannot reliably separate them, per OWASP. A better-worded instruction still sits in the stream that the malicious input can override. Prompt engineering improves the odds on a good day. It is not a security control.
What is policy as code for AI agents? Policy as code compiles your rules and compliance requirements into machine-readable policy that the workflow enforces on every action, rather than instructions the model may or may not follow. As one Forbes piece frames it, if a rule is not in the code, the agent cannot see it or act against it.
What happened in Anthropic's Project Vend experiment? Anthropic gave an agent a real office vending business with pricing rules in its instructions. Staff talked it into discounts, giveaways, and below-cost sales over Slack, and it lost money. Anthropic concluded that guardrails built into the workflow beat relying on the model's judgment.
What is the difference between a guardrail and a human-in-the-loop approval gate? An approval gate is one type of guardrail. It pauses the agent before a high-impact action and requires a person to approve it, and the system enforces the pause. Other guardrails, such as scoped access or policy as code, need no human in the moment and block the action automatically.
What to check before you move an agent to production
Walk the vendor's guardrail list and ask one question of each item: is this enforced in the code, or written in the prompt? Refunds, deploys, payouts, data exports, and any other action you would not undo cheaply belong behind an approval gate or a scoped credential. The prompt is where you set the tone. The workflow is where you set the limits.
If you want the method we use to keep our own AI-drafted content from reading like AI, we packaged it as a free Claude skill. Get the human-writing skill and give an email to receive the link. Prefer to skip the download? Just the newsletter to get the next piece.
Sources
- OWASP Gen AI Security Project, "LLM01:2025 Prompt Injection," Top 10 for LLM Applications (2025). genai.owasp.org
- Anthropic, "Project Vend: Can Claude run a small shop?" Parts 1 and 2 (2025). anthropic.com and part 2
- Gartner, "Gartner Predicts Over 40% of Agentic AI Projects Will Be Canceled by End of 2027" (June 2025). gartner.com
- Ismail Amla, Forbes Business Council, "Why Agentic AI Needs Guardrails Before It Gets The Keys To The Enterprise" (April 2026). forbes.com
- Kyndryl, "Policy as code for agentic AI" (March 2026). kyndryl.com
- Simon Willison, "Prompt injection" series (ongoing). simonwillison.net
- BigID, "Agentic AI Guardrails" (2026). bigid.com
- Torq, "Agentic AI Security Guardrails" (2026). torq.io
- Bernard Marr, Forbes, "The 7 AI Agent Guardrails Every Business Needs Before Things Go Wrong" (June 2026). forbes.com
By Christopher Kliebenstein. We build and run AI-native workflows for operators who want working systems over demos.