AI agent evals - the diligence question executives skip
A demo proves an agent can succeed once, on a path chosen to impress you. Everything after launch is the paths nobody rehearsed. Here is what to ask before you sign off.
Short answer: An AI agent evaluation framework is the set of tests, metrics, and review checkpoints that prove an agent performs reliably across real tasks, beyond the one scripted path a demo shows. The question to ask before funding an agent: what was it tested against, and who signed off on the result.
We build and run agents in production, and we have watched clean demos fall apart on the second real request.
Why do AI agent demos fail once they reach production?
A demo shows the agent clearing one path a person picked in advance. Production hands it hundreds of paths at once, and the small error rate you never saw in the demo compounds at every step.
The math is unforgiving. Chip Huyen has laid it out: an agent that is 95% accurate on a single step succeeds only about 60% of the time across ten steps, and about 0.6% of the time across a hundred. A real task is rarely one step. So a demo that nails a five-step flow tells you almost nothing about the same agent running a forty-step process on live data.
The people building these systems know it. LangChain's State of AI Agents survey found quality cited as the top barrier to production by close to a third of teams, even as a majority reported they already run agents in the wild. The agents are shipping. Confidence that they hold up is the part that lags.
A founder once sent us a demo video and a term sheet in the same email. The agent cleared a nasty support ticket without a stumble, and he was ready to hire four people around it. We asked what else it had been tested on. The thread went quiet for a week. "It worked when they showed me" is one sample, chosen by the person who wants your money.
What is an AI agent evaluation framework?
An AI agent evaluation framework is a repeatable way to grade the agent on real tasks instead of trusting a single run. Anthropic's engineering team describes an eval as a structured task the agent is scored on, which turns a vague sense of "quality" into a signal a team can actually track.
You do not need thousands of tests to start. Anthropic puts the opening number at 20 to 50 tasks drawn from the cases where the agent actually broke. From there the framework has four working parts:
- Automated evals before launch, run on every change so nobody ships on a hunch.
- Production monitoring to catch drift once real traffic hits.
- A/B testing when someone swaps the prompt or the model.
- Human review, kept for calibrating the automated graders rather than checking every output by hand.
None of this is exotic. OpenAI's open-source evals registry is one of the references enterprises point to, and the practice is well documented. The gap is almost always organizational: whether anyone insisted on evals before the demo got funded.
How is evaluating an agent different from testing a chatbot?
Testing a chatbot checks one answer. Evaluating an agent checks a sequence of decisions. A 2025 survey of LLM-agent evaluation draws the line plainly: agents act in dynamic, interactive, multi-step environments, so a method built to score a single static response does not carry over.
For a funder, the tell is simple. If the person pitching you talks about accuracy the way you would grade a search result, they are measuring the wrong thing. Agent evaluation metrics have to cover the whole trajectory: did it pick the right tool, recover from a bad step, and stop when it should have. Grade only the final answer and you miss the three places an agent quietly goes wrong on its way there.
What should executives ask before funding an agent?
Four questions separate a rehearsed demo from a system that will survive real traffic. You do not need to read the eval code. You need straight answers, and a name attached to each one.
| Question | What a good answer sounds like |
|---|---|
| What was it tested against? | 20 to 50 real tasks pulled from cases the agent has actually failed on |
| What is the pass rate, and on which tasks? | A number tied to specific hard cases, with the failures named out loud |
| How will you know if it drifts after launch? | Production monitoring is already live at rollout, with alerts a person watches |
| Who signed off, and on what? | A named owner who is accountable if it breaks in front of a customer |
If the answers go vague on any row, what you are funding is the demo.
Want your own AI drafts to stop reading like AI? The skill we use for it is free - grab the human-writing skill.
Whose job is it to own the answer?
Eval design is a governance decision, and it belongs to the person signing the check. The engineer builds the tests. The funder decides what "good enough to ship" means, because that threshold is a risk call the business has to make. Hand it to the vendor and you have let the seller grade their own work. When an agent gets something wrong in front of a customer, someone has to own that mistake, and the eval threshold is where that ownership is set.
Evaluation is what moves an agent from slide to system, and it is also where you can measure the return on the whole investment.
The market has read the same tea leaves. Braintrust, an eval and observability platform, raised an $80M Series B at an $800M valuation in February 2026, with Notion, Replit, Cloudflare, and Ramp among its customers. A month later, OpenAI acquired Promptfoo, an eval and red-team tool it says is used by more than a quarter of the Fortune 500. Capital is flowing toward the companies that can answer "did it work."
One thing you cannot outsource to a standard is the judgment itself. The OpenTelemetry GenAI conventions now standardize how agent and tool calls get traced, but as of 2026 they remain in development and deliberately leave out output quality, correctness, and safety. The industry has agreed on how to record what an agent did. Deciding whether that was any good remains a human judgment, and it stays with you.
Frequently asked questions
What is an AI agent evaluation framework? It is a repeatable set of graded tasks, metrics, and review checkpoints that measure how an agent performs on real work, beyond a single demo. Anthropic frames each eval as a structured task the agent is scored on, which converts a vague sense of quality into a signal a team can track over time.
How is evaluating an AI agent different from evaluating an LLM or chatbot? A chatbot gives one answer you can grade on its own. An agent takes a sequence of steps in a live environment. A 2025 survey of agent evaluation notes agents work in dynamic, multi-step settings, so single-response scoring misses whether the agent chose the right tools and recovered when a step went wrong.
How many test cases does an agent eval actually need? Fewer than most people expect. Anthropic suggests starting with 20 to 50 tasks drawn from real failures rather than a large synthetic set. The point is coverage of the ways the agent actually breaks. You grow the suite as production surfaces new failure modes.
Should we buy an eval platform or build one in-house? Either works, and the platform is not the real decision. OpenAI's open-source evals and commercial tools like Braintrust or Promptfoo all do the job. Real tasks have to get graded, and someone has to own the pass bar. A bought platform with no owner still ships an untested agent.
Who inside the company should own agent evaluation? The engineer builds the tests; the person funding the agent owns the threshold for shipping it, because that threshold is a risk decision. Evaluation and governance belong in the same hands, since the person accountable for the risk is the one who should set the pass bar.
Before the money goes out
You decide whether an eval suite exists before you fund the agent, and you pick the person who has to stand behind the result. Ask what it was tested against. Ask who signed off. If nobody can answer, what you saw was a demo, and a demo only has to work once.
The free human-writing skill. We packaged the method we use to make AI drafts read like a person into a Claude skill, and we give it away. Leave your email and we will send the link - get the human-writing skill. Prefer to just follow along? Subscribe to the newsletter and we will send the next piece when it is out.
Sources
- Chip Huyen, "Agents" (January 2025). huyenchip.com
- LangChain, "State of AI Agents" (2026). langchain.com
- Anthropic, "Demystifying evals for AI agents," Engineering blog (January 2026). anthropic.com
- OpenAI, "evals" open-source repository. github.com
- Mohammadi et al., "Evaluation and Benchmarking of LLM Agents: A Survey," KDD '25 / arXiv (2025). arxiv.org
- Braintrust, "Announcing our Series B," company blog (February 2026). braintrust.dev
- OpenAI, "OpenAI to acquire Promptfoo" (March 2026). openai.com
- OpenTelemetry, "GenAI semantic conventions." opentelemetry.io
By Christopher Kliebenstein. We build and run AI-native agents and workflows for operators who want working systems over demos.