Most teams can tell you whether their AI agent ran. Far fewer can tell you whether it did the right work. A green log line says an email was drafted. It does not say the price was current, the right customer received it, or the deal moved forward. As agents take on more routine work, that gap becomes the main risk.
This guide explains AI agent evaluation for founders and operators: what an eval is, why agents are harder to test than a chatbot, what a useful eval checks, how to build a first suite in four weeks, and which numbers to watch once agents are live.
Key takeaways
- An agent eval tests finished work in a realistic setting, judged on the result it leaves behind.
- Check three things: the end state, the path taken and the business outcome the task was meant to serve.
- Run each task several times, because an agent that succeeds once may still fail often.
- Start with 20 to 50 tasks drawn from real failures, and read the transcripts yourself.
What is AI agent evaluation?
AI agent evaluation is the practice of giving an agent a set of defined tasks, letting it work in an environment that behaves like production, and scoring the result against clear success criteria. It answers one question: when this agent is handed work like ours, how often does it finish it correctly?
Anthropic's engineering team published a useful vocabulary for this in Demystifying evals for AI agents (January 2026). The terms are worth adopting, because they stop teams talking past each other:
- Task. In Anthropic's words, "a single test with defined inputs and success criteria." For a sales agent, one task might be "draft a quote for this inquiry."
- Trial. "Each attempt at a task is a trial." Because agents do not behave the same way every time, one task usually gets several trials.
- Transcript. The complete record of a trial: every message, tool call and result. Some teams call it a trace or trajectory.
- Outcome. "The final state in the environment at the end of the trial." Did the draft exist, with the right numbers, addressed to the right person?
- Grader. "Logic that scores some aspect of the agent's performance." A grader can be a script, a model with a rubric or a person.
A collection of tasks with their graders is an eval suite. "Our evals" usually means that suite plus the habit of running it whenever the prompt, tools or model change.
Why are AI agents harder to evaluate than chatbots?
A chatbot produces text, and you can judge text by reading it. An agent takes actions across several steps, and each step changes what it sees next. Three things make that harder to test.
The final message can lie. An agent may report "quote sent" when the email is still in drafts, or when it went to the wrong contact. Grading the agent's own summary misses this. You have to check the world it left behind.
Success is inconsistent. The same agent, given the same task, can succeed on Monday and fail on Tuesday. The tau-bench study by Shunyu Yao and colleagues made this concrete. It tested tool-using agents on tasks that involve talking with a simulated user and following domain-specific rules. In the June 2024 paper, even state-of-the-art function-calling agents such as gpt-4o succeeded on fewer than half the tasks, and the chance of an agent getting the same retail task right in all eight tries was under 25%. Models have improved a great deal since then, but the lesson holds: one success tells you little about reliability.
The rules live outside the agent. Real work has constraints the agent was never told: a discount limit, a credit policy, a customer who must not be contacted this week. This is one of the main reasons covered in why AI agents fail in production.
Anthropic's guidance gives two measures for consistency. Pass@k "measures the likelihood that an agent gets at least one correct solution in k attempts." Pass^k "measures the probability that all k trials succeed." For work that faces customers, pass^k is the number that matters.
Model benchmarks vs agent evals vs business outcomes
Three kinds of testing often get mixed up. Each stops at a different point.
| Kind of test | What it asks | Who builds it | Where it stops |
|---|---|---|---|
| Public model benchmark | How capable is this model on a standard set of problems? | Labs and researchers | At general ability, with no knowledge of your company |
| Agent capability eval | What can this agent do well on tasks like ours? | Your team | At what the agent could do, on a good day |
| Agent regression eval | Does the agent still handle everything it used to? | Your team | At protecting work that already passes |
| Outcome check in production | Did the business result the task served actually happen? | Your team, with live data | At the verified result: the quote sent, the meeting booked |
Anthropic draws the line clearly: capability evals ask "What can this agent do well?", while regression evals ask "Does the agent still handle all the tasks it used to?" Capability evals tell you where to give the agent more work. Regression evals tell you last week's prompt change did not quietly break something.
The last row is the one most teams skip. A suite can pass every sandbox test while the live agent drafts emails nobody sends. Production outcome checks close that gap and connect evaluation to enterprise AI observability.
What should an AI agent eval actually check?
A useful eval checks work at four levels.
- Output. Is the result well formed? The email has a subject, the table adds up, the file opens. Code checks handle this cheaply.
- Path. Did the agent use the right sources and tools? An agent that reads last year's price list and lands on the right number has passed by luck.
- End state. Is the world in the right condition? The right price, the right payment terms, the right recipient, nothing sent that should not have been.
- Outcome. Did the work serve its purpose? In a sandbox you approximate this with a rubric. In production you check it against proof, such as the email that went out.
Graders come in three kinds, and Anthropic's guide recommends combining them. Code-based graders, such as "string match checks (exact, regex, fuzzy, etc.)," are fast and objective, so use them for any fact: a price, a date, a field. Model-based graders use a rubric for things a script cannot judge, such as tone. Human graders, such as a subject expert, are slow, and they are the only way to calibrate the other two.
A simple rule: if a mistake would cost money or a customer, a code check should catch it.
Common mistakes in AI agent evaluation
- Grading the agent's summary. The agent says it finished. Check the end state yourself, or the eval measures how confident the agent sounds.
- One trial per task. A single pass hides how often the agent fails. Run each task several times and report pass^k for anything that faces a customer.
- Invented test cases. Tasks written at a desk are cleaner than real work. Anthropic's advice is direct: "20-50 simple tasks drawn from real failures is a great start."
- Never reading transcripts. Scores alone hide broken graders. As Anthropic puts it, "You won't know if your graders are working well unless you read the transcripts and grades from many trials."
- Unfair tasks. If the agent could not possibly know a fact, failing it teaches nothing. Anthropic's test is that failures "should seem fair: it's clear what the agent got wrong and why."
Practitioners agree. Hamel Husain wrote in Your AI Product Needs Evals that "unsuccessful products almost always share a common root cause: a failure to create robust evaluation systems." His other rule is simpler: "You are doing it wrong if you aren't looking at lots of data."
How to build your first AI agent eval suite in four weeks
Anthropic's own account of building Claude Code is a useful model. The team says Claude Code "started with fast iteration based on feedback from Anthropic employees and external users," and added evals later, "first for narrow areas like concision and file edits, and then for more complex behaviors like over-engineering." Start narrow and grow from what breaks.
- Week 1: pick one agent and one job.
Choose one frequent task, such as drafting quote replies. Write one sentence that defines success, including the business result it serves.
- Week 1: collect real cases.
Pull 20 to 50 past examples from email, tickets or chat, with priority on cases where a person had to fix the work. Each becomes a task with its inputs and the correct end state.
- Week 2: write the graders.
Add code checks for every fact that can be checked: prices, dates, recipients, required fields. Add a short rubric for judgement items. Agree which failures are critical.
- Week 2: build a safe environment.
Run the agent against copies of the data, with outbound actions captured and never sent. Anthropic's earlier guide on building effective agents recommends "extensive testing in sandboxed environments, along with the appropriate guardrails."
- Week 3: run, read, fix.
Run every task at least five times. Read twenty transcripts by hand and fix the agent or the grader, whichever was wrong.
- Week 4: wire it to change.
Run the suite whenever the prompt, tools or model change. Every new production failure becomes a new task, so the suite grows from real work.
Before launch, decide what the agent may do without asking. The setting approval rules playbook covers that page, and human in the loop vs human on the loop explains how the approval design should change as the eval record improves.
How to measure AI agent quality over time
Once an agent is live, a handful of numbers tell you whether it is doing the right work. Track them per agent and per task type.
| Measure | What it tells you | Watch for |
|---|---|---|
| Pass^k on the regression suite | How reliably the agent handles known work | Any drop after a prompt, tool or model change |
| Critical failure rate | How often a mistake would have cost money or trust | Should be near zero before you remove an approval step |
| Approval edit rate | How often a person changes the draft before approving | A falling rate means the agent is learning your standards |
| Outcomes verified on proof | Share of tasks where the business result can be shown | The truest measure of right work |
| Rejected as not relevant | How often the work should never have been started | A sign the agent lacks company context |
The last row matters more than it looks. An agent can be accurate and still busy with the wrong work. Evals answer "did it do the task well?" Company context answers "should this task exist?" When the two are measured together, you can raise an agent's authority on evidence, which is the idea behind AI agent accountability and AgentOps.
Investors now describe this discipline as a business in itself. In its Winter 2027 Requests for Startups, Y Combinator published a request by Jon Xu titled Own Your Intelligence. It argues that between an open model and intelligence that improves a business, "Someone has to turn workflows into evals, expertise into training data, production failures into improvements, and carry all of it forward with each new base model."
How GeniOS approaches AI agent evaluation
GeniOS treats the right task as the first thing to check. It reads where a company's work happens, such as email, calendar, chat and the business tools already in use, read-only to start. It forms the company's goals from its own emails, decks and meetings, and checks every team's pending work against them, so work that no longer matters is caught before anyone, human or agent, spends time on it.
What reaches a person is one situation, what to do, why, one owner and a date. When the owner chooses Delegate, the work goes to a person or an AI agent, and nothing is sent until a human approves. That approval is a live review of the agent's work on a real task.
An item closes only on proof, such as the email that went out. Each approval, edit and outcome is recorded, and GeniOS learns from it. Over time that record shows which kinds of work an agent handles reliably, which is the evidence a team needs before it trusts an agent with more.
Worked example: testing a quote agent at a safety-shoe maker
The names, dates and numbers in this example are made up to explain the idea. Vaultex makes safety footwear in Greater Noida. Bata has asked for 12,000 pairs of S3 safety shoes at $12 a pair, about $144,000, with a vendor review on Friday. Reebok samples are running on Line 2, and Adidas has a vendor audit coming that needs IS 15298 test reports. Standard payment terms are 60 days.
Harsh in sales wants an agent to draft quote replies. Before it touches a real buyer, the team turns 30 past quote threads, including the ones Harsh had to rewrite, into tasks.
- Code graders: unit price reflects current material cost, payment terms read 60 days, quantity matches the inquiry, the recipient is the buyer's named contact.
- Model grader: a rubric for tone and whether the reply answers every question the buyer asked.
- Human grader: Harsh reviews ten transcripts a week.
The first run looks good: 27 of 30 tasks pass on a single trial. Five trials per task tell a different story. On the Bata-style tasks, the agent quotes the old $12 price in two of five trials, even though steel toe caps went up 8% this week. The transcripts show why: the agent sometimes reads last month's price sheet instead of the supplier's new email. At a made-up toe-cap cost of $1.50 a pair, the rise is $0.12 a pair, or $1,440 across the Bata order, all of it coming out of margin.
The team fixes the source the agent reads first, adds a code check on the latest cost, and keeps the failure as a regression task. A second gap appears: one draft promises a delivery date on Line 2, which is booked for Reebok samples. Anushka in production owns that slot, so the eval now flags any quote that names a date without her confirmation.
After two weeks, pass^5 on the regression suite is steady and Harsh's edit rate is falling. The agent still drafts and Harsh still approves, but the team now knows, with evidence, where the agent can be trusted.
Conclusion
AI agent evaluation is how a company learns, with evidence, whether its agents do the right work. Test the end state, the path and the outcome. Run each task more than once, build from real failures and read the transcripts. Then connect it to production, where the only result that counts is the one you can prove, and raise an agent's authority as the record earns it. To see how GeniOS checks work against your goals and closes it on proof, see pricing or book a call.
Sources
- Anthropic, Demystifying evals for AI agents, 9 January 2026
- Anthropic, Building effective agents, 19 December 2024
- arXiv, Yao, Shinn, Razavi and Narasimhan, tau-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains, 17 June 2024
- Hamel Husain, Your AI Product Needs Evals, 29 March 2024
- Y Combinator, Requests for Startups: Own Your Intelligence (Jon Xu), Winter 2027