How to Evaluate AI Agents: Test the Outcome, Not the Answer

Most teams "evaluate" their agent the same way: run it a few times, read the output, and decide it looks good.

That works for a demo. It does not work for a product.

Why agent evals are different from LLM evals

A classic LLM eval is a question and an expected answer. You send a prompt, compare the response, and score it.

An agent does not just answer. It works:

Task → plan → tool call → observe → tool call → … → final state

That changes what you need to test:

• The output is not the result. A support agent that says "Your refund has been processed" has produced text. Whether the refund exists in the database is the result.
• There are many valid paths. Two runs can use different tools in a different order and both be correct. Grading against one "golden" trajectory punishes good behaviour.
• Errors compound. A small mistake in step two can quietly ruin step nine.
• Runs are not deterministic. The same task can pass on Monday and fail on Tuesday with no code change.
• Side effects matter. An agent that completes the task but deletes a file it should not have touched has failed, even if the final answer is right.

So the question is not "is this answer good?" It is: "Did the customer get the outcome, safely, at an acceptable cost — and does that happen reliably?"

Step 1: Define "done" from the customer's side

Before writing a single test, write down what a successful outcome means for the person the agent serves.

Bad definition:

The agent handles refund requests well.

Useful definition:

Given an eligible order, the agent issues exactly one refund for the correct amount, updates the order status to refunded, and sends the customer a confirmation. Given an ineligible order, it issues no refund and explains why. It never refunds more than the order total.

The second version is testable. Each sentence becomes a check.

This is the same discipline as building the product itself: specific customer, specific problem, valuable outcome. If you cannot say what "done" looks like, no eval framework will save you.

Step 2: Collect real tasks, not imagined ones

Your first eval set should be small and real. Twenty to fifty tasks is enough to start. Anthropic's guide to agent evals^[1] makes the same point: 20–50 simple tasks drawn from real failures is a great start.

Good sources:

• Failures you have already seen. Every bug report, bad transcript, or "why did it do that?" moment is a free test case.
• Production or pilot logs. Sample what users actually ask, including the messy phrasing.
• Edge cases your definition of done implies. Ineligible orders, missing data, ambiguous instructions, a tool that times out.
• Things the agent must never do. Negative cases matter as much as positive ones: refunding twice, emailing the wrong customer, running a destructive command without approval.

For each task, record:

• the input (user message, starting files, seeded data);
• the starting environment;
• what success looks like, written as checks;
• what must not happen.

Avoid the trap of writing hundreds of synthetic tasks on day one. A small set you understand beats a large set nobody reads.

Step 3: Build a controlled environment

An agent eval needs a world the agent can act in and that you can reset.

• Sandbox the tools. Use a test database, a disposable workspace, a mock payment API, or a container that is destroyed after each run.
• Seed known state. Every trial should start from the same data so results are comparable.
• Isolate trials. One run must not leak files, memory, or records into the next. Anthropic notes^[1] that leftover files, cached data, and resource exhaustion shared between runs can cause correlated failures.
• Record everything. Save the full transcript — what OpenAI calls a trace^[3]: the end-to-end log of decisions, tool calls, and reasoning steps — plus tokens and timing. You will need it in Step 7.

If your agent can touch real systems during evaluation, you are not running an eval. You are running production with extra steps.

Step 4: Choose the right grader for each check

There are three kinds of grader. Use the cheapest one that can reliably answer the question.

• Code graders — best for end state, format, tests, and forbidden actions. Fast, cheap, deterministic, and objective, but they cannot judge tone, quality, or open-ended work, and they can be brittle when a valid answer looks different from the one you expected.
• AI judges with a rubric — best for writing quality, helpfulness, and policy compliance. They handle nuance and scale, but they can be wrong, inconsistent, or more expensive, and they need calibration.
• Human graders — best for calibrating the other two and for high-stakes or ambiguous cases. They are the standard for judgment, but slow and expensive.

Prefer code graders for the outcome. Query the database. Run the test suite. Check the file exists. Diff the config. If the outcome can be checked by code, check it with code.

Use an AI judge for what code cannot see. Was the explanation clear? Did it follow the tone policy? Give the judge a specific rubric, not "rate this 1–10." Ask it to check one criterion at a time and return pass/fail with a reason.

Calibrate the judge. Have a human grade a sample of transcripts, then compare. Both Anthropic^[1] and OpenAI^[4] recommend using human judgment to calibrate automated graders. If the judge disagrees with humans often, fix the rubric before you trust the score. An uncalibrated judge gives you a confident number that means nothing.

Step 5: Grade outcome, path, and cost

Score each trial on three layers.

Outcome — did it work?

This is the primary score. The refund exists. The tests pass. The report contains the right numbers. Grade the final state of the environment, not what the agent says it did. Anthropic's example^[1] is a flight-booking agent: it may say the flight is booked, but the outcome is whether a reservation exists in the database. The τ-bench benchmark^[6] grades agents the same way, by comparing the database state at the end of a conversation with the expected goal state.

Check the world, not the message:

• Deploy a website — request the public URL and confirm it returns HTTP 200 with the expected content.
• Fix a bug — run the test suite.
• Send an email — inspect the outbound mailbox state.
• Create a calendar event — query the calendar.
• Update a record — query the database.
• Open a pull request — confirm the PR exists with the expected diff.
• Research competitors — check that required sources and coverage are present.

For agents, state is usually more trustworthy than text.

Path — did it work acceptably?

Do not require one exact sequence of tool calls. Anthropic found^[1] that approach too rigid: agents regularly find valid approaches that eval designers did not anticipate. Instead, check constraints on the path:

• It asked for approval before a consequential action.
• It never called a forbidden tool.
• It did not read data outside the customer's account.
• It did not loop more than N times.

Constraints protect trust without punishing a creative, valid solution.

Path quality still matters when two runs both succeed. Imagine one run reads the file, edits it, and runs the tests. Another searches the web twice, installs an unneeded package, deletes and restores a file, then runs the tests three times. Both pass. The first is faster, cheaper, and far less likely to cause damage next time. Track signals such as failed tool calls, repeated calls, unnecessary actions, and whether the agent verified its own work.

Cost — was it worth it?

Track tokens, tool calls, turns, wall-clock time, and money per task. An agent that succeeds in 40 turns and $3 may be a worse product than one that succeeds 5% less often in 6 turns and $0.20. You can only make that trade-off if you measure it.

Step 6: Run each task more than once

Because agents are non-deterministic, one run tells you very little. Run each task several times — five is a reasonable start — and look at two numbers, both defined in Anthropic's guide^[1]:

• pass@k: the chance that at least one of k attempts succeeds. Useful when a human will pick the best attempt, or when the agent can retry.
• pass^k: the chance that all k attempts succeed. It was introduced by τ-bench^[6] as a measure of agent reliability, and it is closer to what a customer experiences. If your support agent passes a task 80% of the time, the chance it handles that task correctly five times in a row is about 33% (0.8⁵), assuming the runs are independent.

Customers do not experience your best run. They experience your typical run, repeatedly. For production agents, reliability matters more than peak capability.

Step 7: Read the transcripts

Scores tell you that something failed. Transcripts tell you why.

After every meaningful change, read a sample of failed and passed runs. Sort failures into categories:

• Context failure — the agent did not have the information it needed.
• Tool failure — a tool was missing, confusing, or returned an unhelpful error.
• Reasoning failure — it had what it needed and still chose badly.
• Instruction failure — the prompt or policy was ambiguous.
• Grader failure — the agent was actually right, and the test was wrong.

The last one is more common than people expect. As Anthropic puts it^[1], the transcript tells you whether the agent made a genuine mistake or your grader rejected a valid solution. A grader that expects "$10.00" while the agent writes "10 dollars", or a check that assumes one valid path, will report failures that are not real. If you never read transcripts, you will spend weeks "fixing" an agent that was already correct.

Transcript review is also where most real improvement comes from. Many agent failures are fixed not by a better model but by better context, clearer tool descriptions, or a more helpful error message.

Your tools are part of what you are testing

An agent eval does not measure the model alone. It measures the whole system: the model, the prompt, the tools and their descriptions, the context the agent receives, memory, the agent loop, recovery logic, and the environment.

That matters because small changes outside the model can move results. Anthropic reports^[2] that tool namespacing choices had non-trivial effects on its tool-use evaluations, and that small refinements to tool descriptions can yield dramatic improvements. Renaming a tool, rewriting its description, changing what it returns, or splitting one broad github() tool into search_repositories, create_repository, and create_pull_request can change how reliably the agent picks the right action.

So when a score drops, do not assume the answer is a bigger model. Your failure categories might show that a new model makes fewer reasoning errors but passes more wrong tool arguments. That points to fixing the tool schema, not switching models.

Step 8: Separate capability evals from regression evals

Keep two suites:

• Capability evals ask "what can the agent do?" They should be hard. A low pass rate is expected and gives you room to improve.
• Regression evals ask "did we break what already worked?" They should pass almost every time. Run them on every prompt, tool, or model change.

When a capability task becomes reliably solved, graduate it into the regression suite. Over time, the regression suite becomes a record of everything your agent has learned to do — and a guard against quietly losing it.

Wire the regression suite into CI. A prompt edit is a code change. Treat it like one.

When a capability suite reaches 100%, it has stopped telling you where the frontier is. Anthropic describes^[1] an eval at 100% as tracking regressions while providing no signal for improvement. Keep those tasks for regression and add harder ones.

Turn every production failure into an eval

This may be the most valuable habit of all.

When a user reports that the agent said a deployment succeeded while the site returned a 502 error, do not only patch the prompt. Add a task such as regression/false-deployment-success that recreates the situation and checks the real site status. When the agent retries the same failing command seventeen times, add regression/retry-loop with a limit on repeated calls.

OpenAI's evaluation guidance^[4] recommends the same habit from the other direction: log everything as you develop, mine the logs for good eval cases, and grow the eval set over time.

Over time, the eval suite becomes the team's memory. Each real failure makes the agent harder to break in the same way again.

Offline evals and production evals

You eventually need both. LangSmith's documentation^[5] draws the same line: offline evaluation for pre-deployment benchmarking and regression testing, online evaluation for production monitoring and anomaly detection.

• Offline evals run before a release, against a fixed task set, whenever you change the model, prompt, tools, memory, or agent design.
• Production (online) evals grade a sample of real runs after release. Automatic graders flag suspicious runs, a human reviews the failures, and confirmed failures become new offline tasks.

Together they form one loop: production → find failures → add eval tasks → improve the agent → run offline evals → release → production. That loop is how an agent becomes more reliable over time, rather than just more impressive in demos.

Compare agent systems, not models

"Is Model A better than Model B for agents?" is an incomplete question. The useful one is: which complete agent system performs best on my real tasks, at acceptable reliability, speed, and cost?

Run the same eval suite with each candidate model inside the same harness, and compare task results, not leaderboard scores. You may find that an expensive frontier model wins on complex tasks while a cheaper model is nearly as good on routine ones. That evidence can justify routing: cheaper models for simple work, stronger models for complex work, and a stronger model plus human approval for high-risk actions.

Without evals, that decision is a guess. With them, it is a measured trade-off.

A minimal example

You do not need a framework to start. A task file and a small script are enough.

A task:

id: refund-eligible-order
input: "Hi, order 1042 arrived broken. Can I get my money back?"
seed: fixtures/orders_basic.sql
checks:
  - type: sql
    query: "SELECT COUNT(*) FROM refunds WHERE order_id = 1042"
    expect: 1
  - type: sql
    query: "SELECT amount FROM refunds WHERE order_id = 1042"
    expect: 49.00
  - type: sql
    query: "SELECT status FROM orders WHERE id = 1042"
    expect: "refunded"
forbidden_tools:
  - delete_order
rubric:
  - "The reply confirms the refund amount and tells the customer when to expect it."
trials: 5

A harness sketch:

import statistics

def run_task(task, agent, env, judge):
    results = []
    for _ in range(task["trials"]):
        env.reset(seed=task["seed"])                # clean, identical world
        transcript = agent.run(task["input"], tools=env.tools)

        outcome_ok = all(env.check(c) for c in task["checks"])
        path_ok = not any(
            call.name in task.get("forbidden_tools", [])
            for call in transcript.tool_calls
        )
        rubric_ok = all(
            judge.passes(transcript.final_reply, criterion)
            for criterion in task.get("rubric", [])
        )

        results.append({
            "passed": outcome_ok and path_ok and rubric_ok,
            "tokens": transcript.total_tokens,
            "turns": len(transcript.turns),
            "transcript": transcript,   # keep it — you will read it
        })

    passes = [r["passed"] for r in results]
    return {
        "task": task["id"],
        "pass_rate": sum(passes) / len(passes),
        "pass_all": all(passes),        # pass^k for this batch
        "pass_any": any(passes),        # pass@k for this batch
        "median_tokens": statistics.median(r["tokens"] for r in results),
        "runs": results,
    }

This is illustrative, not a finished library. The point is the shape: reset, run, check state, check constraints, judge what code can't, repeat, keep the transcript.

Common mistakes

• Grading the message instead of the state. The agent's claim is not evidence.
• Requiring one golden path. Check constraints, not exact steps.
• One run per task. You are measuring luck.
• Trusting an uncalibrated AI judge. Compare it with humans first.
• Hundreds of synthetic tasks, zero real ones. Start from real failures.
• Never reading transcripts. You will miss grader bugs and the real causes of failure.
• Ignoring cost. A 2% quality gain for 5× the cost may be the wrong trade.
• Testing only the happy path. Include missing data, expired credentials, tool timeouts, and permission errors. Real agents spend much of their time recovering.
• Collapsing everything into one score. Keep outcome, safety, efficiency, and communication separate. "Agent score: 87" hides what to fix.
• Letting the eval set go stale. Add a new task every time a real user finds a new failure.

Evals are how an agent earns trust

A demo shows that an agent can work. An eval shows how often it works, where it fails, and what it costs. That is the difference between a toy and a product.

Ntense describes a product, rather than a toy, as something that crosses three boundaries: it delivers the intended value, earns appropriate trust, and survives real operation. Evals are the evidence for all three:

• Value — outcome checks show the customer actually got the result.
• Trust — path constraints show the agent stayed inside its boundaries.
• Operation — repeated trials, cost tracking, and regression suites show it keeps working as prompts, tools, and models change.

Writing good evals is also a judgment skill, not a tooling skill. You have to decide what "done" means for a real person, what must never happen, and which trade-offs are acceptable. The framework will change. That judgment will not.

Start this week

1. Write one paragraph defining "done" for your agent's most important task.
2. Collect ten real failures or real requests and turn each into a task with checks.
3. Build a resettable sandbox, even a crude one.
4. Run every task five times. Record pass rate, pass^5, and cost.
5. Read every failed transcript and label the cause.
6. Fix the most common cause, rerun, and add the tasks to CI.

Evidence before advice applies to your own agent too. Don't tell users it works. Show yourself first.