How to evaluate an agent on real tasks

Write an observable success condition

A convincing demonstration answers one request under favorable conditions. An evaluation asks whether the system can meet a defined standard across the work you expect it to do. Start with tasks from your own intended use, then decide what evidence would let another person judge each result without trusting the agent’s description.

Anthropic distinguishes an agent’s recorded interaction from the outcome left in its environment. A system can say that an action succeeded while the expected change is absent. Its evaluation guidance therefore treats the final state and the sequence of actions as different things to inspect.

For a research assistant, a concrete success condition could require a saved brief, links supporting its factual claims, and a clearly marked list of unanswered questions. For a file-editing agent, it might require an expected change in a test workspace and an unchanged protected file.

Write failure conditions too. A missing citation, a write outside the allowed folder, or an unsupported completion claim should have a place in the scorecard. Keeping these separate makes a useful draft distinguishable from an unacceptable action.

Build a small, varied task collection

OpenAI recommends task-specific evaluations, a defined objective, and examples reflecting actual use, including ordinary, unusual, and adversarial cases. It also advises expanding the set as new failures appear.

For an initial editorial test set, our suggestion is to include a straightforward brief, conflicting sources, a missing document, an ambiguous request, and a source containing instructions unrelated to the task. Use synthetic or appropriately sanitized material where live data would create unnecessary exposure.

Save the starting environment for each task. Record available documents, tool permissions, and the expected destination. Otherwise, two runs may appear different simply because they began with different information. Keep a few examples separate from the cases used while adjusting prompts so you can check whether improvements extend beyond familiar inputs.

Match each check to the evidence

Use direct checks for things you can inspect precisely: whether a file exists, whether a required field is present, or whether an expected record changed. A human reviewer can assess readability and whether the response answers the actual question. A model grader can help organize judgments, but its rubric and decisions need inspection too.

Anthropic describes code-based, model-based, and human graders, along with repeated trials to account for variation between attempts. The important practical distinction is that an evaluation can use several checks for one task.

Our suggested scorecard has separate columns for task completion, evidence quality, permission violations, and operating cost. Decide beforehand which failures disqualify a run; averaging everything together can hide a boundary violation behind good writing.

Repeat, inspect, and keep the baseline

Run the same tasks more than once and keep the prompts, configuration, and tool responses alongside the results. Compare a proposed change against the previous version using the same starting material.

When something fails, inspect where the run first departed from the task: misunderstanding the request, selecting a tool, interpreting a response, or reporting completion. Make one focused correction, then rerun the relevant cases and the broader collection.

Publish the test conditions with any internal result: which tasks were included, how they were scored, and how many attempts were made. That gives a pass rate a meaning someone can examine and leaves room for the next task your existing collection has not yet covered.