How evaluation works

Every evaluation in Strands Evals is built from the same parts. An experiment holds the cases you want to test. For each case, a task runs your agent and evaluators score the result. Optionally, detectors read failed runs and explain what went wrong. The rest of this section covers each part in detail.

What an evaluation is made of

An experiment is what you run. It holds a list of cases and the evaluators to apply to each one, and produces a report with every case’s scores.

A case describes one test. It holds the input to send to your agent, plus optional fields that evaluators compare against, such as expected_output, expected_trajectory, or expected_assertion. You can also attach metadata to filter and group results later.

A task is a function that runs your agent on a case’s input and returns the result. The result always includes the output and can also include a trajectory, a record of what the agent did along the way. A trajectory takes one of two forms: a list you build yourself, such as the names of the tools called, or a Session, the full trace of model and tool calls. With the @eval_task decorator, your task can simply return an Agent. Pass it a TracedHandler to capture a Session as well (see Collecting traces).

An evaluator scores one case’s result and returns a score, a pass or fail, and its reasoning. Most evaluators use a model as a judge; pass model= to use a different one than the default. Others are plain code (see Deterministic evaluators).

A detector explains failures instead of scoring them. When you turn on diagnosis with Experiment(diagnosis_config=...), detectors read the Session of each failing case and report what went wrong and why (see Diagnosis).

The levels an evaluator scores at

Evaluators differ in how much of a run they judge. Output-level evaluators read what your task returns directly: the output, plus a list trajectory or interactions if the evaluator uses them. Every other built-in evaluator reads a Session, and its level decides which slice of that Session it judges.

A Session holds one trace per agent invocation (a turn), and each trace holds the spans for that turn’s model calls and tool executions.

Level	Judges	Example
Output	The final output	OutputEvaluator
TOOL_LEVEL	Each tool call, in context	ToolSelectionAccuracyEvaluator
TRACE_LEVEL	The latest turn, in context	HelpfulnessEvaluator
SESSION_LEVEL	The whole conversation	GoalSuccessRateEvaluator

Output is a concept, not a setting: output-level evaluators leave evaluation_level unset. TOOL_LEVEL, TRACE_LEVEL, and SESSION_LEVEL are values of the EvaluationLevel enum. A tool-level evaluator returns one result per tool call, so a case passes only if every call passes, and its score is the average.

Tool-, trace-, and session-level evaluators need your task to return a Session. Decorate the task with @eval_task(TracedHandler()) to collect one. Without it, these evaluators fail the case with a score of 0 and an Evaluator error reason.

When you combine evaluators in one experiment, you usually mix levels on purpose: a fast output check plus a check that reads the whole run. The evaluators overview lists every built-in evaluator with its level.

Where simulators and detectors fit

Two pieces sit on either side of scoring.

Simulators generate the interactions an evaluator needs. A single prompt gives you one turn; a real agent faces a back-and-forth. A simulator plays the user, or stands in for a tool the agent calls, so you can score multi-turn behavior without a human in the loop.

Detectors explain failures. An evaluator tells you a run scored 0.2; a detector reads that run’s Session, finds the span where things went wrong, and proposes a root cause. Detectors run only when you pass a DiagnosisConfig to the experiment. By default they diagnose failing cases; set trigger=DiagnosisTrigger.ALWAYS to diagnose every case. Scoring tells you whether a run failed; diagnosis tells you why.

How it fits together

For a single case, the flow runs left to right: the task produces output and a trajectory, evaluators score them, and, if diagnosis is on, detectors diagnose the case when it fails.

The experiment repeats this for every case, then aggregates the scores into a report you can print, save, and compare against a later run.

Next steps

• Run the flow end to end in the quickstart.
• Pick scorers from the evaluators overview.
• Diagnose failing runs with detectors.
• Drive multi-turn runs with simulators.