Every evaluation in Strands Evals is built from the same parts. An experiment holds the cases you want to test. For each case, a task runs your agent and evaluators score the result. Optionally, detectors read failed runs and explain what went wrong. The rest of this section covers each part in detail.

## What an evaluation is made of

An **experiment** is what you run. It holds a list of cases and the evaluators to apply to each one, and produces a report with every case’s scores.

A **case** describes one test. It holds the input to send to your agent, plus optional fields that evaluators compare against, such as `expected_output`, `expected_trajectory`, or `expected_assertion`. You can also attach metadata to filter and group results later.

A **task** is a function that runs your agent on a case’s input and returns the result. The result always includes the output and can also include a trajectory, a record of what the agent did along the way. A trajectory takes one of two forms: a list you build yourself, such as the names of the tools called, or a `Session`, the full trace of model and tool calls. With the [`@eval_task` decorator](/pr-cms-5317/docs/user-guide/evals-sdk/how-to/eval_task/index.md), your task can simply return an `Agent`. Pass it a `TracedHandler` to capture a `Session` as well (see [Collecting traces](/pr-cms-5317/docs/user-guide/evals-sdk/how-to/eval_task/index.md#collecting-traces-with-tracedhandler)).

An **evaluator** scores one case’s result and returns a score, a pass or fail, and its reasoning. Most evaluators use a model as a judge; pass `model=` to use a different one than the default. Others are plain code (see [Deterministic evaluators](/pr-cms-5317/docs/user-guide/evals-sdk/evaluators/deterministic_evaluators/index.md)).

A **detector** explains failures instead of scoring them. When you turn on diagnosis with `Experiment(diagnosis_config=...)`, detectors read the `Session` of each failing case and report what went wrong and why (see [Diagnosis](/pr-cms-5317/docs/user-guide/evals-sdk/detectors/diagnosis/index.md)).

## The levels an evaluator scores at

Evaluators differ in how much of a run they judge. Output-level evaluators read what your task returns directly: the output, plus a list trajectory or interactions if the evaluator uses them. Every other built-in evaluator reads a `Session`, and its level decides which slice of that `Session` it judges.

A `Session` holds one trace per agent invocation (a turn), and each trace holds the spans for that turn’s model calls and tool executions.

| Level | Judges | Example |
| --- | --- | --- |
| Output | The final output | `OutputEvaluator` |
| `TOOL_LEVEL` | Each tool call, in context | `ToolSelectionAccuracyEvaluator` |
| `TRACE_LEVEL` | The latest turn, in context | `HelpfulnessEvaluator` |
| `SESSION_LEVEL` | The whole conversation | `GoalSuccessRateEvaluator` |

Output is a concept, not a setting: output-level evaluators leave `evaluation_level` unset. `TOOL_LEVEL`, `TRACE_LEVEL`, and `SESSION_LEVEL` are values of the `EvaluationLevel` enum. A tool-level evaluator returns one result per tool call, so a case passes only if every call passes, and its score is the average.

Tool-, trace-, and session-level evaluators need your task to return a `Session`. Decorate the task with `@eval_task(TracedHandler())` to collect one. Without it, these evaluators fail the case with a score of 0 and an `Evaluator error` reason.

When you combine evaluators in one experiment, you usually mix levels on purpose: a fast output check plus a check that reads the whole run. The [evaluators overview](/pr-cms-5317/docs/user-guide/evals-sdk/evaluators/index.md) lists every built-in evaluator with its level.

## Where simulators and detectors fit

Two pieces sit on either side of scoring.

**Simulators** generate the interactions an evaluator needs. A single prompt gives you one turn; a real agent faces a back-and-forth. A [simulator](/pr-cms-5317/docs/user-guide/evals-sdk/simulators/index.md) plays the user, or stands in for a tool the agent calls, so you can score multi-turn behavior without a human in the loop.

**Detectors** explain failures. An evaluator tells you a run scored 0.2; a [detector](/pr-cms-5317/docs/user-guide/evals-sdk/detectors/index.md) reads that run’s `Session`, finds the span where things went wrong, and proposes a root cause. Detectors run only when you pass a `DiagnosisConfig` to the experiment. By default they diagnose failing cases; set `trigger=DiagnosisTrigger.ALWAYS` to diagnose every case. Scoring tells you *whether* a run failed; diagnosis tells you *why*.

## How it fits together

For a single case, the flow runs left to right: the task produces output and a trajectory, evaluators score them, and, if diagnosis is on, detectors diagnose the case when it fails.

> **Diagram:** How one case flows through an experiment, which repeats it for every case. The case (an input and what you expect back) goes to the task, which runs your agent. The task's result (its output and, optionally, its trajectory) goes to the evaluators, which score it. When a case fails, the optional detectors read the run's Session and diagnose why it failed.

The experiment repeats this for every case, then aggregates the scores into a report you can print, save, and compare against a later run.

## Next steps

-   Run the flow end to end in the [quickstart](/pr-cms-5317/docs/user-guide/evals-sdk/quickstart/index.md).
-   Pick scorers from the [evaluators overview](/pr-cms-5317/docs/user-guide/evals-sdk/evaluators/index.md).
-   Diagnose failing runs with [detectors](/pr-cms-5317/docs/user-guide/evals-sdk/detectors/index.md).
-   Drive multi-turn runs with [simulators](/pr-cms-5317/docs/user-guide/evals-sdk/simulators/index.md).