Before you follow along

This video is a snapshot in time. Some APIs and examples have changed since recording. Follow the code on this page, and look for “Changed since recording” notices below.

*[Watch on YouTube](https://www.youtube.com/watch?v=rDtcfG4aZV4&list=PLDzwjhH-4yhU&index=13)*

*Code for this lesson: [`samples/13-evals`](https://github.com/aws-samples/sample-building-with-strands-course/tree/main/samples/13-evals)*

## Why traditional testing breaks down

Agents are nondeterministic. The same input can produce a different (but equally valid) response, a different tool ordering, or a different level of detail. `assert output == expected` doesn’t work. Instead you need LLM-as-a-judge scoring, trajectory validation, multi-turn simulations, and deterministic assertions, combined into eval suites that catch regressions.

Strands Evals is a separate package:

```bash
pip install strands-agents-evals
```

## Basic evaluation structure

An evaluation has three parts: **cases** (inputs plus expectations), **evaluators** (how to score), and a **task** function that runs your agent and returns what the evaluators need. Here we evaluate the customer service agent from earlier lessons on both its final output and the sequence of tools it called:

Run from `samples/13-evals` so the factory can find the sample tools and skills.

```python
from strands import Agent, AgentSkills
from strands_evals import Case, Experiment
from strands_evals.evaluators import OutputEvaluator, TrajectoryEvaluator
from strands_evals.extractors import tools_use_extractor
from strands_evals.types import TaskOutput

from create_agent import create_customer_service_agent

def create_agent():
    return create_customer_service_agent(callback_handler=None)

# Trajectory evaluator: validates tool call ordering
trajectory_evaluator = TrajectoryEvaluator(
    rubric="Match the expected tool order; ignore supplementary skills tool calls.",
    include_inputs=True,
)

# Output evaluator: scores response quality
output_evaluator = OutputEvaluator(
    rubric="Score accuracy, professionalism, and whether the response addresses the request.",
)

def get_response(case: Case) -> TaskOutput:
    agent = create_agent()
    response = agent(case.input)
    trajectory = tools_use_extractor.extract_agent_tools_used_from_messages(agent.messages)
    return TaskOutput(output=str(response), trajectory=trajectory)

cases = [
    Case(
        name="refund-full-workflow",
        input="I'm C-1001. Refund order ORD-5521. Yes, I confirm the refund.",
        expected_output="Refund processed with 3-5 day timeline.",
        expected_trajectory=["lookup_customer", "get_order_history", "process_refund"],
    ),
    Case(
        name="missing-customer-id",
        input="I want a refund.",
        expected_output="Ask for customer ID before proceeding.",
        expected_trajectory=[],
    ),
]

experiment = Experiment(
    cases=cases,
    evaluators=[trajectory_evaluator, output_evaluator],
)

report = experiment.run_evaluations(task=get_response)
report.run_display()
```

📂 [customer\_service\_eval.py](https://github.com/aws-samples/sample-building-with-strands-course/tree/main/samples/13-evals/customer_service_eval.py)

## Evaluation types

| Type | What It Checks |
| --- | --- |
| **Output evaluation** | Quality, tone, correctness of the final response |
| **Trajectory evaluation** | Whether the agent followed the correct tool sequence |
| **Simulation** | Multi-turn stress testing over time |
| **Deterministic** | JSON schema, response length, format. No LLM needed |
| **Chaos testing** | Does the agent recover when tools fail? |
| **Red teaming** | Does the agent resist adversarial attacks? |
| **Experiment Generator** | Auto-bootstraps test cases from agent capabilities |

## Chaos testing

Production isn’t ideal. Tools time out, APIs go down, responses come back truncated. Chaos testing verifies your agent handles those failures gracefully. The `ChaosPlugin` intercepts tool calls and injects configurable failures.

This example uses the course’s mock order tool. Each case starts with a fresh agent.

```python
from strands import Agent
from strands_evals import Case
from strands_evals.chaos import ChaosCase, ChaosExperiment, ChaosPlugin
from strands_evals.chaos.effects import Timeout, NetworkError, TruncateFields
from strands_evals.evaluators import OutputEvaluator
from customer_service_tools import get_order_history

effect_maps = {
    "api_timeout": {"tool_effects": {"get_order_history": [Timeout()]}},
    "api_down": {"tool_effects": {"get_order_history": [NetworkError()]}},
    "partial_response": {
        "tool_effects": {"get_order_history": [TruncateFields(max_length=200)]}
    },
}
chaos_cases = ChaosCase.expand(
    [Case(name="order_status", input="Check the orders for customer C-1001.")],
    effect_maps,
    include_no_effect_baseline=True,
)

def run_case(case):
    agent = Agent(
        tools=[get_order_history], plugins=[ChaosPlugin()], callback_handler=None,
        system_prompt="Use order data. If it is unavailable, explain the failure.",
    )
    return {"output": str(agent(case.input))}

experiment = ChaosExperiment(
    cases=chaos_cases,
    evaluators=[OutputEvaluator(
        rubric="Use returned order data or acknowledge a tool failure. Do not invent orders."
    )],
)
report = experiment.run_evaluations(task=run_case)
report.run_display()
```

Does the agent recover when `get_order_history` times out? When `get_order_history` returns a network error? Chaos testing lets you find out while you build, not after customers do.

## Red teaming

Red teaming tests whether your agent resists adversarial pressure: jailbreaks, data exfiltration attempts, excessive agency. The generator inspects your agent’s tools and system prompt and produces targeted attacks; the strategy controls how those attacks escalate across turns.

```python
from create_agent import create_customer_service_agent
from strands_evals.redteam import (
    AdversarialCaseGenerator,
    CrescendoStrategy,
    RedTeamExperiment,
)

cases = AdversarialCaseGenerator().generate_cases(
    agent=create_customer_service_agent(),
    risk_categories=["data_exfiltration", "excessive_agency"],
    num_cases=3,
)

experiment = RedTeamExperiment(
    cases=cases,
    attack_strategies=[CrescendoStrategy()],
)

# fresh agent per case so attacks don't bleed across runs
report = experiment.run_evaluations(agent_factory=create_customer_service_agent)
```

Red teaming probes the tested guardrails under adversarial conditions. Passing these cases does not prove resistance to every attack.

## Best practices

-   Evaluate multi-turn interactions, not just single turns
-   Combine LLM judges with deterministic assertions
-   Wire evals into CI/CD to catch regressions automatically
-   Evolve your eval suite as new failure modes emerge
-   Use a different model for evaluation than for generation
-   Use chaos testing to verify resilience before production
-   Run red team experiments to validate guardrails

## Resources

-   📖 [Strands Evals Quickstart](/pr-cms-5317/docs/user-guide/evals-sdk/quickstart/index.md)
-   📖 [Blog: Reduced cost, better isolation, and more resilience](https://strandsagents.com/blog/reduced-cost-better-isolation-more-resilience/)
-   🛠️ [Strands Evals on GitHub](https://github.com/strands-agents/evals)