*[Watch on YouTube](https://www.youtube.com/watch?v=rDtcfG4aZV4&list=PLDzwjhH-4yhU&index=13)*

About this lesson

The videos in this course are a snapshot in time. Strands is under active development, so the code featured on this page reflects the most up-to-date patterns, but the concepts covered in the video still apply. When in doubt, trust the code.

*Code for this lesson: [`samples/13-evals`](https://github.com/aws-samples/sample-building-with-strands-course/tree/main/samples/13-evals)*

## Why Traditional Testing Breaks Down

Agents are nondeterministic. The same input can produce a different (but equally valid) response, a different tool ordering, or a different level of detail. `assert output == expected` doesn’t work. Instead you need LLM-as-a-judge scoring, trajectory validation, multi-turn simulations, and deterministic assertions, combined into eval suites that catch regressions.

Strands Evals is a separate package:

```bash
pip install strands-agents-evals
```

## Basic Evaluation Structure

An evaluation has three parts: **cases** (inputs plus expectations), **evaluators** (how to score), and a **task** function that runs your agent and returns what the evaluators need. Here we evaluate the customer service agent from earlier lessons on both its final output and the sequence of tools it called:

```python
from strands import Agent, AgentSkills
from strands_evals import Case, Experiment
from strands_evals.evaluators import OutputEvaluator, TrajectoryEvaluator
from strands_evals.extractors import tools_use_extractor
from strands_evals.types import TaskOutput

def create_agent():
    return Agent(
        tools=[lookup_customer, get_order_history, process_refund],
        plugins=[AgentSkills(skills=["./skills"]), RefundWorkflowHandler()],
        system_prompt=SYSTEM_PROMPT,
        callback_handler=None,
    )

# Trajectory evaluator: validates tool call ordering
trajectory_evaluator = TrajectoryEvaluator(
    rubric="Score 1.0 if workflow sequence matches expected trajectory...",
    include_inputs=True,
)

# Output evaluator: scores response quality
output_evaluator = OutputEvaluator(
    rubric="Score 1.0 if professional, empathetic, addresses the concern...",
)

def get_response(case: Case) -> TaskOutput:
    agent = create_agent()
    response = agent(case.input)
    trajectory = tools_use_extractor.extract_agent_tools_used_from_messages(agent.messages)
    return TaskOutput(output=str(response), trajectory=trajectory)

cases = [
    Case(
        name="refund-full-workflow",
        input="I'm customer C-1001. Refund my headphones order ORD-5521 please.",
        expected_output="Refund processed with 3-5 day timeline.",
        expected_trajectory=["lookup_customer", "get_order_history", "process_refund"],
    ),
    Case(
        name="missing-customer-id",
        input="I want a refund.",
        expected_output="Ask for customer ID before proceeding.",
        expected_trajectory=["lookup_customer"],
    ),
]

experiment = Experiment(
    cases=cases,
    evaluators=[trajectory_evaluator, output_evaluator],
)

report = experiment.run_evaluations(task=get_response)
```

📂 [customer\_service\_eval.py](https://github.com/aws-samples/sample-building-with-strands-course/tree/main/samples/13-evals/customer_service_eval.py)

## Evaluation Types

| Type | What It Checks |
| --- | --- |
| **Output evaluation** | Quality, tone, correctness of the final response |
| **Trajectory evaluation** | Whether the agent followed the correct tool sequence |
| **Simulation** | Multi-turn stress testing over time |
| **Deterministic** | JSON schema, response length, format. No LLM needed |
| **Chaos testing** | Does the agent recover when tools fail? |
| **Red teaming** | Does the agent resist adversarial attacks? |
| **Experiment Generator** | Auto-bootstraps test cases from agent capabilities |

## Chaos Testing

Production isn’t ideal. Tools time out, APIs go down, responses come back truncated. Chaos testing verifies your agent handles those failures gracefully. The `ChaosPlugin` intercepts tool calls and injects configurable failures.

This snippet evaluates a Pokemon team advisor agent (the full agent lives in the Strands samples repo):

```python
from strands import Agent
from strands_evals.chaos import ChaosCase, ChaosExperiment, ChaosPlugin
from strands_evals.chaos.effects import Timeout, NetworkError, TruncateFields
from strands_evals.evaluators.deterministic import Contains

chaos = ChaosPlugin()
agent = Agent(
    tools=[get_pokemon, get_move],
    context_manager="auto",
    plugins=[chaos],
)

# Define failure scenarios
effect_maps = {
    "api_timeout": {"tool_effects": {"get_move": [Timeout()]}},
    "api_down": {"tool_effects": {"get_pokemon": [NetworkError()]}},
    "partial_response": {"tool_effects": {"get_pokemon": [TruncateFields(max_length=200)]}},
}

# Expand cases across all failure modes (+ a no-failure baseline)
chaos_cases = ChaosCase.expand(
    [Case(name="earthquake_ice_beam", input="Which Pokemon learns both Earthquake and Ice Beam with the highest Attack?")],
    effect_maps,
    include_no_effect_baseline=True,
)

experiment = ChaosExperiment(
    cases=chaos_cases,
    evaluators=[Contains(value="rampardos", case_sensitive=False, name="correct_answer")],
)

report = experiment.run_evaluations(task=lambda case: {"output": str(agent(case.input))})
```

📂 [pokemon-team-advisor](https://github.com/strands-agents/samples/tree/main/python/05-technical-use-cases/pokemon-team-advisor) (full agent in the Strands samples repo)

Does the agent recover when `get_move` times out? When `get_pokemon` returns a network error? Chaos testing lets you find out while you build, not after customers do.

## Red Teaming

Red teaming tests whether your agent resists adversarial pressure: jailbreaks, data exfiltration attempts, excessive agency. The generator inspects your agent’s tools and system prompt and produces targeted attacks; the strategy controls how those attacks escalate across turns.

```python
from strands_evals.experimental.redteam import RedTeamExperiment
from strands_evals.experimental.redteam.generators.adversarial import AdversarialCaseGenerator
from strands_evals.experimental.redteam.strategies import CrescendoStrategy

cases = AdversarialCaseGenerator().generate_cases(
    agent=agent,
    risk_categories=["data_exfiltration", "excessive_agency"],
    num_cases=3,
)

experiment = RedTeamExperiment(
    cases=cases,
    agent_factory=create_agent,  # fresh agent per case so attacks don't bleed across runs
    attack_strategies=[CrescendoStrategy()],
)

report = experiment.run_evaluations()
```

Red teaming validates that your guardrails hold under adversarial conditions, not just happy-path inputs.

## Best Practices

-   Evaluate multi-turn interactions, not just single turns
-   Combine LLM judges with deterministic assertions
-   Wire evals into CI/CD to catch regressions automatically
-   Evolve your eval suite as new failure modes emerge
-   Use a different model for evaluation than for generation
-   Use chaos testing to verify resilience before production
-   Run red team experiments to validate guardrails

## Resources

-   📖 [Strands Evals Quickstart](/pr-cms-4519/docs/user-guide/evals-sdk/quickstart/index.md)
-   📖 [Blog: Reduced cost, better isolation, and more resilience](https://strandsagents.com/blog/reduced-cost-better-isolation-more-resilience/)
-   🛠️ [Strands Evals on GitHub](https://github.com/strands-agents/evals)