Skip to content

Lesson 13: Evaluating agents

Play

Watch on YouTube

Code for this lesson: samples/13-evals

Why traditional testing breaks down

Section titled “Why traditional testing breaks down”

Agents are nondeterministic. The same input can produce a different (but equally valid) response, a different tool ordering, or a different level of detail. assert output == expected doesn’t work. Instead you need LLM-as-a-judge scoring, trajectory validation, multi-turn simulations, and deterministic assertions, combined into eval suites that catch regressions.

Strands Evals is a separate package:

Terminal window
pip install strands-agents-evals

Basic evaluation structure

Section titled “Basic evaluation structure”

An evaluation has three parts: cases (inputs plus expectations), evaluators (how to score), and a task function that runs your agent and returns what the evaluators need. Here we evaluate the customer service agent from earlier lessons on both its final output and the sequence of tools it called:

Run from samples/13-evals so the factory can find the sample tools and skills.

from strands import Agent, AgentSkills
from strands_evals import Case, Experiment
from strands_evals.evaluators import OutputEvaluator, TrajectoryEvaluator
from strands_evals.extractors import tools_use_extractor
from strands_evals.types import TaskOutput


from create_agent import create_customer_service_agent


def create_agent():
    return create_customer_service_agent(callback_handler=None)


# Trajectory evaluator: validates tool call ordering
trajectory_evaluator = TrajectoryEvaluator(
    rubric="Match the expected tool order; ignore supplementary skills tool calls.",
    include_inputs=True,
)


# Output evaluator: scores response quality
output_evaluator = OutputEvaluator(
    rubric="Score accuracy, professionalism, and whether the response addresses the request.",
)


def get_response(case: Case) -> TaskOutput:
    agent = create_agent()
    response = agent(case.input)
    trajectory = tools_use_extractor.extract_agent_tools_used_from_messages(agent.messages)
    return TaskOutput(output=str(response), trajectory=trajectory)


cases = [
    Case(
        name="refund-full-workflow",
        input="I'm C-1001. Refund order ORD-5521. Yes, I confirm the refund.",
        expected_output="Refund processed with 3-5 day timeline.",
        expected_trajectory=["lookup_customer", "get_order_history", "process_refund"],
    ),
    Case(
        name="missing-customer-id",
        input="I want a refund.",
        expected_output="Ask for customer ID before proceeding.",
        expected_trajectory=[],
    ),
]


experiment = Experiment(
    cases=cases,
    evaluators=[trajectory_evaluator, output_evaluator],
)


report = experiment.run_evaluations(task=get_response)
report.run_display()

📂 customer_service_eval.py

Evaluation types

Section titled “Evaluation types”
TypeWhat It Checks
Output evaluationQuality, tone, correctness of the final response
Trajectory evaluationWhether the agent followed the correct tool sequence
SimulationMulti-turn stress testing over time
DeterministicJSON schema, response length, format. No LLM needed
Chaos testingDoes the agent recover when tools fail?
Red teamingDoes the agent resist adversarial attacks?
Experiment GeneratorAuto-bootstraps test cases from agent capabilities

Chaos testing

Section titled “Chaos testing”

Production isn’t ideal. Tools time out, APIs go down, responses come back truncated. Chaos testing verifies your agent handles those failures gracefully. The ChaosPlugin intercepts tool calls and injects configurable failures.

This example uses the course’s mock order tool. Each case starts with a fresh agent.

from strands import Agent
from strands_evals import Case
from strands_evals.chaos import ChaosCase, ChaosExperiment, ChaosPlugin
from strands_evals.chaos.effects import Timeout, NetworkError, TruncateFields
from strands_evals.evaluators import OutputEvaluator
from customer_service_tools import get_order_history


effect_maps = {
    "api_timeout": {"tool_effects": {"get_order_history": [Timeout()]}},
    "api_down": {"tool_effects": {"get_order_history": [NetworkError()]}},
    "partial_response": {
        "tool_effects": {"get_order_history": [TruncateFields(max_length=200)]}
    },
}
chaos_cases = ChaosCase.expand(
    [Case(name="order_status", input="Check the orders for customer C-1001.")],
    effect_maps,
    include_no_effect_baseline=True,
)


def run_case(case):
    agent = Agent(
        tools=[get_order_history], plugins=[ChaosPlugin()], callback_handler=None,
        system_prompt="Use order data. If it is unavailable, explain the failure.",
    )
    return {"output": str(agent(case.input))}


experiment = ChaosExperiment(
    cases=chaos_cases,
    evaluators=[OutputEvaluator(
        rubric="Use returned order data or acknowledge a tool failure. Do not invent orders."
    )],
)
report = experiment.run_evaluations(task=run_case)
report.run_display()

Does the agent recover when get_order_history times out? When get_order_history returns a network error? Chaos testing lets you find out while you build, not after customers do.

Red teaming

Section titled “Red teaming”

Red teaming tests whether your agent resists adversarial pressure: jailbreaks, data exfiltration attempts, excessive agency. The generator inspects your agent’s tools and system prompt and produces targeted attacks; the strategy controls how those attacks escalate across turns.

from create_agent import create_customer_service_agent
from strands_evals.redteam import (
    AdversarialCaseGenerator,
    CrescendoStrategy,
    RedTeamExperiment,
)


cases = AdversarialCaseGenerator().generate_cases(
    agent=create_customer_service_agent(),
    risk_categories=["data_exfiltration", "excessive_agency"],
    num_cases=3,
)


experiment = RedTeamExperiment(
    cases=cases,
    attack_strategies=[CrescendoStrategy()],
)


# fresh agent per case so attacks don't bleed across runs
report = experiment.run_evaluations(agent_factory=create_customer_service_agent)

Red teaming probes the tested guardrails under adversarial conditions. Passing these cases does not prove resistance to every attack.

Best practices

Section titled “Best practices”
  • Evaluate multi-turn interactions, not just single turns
  • Combine LLM judges with deterministic assertions
  • Wire evals into CI/CD to catch regressions automatically
  • Evolve your eval suite as new failure modes emerge
  • Use a different model for evaluation than for generation
  • Use chaos testing to verify resilience before production
  • Run red team experiments to validate guardrails

Resources

Section titled “Resources”