Skip to content

Lesson 13: Evaluating Agents

Play

Watch on YouTube

Code for this lesson: samples/13-evals

Why Traditional Testing Breaks Down

Section titled “Why Traditional Testing Breaks Down”

Agents are nondeterministic. The same input can produce a different (but equally valid) response, a different tool ordering, or a different level of detail. assert output == expected doesn’t work. Instead you need LLM-as-a-judge scoring, trajectory validation, multi-turn simulations, and deterministic assertions, combined into eval suites that catch regressions.

Strands Evals is a separate package:

Terminal window
pip install strands-agents-evals

Basic Evaluation Structure

Section titled “Basic Evaluation Structure”

An evaluation has three parts: cases (inputs plus expectations), evaluators (how to score), and a task function that runs your agent and returns what the evaluators need. Here we evaluate the customer service agent from earlier lessons on both its final output and the sequence of tools it called:

from strands import Agent, AgentSkills
from strands_evals import Case, Experiment
from strands_evals.evaluators import OutputEvaluator, TrajectoryEvaluator
from strands_evals.extractors import tools_use_extractor
from strands_evals.types import TaskOutput


def create_agent():
    return Agent(
        tools=[lookup_customer, get_order_history, process_refund],
        plugins=[AgentSkills(skills=["./skills"]), RefundWorkflowHandler()],
        system_prompt=SYSTEM_PROMPT,
        callback_handler=None,
    )


# Trajectory evaluator: validates tool call ordering
trajectory_evaluator = TrajectoryEvaluator(
    rubric="Score 1.0 if workflow sequence matches expected trajectory...",
    include_inputs=True,
)


# Output evaluator: scores response quality
output_evaluator = OutputEvaluator(
    rubric="Score 1.0 if professional, empathetic, addresses the concern...",
)


def get_response(case: Case) -> TaskOutput:
    agent = create_agent()
    response = agent(case.input)
    trajectory = tools_use_extractor.extract_agent_tools_used_from_messages(agent.messages)
    return TaskOutput(output=str(response), trajectory=trajectory)


cases = [
    Case(
        name="refund-full-workflow",
        input="I'm customer C-1001. Refund my headphones order ORD-5521 please.",
        expected_output="Refund processed with 3-5 day timeline.",
        expected_trajectory=["lookup_customer", "get_order_history", "process_refund"],
    ),
    Case(
        name="missing-customer-id",
        input="I want a refund.",
        expected_output="Ask for customer ID before proceeding.",
        expected_trajectory=["lookup_customer"],
    ),
]


experiment = Experiment(
    cases=cases,
    evaluators=[trajectory_evaluator, output_evaluator],
)


report = experiment.run_evaluations(task=get_response)

📂 customer_service_eval.py

Evaluation Types

Section titled “Evaluation Types”
TypeWhat It Checks
Output evaluationQuality, tone, correctness of the final response
Trajectory evaluationWhether the agent followed the correct tool sequence
SimulationMulti-turn stress testing over time
DeterministicJSON schema, response length, format. No LLM needed
Chaos testingDoes the agent recover when tools fail?
Red teamingDoes the agent resist adversarial attacks?
Experiment GeneratorAuto-bootstraps test cases from agent capabilities

Chaos Testing

Section titled “Chaos Testing”

Production isn’t ideal. Tools time out, APIs go down, responses come back truncated. Chaos testing verifies your agent handles those failures gracefully. The ChaosPlugin intercepts tool calls and injects configurable failures.

This snippet evaluates a Pokemon team advisor agent (the full agent lives in the Strands samples repo):

from strands import Agent
from strands_evals.chaos import ChaosCase, ChaosExperiment, ChaosPlugin
from strands_evals.chaos.effects import Timeout, NetworkError, TruncateFields
from strands_evals.evaluators.deterministic import Contains


chaos = ChaosPlugin()
agent = Agent(
    tools=[get_pokemon, get_move],
    context_manager="auto",
    plugins=[chaos],
)


# Define failure scenarios
effect_maps = {
    "api_timeout": {"tool_effects": {"get_move": [Timeout()]}},
    "api_down": {"tool_effects": {"get_pokemon": [NetworkError()]}},
    "partial_response": {"tool_effects": {"get_pokemon": [TruncateFields(max_length=200)]}},
}


# Expand cases across all failure modes (+ a no-failure baseline)
chaos_cases = ChaosCase.expand(
    [Case(name="earthquake_ice_beam", input="Which Pokemon learns both Earthquake and Ice Beam with the highest Attack?")],
    effect_maps,
    include_no_effect_baseline=True,
)


experiment = ChaosExperiment(
    cases=chaos_cases,
    evaluators=[Contains(value="rampardos", case_sensitive=False, name="correct_answer")],
)


report = experiment.run_evaluations(task=lambda case: {"output": str(agent(case.input))})

📂 pokemon-team-advisor (full agent in the Strands samples repo)

Does the agent recover when get_move times out? When get_pokemon returns a network error? Chaos testing lets you find out while you build, not after customers do.

Red Teaming

Section titled “Red Teaming”

Red teaming tests whether your agent resists adversarial pressure: jailbreaks, data exfiltration attempts, excessive agency. The generator inspects your agent’s tools and system prompt and produces targeted attacks; the strategy controls how those attacks escalate across turns.

from strands_evals.experimental.redteam import RedTeamExperiment
from strands_evals.experimental.redteam.generators.adversarial import AdversarialCaseGenerator
from strands_evals.experimental.redteam.strategies import CrescendoStrategy


cases = AdversarialCaseGenerator().generate_cases(
    agent=agent,
    risk_categories=["data_exfiltration", "excessive_agency"],
    num_cases=3,
)


experiment = RedTeamExperiment(
    cases=cases,
    agent_factory=create_agent,  # fresh agent per case so attacks don't bleed across runs
    attack_strategies=[CrescendoStrategy()],
)


report = experiment.run_evaluations()

Red teaming validates that your guardrails hold under adversarial conditions, not just happy-path inputs.

Best Practices

Section titled “Best Practices”
  • Evaluate multi-turn interactions, not just single turns
  • Combine LLM judges with deterministic assertions
  • Wire evals into CI/CD to catch regressions automatically
  • Evolve your eval suite as new failure modes emerge
  • Use a different model for evaluation than for generation
  • Use chaos testing to verify resilience before production
  • Run red team experiments to validate guardrails

Resources

Section titled “Resources”