## Overview

Simulators evaluate your agent across a full interaction instead of a single input/output pair. They drive multi-turn conversations and generate tool responses at runtime, so you can test how the agent handles a dialogue and works toward a goal. Strands Evals ships two: `ActorSimulator` plays a simulated user (or any other conversational participant), and `ToolSimulator` stands in for the tools the agent calls.

## Why simulators?

Traditional evaluation approaches have limitations when assessing conversational agents:

**Static Evaluators:**

-   Evaluate single input/output pairs
-   Cannot test multi-turn conversation flow
-   Miss context-dependent behaviors
-   Don’t capture goal-oriented interactions

**Simulators:**

-   Generate dynamic, multi-turn conversations
-   Adapt responses based on agent behavior
-   Test goal completion in realistic scenarios
-   Evaluate conversation flow and context maintenance
-   Enable testing without predefined scripts
-   Simulate tool behavior without live infrastructure

## When to use simulators

Use simulators when you need to:

-   **Evaluate Multi-turn Conversations**: Test agents across multiple conversation turns
-   **Assess Goal Completion**: Verify agents can achieve user objectives through dialogue
-   **Test Conversation Flow**: Evaluate how agents handle context and follow-up questions
-   **Generate Diverse Interactions**: Create varied conversation patterns automatically
-   **Evaluate Without Scripts**: Test agents without predefined conversation paths
-   **Simulate Real Users**: Generate realistic user behavior patterns
-   **Test Tool Usage Without Infrastructure**: Evaluate agent tool-use behavior without live APIs, databases, or services

## ActorSimulator

The `ActorSimulator` is the core simulator class in Strands Evals. An “actor” is any conversational participant: a user, a customer service representative, a domain expert, an adversarial tester, or anything else that engages in dialogue. The simulator holds an actor profile, generates responses from the conversation history, and tracks goal completion. Change the profile and system prompt and you change who the simulator plays.

### User simulation

The most common use of `ActorSimulator` is **user simulation**: playing a realistic end-user interacting with your agent during evaluation. This is the primary use case the documentation covers.

See the [User Simulation Guide](/pr-cms-4519/docs/user-guide/evals-sdk/simulators/user_simulation/index.md) for the full walkthrough.

### Other actor types

To simulate an actor other than a user, supply a custom profile. The same class covers:

-   **Customer Support Representatives**: Test agent-to-agent interactions
-   **Domain Experts**: Simulate specialized-knowledge conversations
-   **Adversarial Actors**: Test how the agent handles hostile or malformed input and edge cases
-   **Internal Staff**: Evaluate internal tooling workflows

## ToolSimulator

The `ToolSimulator` stands in for the tools your agent calls. Instead of executing the real function, it asks an LLM to generate a schema-valid response and keeps state across calls so related tools stay consistent.

Reach for it when the real tools require live infrastructure, when you need controllable behavior for evaluation, or when the tools are still under development.

```python
from typing import Any
from pydantic import BaseModel, Field
from strands import Agent
from strands_evals.simulation.tool_simulator import ToolSimulator

tool_simulator = ToolSimulator()

class WeatherResponse(BaseModel):
    temperature: float = Field(..., description="Temperature in Fahrenheit")
    conditions: str = Field(..., description="Weather conditions")

@tool_simulator.tool(output_schema=WeatherResponse)
def get_weather(city: str) -> dict[str, Any]:
    """Get current weather for a city."""
    pass

weather_tool = tool_simulator.get_tool("get_weather")
agent = Agent(tools=[weather_tool], callback_handler=None)
response = agent("What's the weather in Seattle?")
```

Key capabilities:

-   **Decorator-based registration** with automatic metadata extraction from function signatures
-   **Schema-validated responses** via Pydantic output models
-   **Shared state** across related tools via `share_state_id` (e.g., sensor + controller operating on the same environment)
-   **Stateful context** with initial state descriptions and bounded call history cache

See the [Tool Simulation Guide](/pr-cms-4519/docs/user-guide/evals-sdk/simulators/tool_simulation/index.md) for the full walkthrough.

## Extensibility

The simulator framework is designed to be extensible. `ActorSimulator` and `ToolSimulator` provide general-purpose foundations, and additional specialized simulators can be built for specific evaluation patterns as needs emerge.

## Simulators vs evaluators

Understanding when to use simulators versus evaluators:

| Aspect | Evaluators | ActorSimulator | ToolSimulator |
| --- | --- | --- | --- |
| **Role** | Passive assessment | Active conversation participant | Simulated tool execution |
| **Turns** | Single turn | Multi-turn | Per tool call |
| **Adaptation** | Static criteria | Dynamic responses | Stateful responses |
| **Use Case** | Output quality | Conversation flow | Tool-use behavior |
| **Goal** | Score responses | Drive interactions | Replace infrastructure |

**Use Together:** Simulators and evaluators complement each other. Use simulators to generate multi-turn conversations, then use evaluators to assess the quality of those interactions.

## Integration with evaluators

Simulators work with trace-based evaluators:

```python
import asyncio

from strands import Agent
from strands_evals import Case, Experiment, ActorSimulator
from strands_evals.evaluators import HelpfulnessEvaluator, GoalSuccessRateEvaluator
from strands_evals.mappers import StrandsInMemorySessionMapper
from strands_evals.telemetry import StrandsEvalsTelemetry

# Setup telemetry
telemetry = StrandsEvalsTelemetry().setup_in_memory_exporter()
memory_exporter = telemetry.in_memory_exporter

def task_function(case: Case) -> dict:
    # Create simulator to drive conversation
    simulator = ActorSimulator.from_case_for_user_simulator(
        case=case,
        max_turns=10
    )

    # Create agent to evaluate
    agent = Agent(
        trace_attributes={
            "gen_ai.conversation.id": case.session_id,
            "session.id": case.session_id
        },
        callback_handler=None
    )

    # Run multi-turn conversation
    user_message = case.input

    while simulator.has_next():
        agent_response = agent(user_message)
        turn_spans = list(memory_exporter.get_finished_spans())

        user_result = simulator.act(str(agent_response))
        user_message = str(user_result.structured_output.message)

    all_spans = memory_exporter.get_finished_spans()
    # Map to session for evaluation
    mapper = StrandsInMemorySessionMapper()
    session = mapper.map_to_session(all_spans, session_id=case.session_id)

    return {"output": str(agent_response), "trajectory": session}

# Use evaluators to assess simulated conversations
evaluators = [
    HelpfulnessEvaluator(),
    GoalSuccessRateEvaluator()
]

# Setup test cases
test_cases = [
    Case(
        input="I need to book a flight to Paris",
        metadata={"task_description": "Flight booking confirmed"}
    ),
    Case(
        input="Help me write a Python function to sort a list",
        metadata={"task_description": "Programming assistance"}
    )
]

experiment = Experiment(cases=test_cases, evaluators=evaluators)

async def main():
    report = await experiment.run_evaluations_async(task_function)

asyncio.run(main())
```

## Best practices

### 1\. Define clear goals

Simulators work best with well-defined objectives:

```python
case = Case(
    input="I need to book a flight",
    metadata={
        "task_description": "Flight booked with confirmation number and email sent"
    }
)
```

### 2\. Set appropriate turn limits

Balance thoroughness with efficiency:

```python
# Simple tasks: 3-5 turns
simulator = ActorSimulator.from_case_for_user_simulator(case=case, max_turns=5)

# Complex tasks: 8-15 turns
simulator = ActorSimulator.from_case_for_user_simulator(case=case, max_turns=12)
```

### 3\. Combine with multiple evaluators

Assess different aspects of simulated conversations:

```python
evaluators = [
    HelpfulnessEvaluator(),      # User experience
    GoalSuccessRateEvaluator(),  # Task completion
    FaithfulnessEvaluator()      # Response accuracy
]
```

### 4\. Log conversations for analysis

Capture conversation details for debugging:

```python
conversation_log = []
while simulator.has_next():
    # ... conversation logic ...
    conversation_log.append({
        "turn": turn_number,
        "agent": agent_message,
        "simulator": simulator_message,
        "reasoning": simulator_reasoning
    })
```

## Common patterns

### Pattern 1: Goal completion testing

The simulator sets `result.structured_output.stop = True` (and its own `simulator.stop` flag) when the actor signals it has completed the goal. Inspect that flag rather than scanning the message text:

```python
def test_goal_completion(case: Case) -> bool:
    simulator = ActorSimulator.from_case_for_user_simulator(case=case)
    agent = Agent(system_prompt="Your prompt")

    user_message = case.input
    while simulator.has_next():
        agent_response = agent(user_message)
        user_result = simulator.act(str(agent_response))
        user_message = str(user_result.structured_output.message)

    # has_next() is False either because the simulator stopped (goal reached)
    # or because max_turns was hit. Distinguish via stop_reason if needed.
    return user_result.structured_output.stop and (
        getattr(user_result.structured_output, "stop_reason", "") == "goal_completed"
    )
```

### Pattern 2: Conversation flow analysis

```python
def analyze_conversation_flow(case: Case) -> dict:
    simulator = ActorSimulator.from_case_for_user_simulator(case=case)
    agent = Agent(system_prompt="Your prompt")

    metrics = {
        "turns": 0,
        "agent_questions": 0,
        "user_clarifications": 0
    }

    user_message = case.input
    while simulator.has_next():
        agent_response = agent(user_message)
        if "?" in str(agent_response):
            metrics["agent_questions"] += 1

        user_result = simulator.act(str(agent_response))
        user_message = str(user_result.structured_output.message)
        metrics["turns"] += 1

    return metrics
```

### Pattern 3: Comparative evaluation

```python
def compare_agent_configurations(case: Case, configs: list) -> dict:
    results = {}

    for config in configs:
        simulator = ActorSimulator.from_case_for_user_simulator(case=case)
        agent = Agent(**config)

        # Run conversation and collect metrics
        # ... evaluation logic ...

        results[config["name"]] = metrics

    return results
```

## Next steps

-   [User Simulation Guide](/pr-cms-4519/docs/user-guide/evals-sdk/simulators/user_simulation/index.md): Simulate multi-turn user conversations
-   [Customizing User Simulation](/pr-cms-4519/docs/user-guide/evals-sdk/simulators/customize_user_simulation/index.md): Custom profiles, prompts, tools, and models
-   [Tool Simulation Guide](/pr-cms-4519/docs/user-guide/evals-sdk/simulators/tool_simulation/index.md): Simulate tool behavior with LLM-powered responses
-   [Evaluators](/pr-cms-4519/docs/user-guide/evals-sdk/evaluators/output_evaluator/index.md): Combine with evaluators

## Related documentation

-   [Quickstart Guide](/pr-cms-4519/docs/user-guide/evals-sdk/quickstart/index.md): Get started with Strands Evals
-   [Evaluators Overview](/pr-cms-4519/docs/user-guide/evals-sdk/evaluators/output_evaluator/index.md): Learn about evaluators
-   [Experiment Generator](/pr-cms-4519/docs/user-guide/evals-sdk/experiment_generator/index.md): Generate test cases automatically

## Related pages

- [Experiment generator](/pr-cms-4519/docs/user-guide/evals-sdk/experiment_generator/index.md) (1 shared tag)
- [Plan topics for coverage](/pr-cms-4519/docs/user-guide/evals-sdk/topic_planning/index.md) (1 shared tag)
- [Customizing user simulation](/pr-cms-4519/docs/user-guide/evals-sdk/simulators/customize_user_simulation/index.md) (1 shared tag)
- [User simulation](/pr-cms-4519/docs/user-guide/evals-sdk/simulators/user_simulation/index.md) (1 shared tag)
- [Chaos testing](/pr-cms-4519/docs/user-guide/evals-sdk/chaos_testing/index.md) (1 shared tag)
- [Tool simulation](/pr-cms-4519/docs/user-guide/evals-sdk/simulators/tool_simulation/index.md) (1 shared tag)
- [Failure communication evaluator](/pr-cms-4519/docs/user-guide/evals-sdk/evaluators/failure_communication_evaluator/index.md) (1 shared tag)
- [Partial completion evaluator](/pr-cms-4519/docs/user-guide/evals-sdk/evaluators/partial_completion_evaluator/index.md) (1 shared tag)
- [Recovery strategy evaluator](/pr-cms-4519/docs/user-guide/evals-sdk/evaluators/recovery_strategy_evaluator/index.md) (1 shared tag)