## Overview

When you need an evaluation suite but do not want to hand-author every case, `ExperimentGenerator` writes them for you. Give it a description of your agent’s tools and task; it uses an LLM to generate test cases, spreads them across topics and difficulty levels, and can attach a scored rubric for the default evaluators.

Reach for it when you want to:

-   Bootstrap an evaluation experiment without writing cases by hand.
-   Spread cases across multiple topics or scenarios (see [Plan topics for coverage](/pr-cms-4519/docs/user-guide/evals-sdk/topic_planning/index.md)).
-   Generate rubrics automatically for the default evaluators.
-   Expand an existing experiment with new cases, or adapt one task’s experiment to a similar task.
-   Support custom input, output, and trajectory types, and generate cases in parallel.

## Basic usage

### Simple generation from context

```python
import asyncio
from strands_evals.generators import ExperimentGenerator
from strands_evals.evaluators import OutputEvaluator

# Initialize generator
generator = ExperimentGenerator[str, str](
    input_type=str,
    output_type=str,
    include_expected_output=True
)

# Generate experiment from context
async def generate_experiment():
    experiment = await generator.from_context_async(
        context="""
        Available tools:
        - calculator(expression: str) -> float: Evaluate mathematical expressions
        - current_time() -> str: Get current date and time
        """,
        task_description="Math and time assistant",
        num_cases=5,
        evaluator=OutputEvaluator
    )
    return experiment

# Run generation
experiment = asyncio.run(generate_experiment())
print(f"Generated {len(experiment.cases)} test cases")
```

To distribute generated cases across diverse, non-overlapping topics (either by passing `num_topics` or by planning topics explicitly with `TopicPlanner`), see [Plan topics for coverage](/pr-cms-4519/docs/user-guide/evals-sdk/topic_planning/index.md).

### From the command line

The [`strands-evals generate`](/pr-cms-4519/docs/user-guide/evals-sdk/cli/generate/index.md) subcommand wraps the same generator, so you can bootstrap an experiment without writing a script:

```bash
# Without topics: 10 cases from a context description
strands-evals generate \
  --context "$(cat tools.txt)" \
  --task-description "Math and time assistant" \
  --num-cases 10 \
  --evaluator OutputEvaluator \
  -o experiments/generated.json

# With topics: spread 15 cases across 3 planned topics
strands-evals generate \
  --context "$(cat tools.txt)" \
  --task-description "Customer service assistant" \
  --num-cases 15 \
  --num-topics 3 \
  --evaluator TrajectoryEvaluator \
  -o experiments/generated.json
```

The CLI can also expand an existing experiment file via `--experiment`; see [`strands-evals generate`](/pr-cms-4519/docs/user-guide/evals-sdk/cli/generate/index.md) for the full flag set.

## Generation methods

### 1\. From context

Generate experiments based on specific context that test cases should reference:

```python
async def generate_from_context():
    experiment = await generator.from_context_async(
        context="Agent with weather API and location tools",
        task_description="Weather information assistant",
        num_cases=10,
        num_topics=2,  # Optional: distribute across topics
        evaluator=OutputEvaluator
    )
    return experiment
```

### 2\. From scratch

Generate experiments from topic lists and task descriptions:

```python
async def generate_from_scratch():
    experiment = await generator.from_scratch_async(
        topics=["product search", "order tracking", "returns"],
        task_description="E-commerce customer service",
        num_cases=12,
        evaluator=TrajectoryEvaluator
    )
    return experiment
```

### 3\. From existing experiment

Create new experiments inspired by existing ones:

```python
async def generate_from_experiment():
    # Load existing experiment
    source_experiment = Experiment.from_file("original_experiment.json")

    # Generate similar experiment for new task
    new_experiment = await generator.from_experiment_async(
        source_experiment=source_experiment,
        task_description="New task with similar structure",
        num_cases=8,
        extra_information="Additional context about tools and capabilities"
    )
    return new_experiment
```

### 4\. Update existing experiment

Extend experiments with additional test cases:

```python
async def update_experiment():
    source_experiment = Experiment.from_file("current_experiment.json")

    updated_experiment = await generator.update_current_experiment_async(
        source_experiment=source_experiment,
        task_description="Enhanced task description",
        num_cases=5,  # Add 5 new cases
        context="Additional context for new cases",
        add_new_cases=True,
        add_new_rubric=True
    )
    return updated_experiment
```

## Configuration options

### Input/output types

Configure the structure of generated test cases:

```python
from typing import Dict, List

# Complex types
generator = ExperimentGenerator[Dict[str, str], List[str]](
    input_type=Dict[str, str],
    output_type=List[str],
    include_expected_output=True,
    include_expected_trajectory=True,
    include_metadata=True
)
```

### Parallel generation

Control concurrent test case generation:

```python
generator = ExperimentGenerator[str, str](
    input_type=str,
    output_type=str,
    max_parallel_num_cases=20  # Generate up to 20 cases in parallel
)
```

### Custom prompts

Customize generation behavior with custom prompts:

```python
generator = ExperimentGenerator[str, str](
    input_type=str,
    output_type=str,
    case_system_prompt="Custom prompt for case generation...",
    rubric_system_prompt="Custom prompt for rubric generation...",
)
```

Both prompts default to the built-in templates in `strands_evals.generators.prompt_template.prompt_templates`.

## Complete example: Multi-step dataset generation

```python
import asyncio
from strands_evals.generators import ExperimentGenerator
from strands_evals.evaluators import OutputEvaluator, TrajectoryEvaluator

async def build_dataset():
    # Initialize generator with trajectory support
    generator = ExperimentGenerator[str, str](
        input_type=str,
        output_type=str,
        include_expected_output=True,
        include_expected_trajectory=True,
        include_metadata=True
    )

    # Step 1: Generate initial experiment with topic planning
    print("Step 1: Generating initial experiment...")
    experiment = await generator.from_context_async(
        context="""
        Multi-agent system with:
        - Research agent: Searches and analyzes information
        - Writing agent: Creates content and summaries
        - Review agent: Validates and improves outputs

        Tools available:
        - web_search(query: str) -> str
        - summarize(text: str) -> str
        - fact_check(claim: str) -> bool
        """,
        task_description="Research and content creation assistant",
        num_cases=15,
        num_topics=3,  # Research, Writing, Review
        evaluator=TrajectoryEvaluator
    )

    print(f"Generated {len(experiment.cases)} cases across 3 topics")

    # Step 2: Add more cases to expand coverage
    print("\nStep 2: Expanding experiment...")
    expanded_experiment = await generator.update_current_experiment_async(
        source_experiment=experiment,
        task_description="Research and content creation with edge cases",
        num_cases=5,
        context="Focus on error handling and complex multi-step scenarios",
        add_new_cases=True,
        add_new_rubric=False  # Keep existing rubric
    )

    print(f"Expanded to {len(expanded_experiment.cases)} total cases")

    # Step 3: Add a second LLM-judge evaluator built on a generated rubric.
    # construct_evaluator_async only accepts the default evaluator classes
    # (OutputEvaluator, TrajectoryEvaluator, InteractionsEvaluator).
    print("\nStep 3: Adding output-quality evaluator...")
    output_eval = await generator.construct_evaluator_async(
        prompt="Evaluate output quality for research and content creation tasks",
        evaluator=OutputEvaluator
    )
    expanded_experiment.evaluators.append(output_eval)

    # For non-default evaluators (e.g. HelpfulnessEvaluator), instantiate directly:
    # expanded_experiment.evaluators.append(HelpfulnessEvaluator())

    # Step 4: Save experiment
    expanded_experiment.to_file("research_dataset")
    print("\nDataset saved to ./research_dataset.json")

    return expanded_experiment

# Run the multi-step generation
experiment = asyncio.run(build_dataset())

# Examine results
print(f"\nFinal experiment:")
print(f"- Total cases: {len(experiment.cases)}")
print(f"- Evaluators: {len(experiment.evaluators)}")
print(f"- Categories: {set(c.metadata.get('category', 'unknown') for c in experiment.cases if c.metadata)}")
```

## Difficulty levels

The generator automatically distributes test cases across difficulty levels:

-   **Easy**: ~30% of cases - Basic, straightforward scenarios
-   **Medium**: ~50% of cases - Standard complexity
-   **Hard**: ~20% of cases - Complex, edge cases

## Supported evaluators

The generator can automatically create rubrics for these default evaluators:

-   `OutputEvaluator`: Evaluates output quality
-   `TrajectoryEvaluator`: Evaluates tool usage sequences
-   `InteractionsEvaluator`: Evaluates conversation interactions

For other evaluators, pass `evaluator=None` or use `Evaluator()` as a placeholder.

## Best practices

### 1\. Provide rich context

```python
# Good: Detailed context
context = """
Agent capabilities:
- Tool 1: search_database(query: str) -> List[Result]
  Returns up to 10 results from knowledge base
- Tool 2: analyze_sentiment(text: str) -> Dict[str, float]
  Returns sentiment scores (positive, negative, neutral)

Agent behavior:
- Always searches before answering
- Cites sources in responses
- Handles "no results" gracefully
"""

# Less effective: Vague context
context = "Agent with search and analysis tools"
```

### 2\. Use topic planning for large datasets

```python
# For 15+ cases, use topic planning
experiment = await generator.from_context_async(
    context=context,
    task_description=task,
    num_cases=20,
    num_topics=4  # Ensures diverse coverage
)
```

### 3\. Iterate and expand

```python
# Start small
initial = await generator.from_context_async(
    context=context,
    task_description=task,
    num_cases=5
)

# Test and refine
# ... run evaluations ...

# Expand based on findings
expanded = await generator.update_current_experiment_async(
    source_experiment=initial,
    task_description=task,
    num_cases=10,
    context="Focus on areas where initial cases showed weaknesses"
)
```

### 4\. Save intermediate results

```python
# Save after each generation step
experiment.to_file(f"experiment_v{version}")
```

## Common patterns

### Pattern 1: Bootstrap evaluation suite

```python
async def bootstrap_evaluation():
    generator = ExperimentGenerator[str, str](str, str)

    experiment = await generator.from_context_async(
        context="Your agent context here",
        task_description="Your task here",
        num_cases=10,
        num_topics=2,
        evaluator=OutputEvaluator
    )

    experiment.to_file("initial_suite")
    return experiment
```

### Pattern 2: Adapt existing experiments

```python
async def adapt_for_new_task():
    source = Experiment.from_file("existing_experiment.json")
    generator = ExperimentGenerator[str, str](str, str)

    adapted = await generator.from_experiment_async(
        source_experiment=source,
        task_description="New task description",
        num_cases=len(source.cases),
        extra_information="New context and tools"
    )

    return adapted
```

### Pattern 3: Incremental expansion

```python
async def expand_incrementally():
    experiment = Experiment.from_file("current.json")
    generator = ExperimentGenerator[str, str](str, str)

    # Add edge cases
    experiment = await generator.update_current_experiment_async(
        source_experiment=experiment,
        task_description="Focus on edge cases",
        num_cases=5,
        context="Error handling, boundary conditions",
        add_new_cases=True,
        add_new_rubric=False
    )

    # Add performance cases
    experiment = await generator.update_current_experiment_async(
        source_experiment=experiment,
        task_description="Focus on performance",
        num_cases=5,
        context="Large inputs, complex queries",
        add_new_cases=True,
        add_new_rubric=False
    )

    return experiment
```

## Troubleshooting

### Issue: Generated cases are too similar

**Solution**: Use topic planning with more topics

```python
experiment = await generator.from_context_async(
    context=context,
    task_description=task,
    num_cases=20,
    num_topics=5  # Increase topic diversity
)
```

### Issue: Cases don’t match expected complexity

**Solution**: Provide more detailed context and examples

```python
context = """
Detailed context with:
- Specific tool descriptions
- Expected behavior patterns
- Example scenarios
- Edge cases to consider
"""
```

### Issue: Rubric generation fails

**Solution**: Use explicit rubric or skip automatic generation

```python
# Option 1: Provide custom rubric
evaluator = OutputEvaluator(rubric="Your custom rubric here")
experiment = Experiment(cases=cases, evaluators=[evaluator])

# Option 2: Generate without evaluator
experiment = await generator.from_context_async(
    context=context,
    task_description=task,
    num_cases=10,
    evaluator=None  # No automatic rubric generation
)
```

## Related documentation

-   [Quickstart Guide](/pr-cms-4519/docs/user-guide/evals-sdk/quickstart/index.md): Get started with Strands Evals
-   [`strands-evals generate`](/pr-cms-4519/docs/user-guide/evals-sdk/cli/generate/index.md): Generate experiments from the command line
-   [Plan topics for coverage](/pr-cms-4519/docs/user-guide/evals-sdk/topic_planning/index.md): Distribute cases across topics with `TopicPlanner`
-   [Output Evaluator](/pr-cms-4519/docs/user-guide/evals-sdk/evaluators/output_evaluator/index.md): Learn about output evaluation
-   [Trajectory Evaluator](/pr-cms-4519/docs/user-guide/evals-sdk/evaluators/trajectory_evaluator/index.md): Understand trajectory evaluation
-   [Dataset Management](/pr-cms-4519/docs/user-guide/evals-sdk/how-to/experiment_management/index.md): Manage and organize datasets
-   [Serialization](/pr-cms-4519/docs/user-guide/evals-sdk/how-to/serialization/index.md): Save and load experiments

## Related pages

- [Plan topics for coverage](/pr-cms-4519/docs/user-guide/evals-sdk/topic_planning/index.md) (1 shared tag)
- [Simulators](/pr-cms-4519/docs/user-guide/evals-sdk/simulators/index.md) (1 shared tag)
- [Customizing user simulation](/pr-cms-4519/docs/user-guide/evals-sdk/simulators/customize_user_simulation/index.md) (1 shared tag)
- [User simulation](/pr-cms-4519/docs/user-guide/evals-sdk/simulators/user_simulation/index.md) (1 shared tag)
- [Chaos testing](/pr-cms-4519/docs/user-guide/evals-sdk/chaos_testing/index.md) (1 shared tag)
- [Tool simulation](/pr-cms-4519/docs/user-guide/evals-sdk/simulators/tool_simulation/index.md) (1 shared tag)
- [Failure communication evaluator](/pr-cms-4519/docs/user-guide/evals-sdk/evaluators/failure_communication_evaluator/index.md) (1 shared tag)
- [Partial completion evaluator](/pr-cms-4519/docs/user-guide/evals-sdk/evaluators/partial_completion_evaluator/index.md) (1 shared tag)
- [Recovery strategy evaluator](/pr-cms-4519/docs/user-guide/evals-sdk/evaluators/recovery_strategy_evaluator/index.md) (1 shared tag)