Experiment generator

Overview

When you need an evaluation suite but do not want to hand-author every case, ExperimentGenerator writes them for you. Give it a description of your agent’s tools and task; it uses an LLM to generate test cases, spreads them across topics and difficulty levels, and can attach a scored rubric for the default evaluators.

Reach for it when you want to:

• Bootstrap an evaluation experiment without writing cases by hand.
• Spread cases across multiple topics or scenarios (see Plan topics for coverage).
• Generate rubrics automatically for the default evaluators.
• Expand an existing experiment with new cases, or adapt one task’s experiment to a similar task.
• Support custom input, output, and trajectory types, and generate cases in parallel.

Basic usage

Simple generation from context

import asyncio
from strands_evals.generators import ExperimentGenerator
from strands_evals.evaluators import OutputEvaluator

# Initialize generator
generator = ExperimentGenerator[str, str](
    input_type=str,
    output_type=str,
    include_expected_output=True
)

# Generate experiment from context
async def generate_experiment():
    experiment = await generator.from_context_async(
        context="""
        Available tools:
        - calculator(expression: str) -> float: Evaluate mathematical expressions
        - current_time() -> str: Get current date and time
        """,
        task_description="Math and time assistant",
        num_cases=5,
        evaluator=OutputEvaluator
    )
    return experiment

# Run generation
experiment = asyncio.run(generate_experiment())
print(f"Generated {len(experiment.cases)} test cases")

To distribute generated cases across diverse, non-overlapping topics (either by passing num_topics or by planning topics explicitly with TopicPlanner), see Plan topics for coverage.

From the command line

The strands-evals generate subcommand wraps the same generator, so you can bootstrap an experiment without writing a script:

# Without topics: 10 cases from a context description
strands-evals generate \
  --context "$(cat tools.txt)" \
  --task-description "Math and time assistant" \
  --num-cases 10 \
  --evaluator OutputEvaluator \
  -o experiments/generated.json

# With topics: spread 15 cases across 3 planned topics
strands-evals generate \
  --context "$(cat tools.txt)" \
  --task-description "Customer service assistant" \
  --num-cases 15 \
  --num-topics 3 \
  --evaluator TrajectoryEvaluator \
  -o experiments/generated.json

The CLI can also expand an existing experiment file via --experiment; see strands-evals generate for the full flag set.

Generation methods

1. From context

Generate experiments based on specific context that test cases should reference:

async def generate_from_context():
    experiment = await generator.from_context_async(
        context="Agent with weather API and location tools",
        task_description="Weather information assistant",
        num_cases=10,
        num_topics=2,  # Optional: distribute across topics
        evaluator=OutputEvaluator
    )
    return experiment

2. From scratch

Generate experiments from topic lists and task descriptions:

async def generate_from_scratch():
    experiment = await generator.from_scratch_async(
        topics=["product search", "order tracking", "returns"],
        task_description="E-commerce customer service",
        num_cases=12,
        evaluator=TrajectoryEvaluator
    )
    return experiment

3. From existing experiment

Create new experiments inspired by existing ones:

async def generate_from_experiment():
    # Load existing experiment
    source_experiment = Experiment.from_file("original_experiment.json")

    # Generate similar experiment for new task
    new_experiment = await generator.from_experiment_async(
        source_experiment=source_experiment,
        task_description="New task with similar structure",
        num_cases=8,
        extra_information="Additional context about tools and capabilities"
    )
    return new_experiment

4. Update existing experiment

Extend experiments with additional test cases:

async def update_experiment():
    source_experiment = Experiment.from_file("current_experiment.json")

    updated_experiment = await generator.update_current_experiment_async(
        source_experiment=source_experiment,
        task_description="Enhanced task description",
        num_cases=5,  # Add 5 new cases
        context="Additional context for new cases",
        add_new_cases=True,
        add_new_rubric=True
    )
    return updated_experiment

Configuration options

Input/output types

Configure the structure of generated test cases:

from typing import Dict, List

# Complex types
generator = ExperimentGenerator[Dict[str, str], List[str]](
    input_type=Dict[str, str],
    output_type=List[str],
    include_expected_output=True,
    include_expected_trajectory=True,
    include_metadata=True
)

Parallel generation

Control concurrent test case generation:

generator = ExperimentGenerator[str, str](
    input_type=str,
    output_type=str,
    max_parallel_num_cases=20  # Generate up to 20 cases in parallel
)

Custom prompts

Customize generation behavior with custom prompts:

generator = ExperimentGenerator[str, str](
    input_type=str,
    output_type=str,
    case_system_prompt="Custom prompt for case generation...",
    rubric_system_prompt="Custom prompt for rubric generation...",
)

Both prompts default to the built-in templates in strands_evals.generators.prompt_template.prompt_templates.

Complete example: Multi-step dataset generation

import asyncio
from strands_evals.generators import ExperimentGenerator
from strands_evals.evaluators import OutputEvaluator, TrajectoryEvaluator

async def build_dataset():
    # Initialize generator with trajectory support
    generator = ExperimentGenerator[str, str](
        input_type=str,
        output_type=str,
        include_expected_output=True,
        include_expected_trajectory=True,
        include_metadata=True
    )

    # Step 1: Generate initial experiment with topic planning
    print("Step 1: Generating initial experiment...")
    experiment = await generator.from_context_async(
        context="""
        Multi-agent system with:
        - Research agent: Searches and analyzes information
        - Writing agent: Creates content and summaries
        - Review agent: Validates and improves outputs

        Tools available:
        - web_search(query: str) -> str
        - summarize(text: str) -> str
        - fact_check(claim: str) -> bool
        """,
        task_description="Research and content creation assistant",
        num_cases=15,
        num_topics=3,  # Research, Writing, Review
        evaluator=TrajectoryEvaluator
    )

    print(f"Generated {len(experiment.cases)} cases across 3 topics")

    # Step 2: Add more cases to expand coverage
    print("\nStep 2: Expanding experiment...")
    expanded_experiment = await generator.update_current_experiment_async(
        source_experiment=experiment,
        task_description="Research and content creation with edge cases",
        num_cases=5,
        context="Focus on error handling and complex multi-step scenarios",
        add_new_cases=True,
        add_new_rubric=False  # Keep existing rubric
    )

    print(f"Expanded to {len(expanded_experiment.cases)} total cases")

    # Step 3: Add a second LLM-judge evaluator built on a generated rubric.
    # construct_evaluator_async only accepts the default evaluator classes
    # (OutputEvaluator, TrajectoryEvaluator, InteractionsEvaluator).
    print("\nStep 3: Adding output-quality evaluator...")
    output_eval = await generator.construct_evaluator_async(
        prompt="Evaluate output quality for research and content creation tasks",
        evaluator=OutputEvaluator
    )
    expanded_experiment.evaluators.append(output_eval)

    # For non-default evaluators (e.g. HelpfulnessEvaluator), instantiate directly:
    # expanded_experiment.evaluators.append(HelpfulnessEvaluator())

    # Step 4: Save experiment
    expanded_experiment.to_file("research_dataset")
    print("\nDataset saved to ./research_dataset.json")

    return expanded_experiment

# Run the multi-step generation
experiment = asyncio.run(build_dataset())

# Examine results
print(f"\nFinal experiment:")
print(f"- Total cases: {len(experiment.cases)}")
print(f"- Evaluators: {len(experiment.evaluators)}")
print(f"- Categories: {set(c.metadata.get('category', 'unknown') for c in experiment.cases if c.metadata)}")

Difficulty levels

The generator automatically distributes test cases across difficulty levels:

• Easy: ~30% of cases - Basic, straightforward scenarios
• Medium: ~50% of cases - Standard complexity
• Hard: ~20% of cases - Complex, edge cases

Supported evaluators

The generator can automatically create rubrics for these default evaluators:

• OutputEvaluator: Evaluates output quality
• TrajectoryEvaluator: Evaluates tool usage sequences
• InteractionsEvaluator: Evaluates conversation interactions

For other evaluators, pass evaluator=None or use Evaluator() as a placeholder.

Best practices

1. Provide rich context

# Good: Detailed context
context = """
Agent capabilities:
- Tool 1: search_database(query: str) -> List[Result]
  Returns up to 10 results from knowledge base
- Tool 2: analyze_sentiment(text: str) -> Dict[str, float]
  Returns sentiment scores (positive, negative, neutral)

Agent behavior:
- Always searches before answering
- Cites sources in responses
- Handles "no results" gracefully
"""

# Less effective: Vague context
context = "Agent with search and analysis tools"

2. Use topic planning for large datasets

# For 15+ cases, use topic planning
experiment = await generator.from_context_async(
    context=context,
    task_description=task,
    num_cases=20,
    num_topics=4  # Ensures diverse coverage
)

3. Iterate and expand

# Start small
initial = await generator.from_context_async(
    context=context,
    task_description=task,
    num_cases=5
)

# Test and refine
# ... run evaluations ...

# Expand based on findings
expanded = await generator.update_current_experiment_async(
    source_experiment=initial,
    task_description=task,
    num_cases=10,
    context="Focus on areas where initial cases showed weaknesses"
)

4. Save intermediate results

# Save after each generation step
experiment.to_file(f"experiment_v{version}")

Common patterns

Pattern 1: Bootstrap evaluation suite

async def bootstrap_evaluation():
    generator = ExperimentGenerator[str, str](str, str)

    experiment = await generator.from_context_async(
        context="Your agent context here",
        task_description="Your task here",
        num_cases=10,
        num_topics=2,
        evaluator=OutputEvaluator
    )

    experiment.to_file("initial_suite")
    return experiment

Pattern 2: Adapt existing experiments

async def adapt_for_new_task():
    source = Experiment.from_file("existing_experiment.json")
    generator = ExperimentGenerator[str, str](str, str)

    adapted = await generator.from_experiment_async(
        source_experiment=source,
        task_description="New task description",
        num_cases=len(source.cases),
        extra_information="New context and tools"
    )

    return adapted

Pattern 3: Incremental expansion

async def expand_incrementally():
    experiment = Experiment.from_file("current.json")
    generator = ExperimentGenerator[str, str](str, str)

    # Add edge cases
    experiment = await generator.update_current_experiment_async(
        source_experiment=experiment,
        task_description="Focus on edge cases",
        num_cases=5,
        context="Error handling, boundary conditions",
        add_new_cases=True,
        add_new_rubric=False
    )

    # Add performance cases
    experiment = await generator.update_current_experiment_async(
        source_experiment=experiment,
        task_description="Focus on performance",
        num_cases=5,
        context="Large inputs, complex queries",
        add_new_cases=True,
        add_new_rubric=False
    )

    return experiment

Troubleshooting

Issue: Generated cases are too similar

Solution: Use topic planning with more topics

experiment = await generator.from_context_async(
    context=context,
    task_description=task,
    num_cases=20,
    num_topics=5  # Increase topic diversity
)

Issue: Cases don’t match expected complexity

Solution: Provide more detailed context and examples

context = """
Detailed context with:
- Specific tool descriptions
- Expected behavior patterns
- Example scenarios
- Edge cases to consider
"""

Issue: Rubric generation fails

Solution: Use explicit rubric or skip automatic generation

# Option 1: Provide custom rubric
evaluator = OutputEvaluator(rubric="Your custom rubric here")
experiment = Experiment(cases=cases, evaluators=[evaluator])

# Option 2: Generate without evaluator
experiment = await generator.from_context_async(
    context=context,
    task_description=task,
    num_cases=10,
    evaluator=None  # No automatic rubric generation
)

Related documentation

• Quickstart Guide: Get started with Strands Evals
• strands-evals generate: Generate experiments from the command line
• Plan topics for coverage: Distribute cases across topics with TopicPlanner
• Output Evaluator: Learn about output evaluation
• Trajectory Evaluator: Understand trajectory evaluation
• Dataset Management: Manage and organize datasets
• Serialization: Save and load experiments

Tags

simulation9