## Overview

Point an AI assistant at your agent and it runs the whole evaluation for you: it plans the evaluation, generates test data, scores the agent with the Strands Evals SDK, and writes a report. This is the assistant driving the entire workflow, not an evaluator that happens to use an LLM as its judge. The workflow ships as the Eval SOP, an [Agent SOP](https://github.com/strands-agents/agent-sop): a markdown standard for encoding agent workflows as natural-language instructions with parameterized inputs and RFC 2119 (MUST, SHOULD, MAY) constraints.

Designing agent evaluations by hand is slow and easy to get wrong. Choosing the right evaluators, covering edge cases and failure modes, writing the SDK code, and keeping results consistent across a team all take real effort. Eval SOP structures that work into four phases (Plan, Data, Eval, Report) that run the same way across different AI assistants, and distributes through MCP servers, Anthropic Skills, or direct integration with a Strands agent.

## Installation and setup

### Install strands-agents-sops

```bash
# Using pip
pip install strands-agents-sops

# Or using Homebrew
brew install strands-agents-sops
```

### Setup evaluation project

Create a self-contained evaluation workspace:

```bash
mkdir agent-evaluation-project
cd agent-evaluation-project

# Copy your agent to evaluate (must be self-contained)
cp -r /path/to/your/agent .
```

Expected structure:

```plaintext
agent-evaluation-project/
├── your-agent/           # Agent to evaluate
├── evals-main/          # Strands Evals SDK (optional)
└── eval/                # Generated evaluation artifacts
    ├── eval-plan.md
    ├── test-cases.jsonl
    ├── results/
    ├── run_evaluation.py
    └── eval-report.md
```

## Usage options

### Option 1: MCP integration (recommended)

Set up MCP server for AI assistant integration:

```bash
# Download Eval SOP
mkdir ~/my-sops
# Copy eval.sop.md to ~/my-sops/

# Configure MCP server
strands-agents-sops mcp --sop-paths ~/my-sops
```

Add to your AI assistant’s MCP configuration:

```json
{
  "mcpServers": {
    "Eval": {
      "command": "strands-agents-sops",
      "args": ["mcp", "--sop-paths", "~/my-sops"]
    }
  }
}
```

#### Usage with Claude Code

```bash
cd agent-evaluation-project
claude

# In Claude session:
 /my-sops:eval (MCP) generate an evaluation plan for this agent at ./your-agent using strands evals sdk at ./evals-main
```

The workflow proceeds through four phases:

1.  **Planning**: `/Eval generate an evaluation plan`
2.  **Data Generation**: `yes` (when prompted) or `/Eval generate the test data`
3.  **Evaluation**: `yes` (when prompted) or `/Eval evaluate the agent using strands evals`
4.  **Reporting**: `/Eval generate an evaluation report based on /path/to/results.json`

### Option 2: Direct Strands agent integration

```python
from strands import Agent
from strands.vended_tools import file_editor, shell
from strands_agents_sops import eval

agent = Agent(
    system_prompt=eval,
    tools=[file_editor, shell],
)

# Initial message to start the evaluation
agent("Start Eval sop for evaluating my QA agent")

# Multi-turn conversation loop
while True:
    user_input = input("\nYou: ")
    if user_input.lower() in ("exit", "quit", "done"):
        print("Evaluation session ended.")
        break

    agent(user_input)
```

You can bypass tool consent when running Eval SOP by setting the following environment variable:

```python
import os

os.environ["BYPASS_TOOL_CONSENT"] = "true"
```

### Option 3: Anthropic skills

Convert to Claude Skills format:

```bash
strands-agents-sops skills --sop-paths ~/my-sops --output-dir ./skills
```

Upload the generated `skills/eval/SKILL.md` to Claude.ai or use via Claude API.

## Evaluation workflow

### Phase 1: Planning

Eval analyzes your agent and creates an evaluation plan:

-   **Architecture Analysis**: Examines agent code, tools, and capabilities
-   **Use Case Identification**: Determines primary and secondary use cases
-   **Evaluator Selection**: Recommends appropriate evaluators (output, trajectory, helpfulness)
-   **Success Criteria**: Defines measurable success metrics
-   **Risk Assessment**: Identifies potential failure modes and edge cases

**Output**: `eval/eval-plan.md` with structured evaluation methodology

### Phase 2: Test data generation

Creates diverse test cases:

-   **Scenario Coverage**: Generates tests for normal operation, edge cases, and failure modes
-   **Difficulty Gradation**: Creates tests ranging from simple to complex scenarios
-   **Domain Relevance**: Ensures test cases match your agent’s intended use cases
-   **Bias Mitigation**: Generates diverse inputs to avoid evaluation bias

**Output**: `eval/test-cases.jsonl` with structured test cases

### Phase 3: Evaluation execution

Implements and runs the evaluations:

-   **Script Generation**: Creates evaluation scripts using Strands Evaluation SDK best practices
-   **Evaluator Configuration**: Properly configures evaluators with appropriate rubrics and parameters
-   **Execution Management**: Handles evaluation execution with error recovery
-   **Results Collection**: Aggregates results across all test cases and evaluators

**Output**: `eval/results/` directory with detailed evaluation data

### Phase 4: Reporting

Generates insights and recommendations:

-   **Performance Analysis**: Analyzes results across different dimensions and scenarios
-   **Failure Pattern Identification**: Identifies common failure modes and their causes
-   **Improvement Recommendations**: Provides specific, actionable suggestions for agent enhancement
-   **Stakeholder Communication**: Creates reports suitable for different audiences

**Output**: `eval/eval-report.md` with analysis and recommendations

## Example output

### Generated evaluation plan

The evaluation plan follows a structured format with analysis and implementation guidance:

```markdown
# Evaluation Plan for QA+Search Agent

## 1. Evaluation Requirements
- **User Input:** "generate an evaluation plan for this qa agent..."
- **Interpreted Evaluation Requirements:** Evaluate the QA agent's ability to answer questions using web search capabilities...

## 2. Agent Analysis
| **Attribute**         | **Details**                                                 |
| :-------------------- | :---------------------------------------------------------- |
| **Agent Name**        | QA+Search                                                   |
| **Purpose**           | Answer questions by searching the web using Tavily API... |
| **Core Capabilities** | Web search integration, information synthesis...            |

**Agent Architecture Diagram:**
(Mermaid diagram showing User Query → Agent → WebSearchTool → Tavily API flow)

## 3. Evaluation Metrics
### Answer Quality Score
- **Evaluation Area:** Final response quality
- **Method:** LLM-as-Judge (using OutputEvaluator with custom rubric)
- **Scoring Scale:** 0.0 to 1.0
- **Pass Threshold:** 0.75 or higher

## 4. Test Data Generation
- **Simple Factual Questions**: Questions requiring basic web search...
- **Multi-Step Reasoning Questions**: Questions requiring synthesis...

## 5. Evaluation Implementation Design
### 5.1 Evaluation Code Structure
./                           # Repository root directory
├── requirements.txt         # Consolidated dependencies
└── eval/                    # Evaluation workspace
    ├── README.md            # Running instructions
    ├── run_evaluation.py    # Strands Evals SDK implementation
    └── results/             # Evaluation outputs

## 6. Progress Tracking
### 6.1 User Requirements Log
| **Timestamp** | **Source** | **Requirement** |
| :------------ | :--------- | :-------------- |
| 2025-12-01    | eval sop    | Generate evaluation plan... |
```

### Generated test cases

Test cases are generated in JSONL format with structured metadata:

```json
{
  "name": "factual-question-1",
  "input": "What is the capital of France?",
  "expected_output": "The capital of France is Paris.",
  "metadata": {"category": "factual", "difficulty": "easy"}
}
```

### Generated evaluation report

The evaluation report provides analysis and specific recommendations:

```markdown
# Agent Evaluation Report for QA+Search Agent

## Executive Summary
- **Test Scale**: 2 test cases
- **Success Rate**: 100%
- **Overall Score**: 1.000 (Perfect)
- **Status**: Excellent
- **Action Priority**: Continue monitoring; consider expanding test coverage...

## Evaluation Results
### Test Case Coverage
- **Simple Factual Questions (Geography)**: Questions requiring basic factual information...
- **Simple Factual Questions (Sports/Time-sensitive)**: Questions requiring current event information...

### Results
| **Metric**              | **Score** | **Target** | **Status** |
| :---------------------- | :-------- | :--------- | :--------- |
| Answer Quality Score    | 1.00      | 0.75+      | Pass      |
| Overall Test Pass Rate  | 100%      | 75%+       | Pass      |

## Agent Success Analysis
### Strengths
- **Perfect Accuracy**: The agent correctly answered 100% of test questions...
- **Evidence**: Both test cases scored 1.0/1.0 (perfect scores)
- **Contributing Factors**: Effective use of web search tool...

## Agent Failure Analysis
### No Failures Detected
The evaluation identified zero failures across all test cases...

## Action Items & Recommendations
### Expand Test Coverage - Priority 1 (Enhancement)
- **Description**: Increase the number and diversity of test cases...
- **Actions**:
  - [ ] Add 5-10 additional test cases covering edge cases
  - [ ] Include multi-step reasoning scenarios
  - [ ] Add test cases for error conditions

## Artifacts & Reproduction
### Reference Materials
- **Agent Code**: `qa_agent/qa_agent.py`
- **Test Cases**: `eval/test-cases.jsonl`
- **Results**: `eval/results/.../evaluation_report.json`

### Reproduction Steps
source .venv/bin/activate
python eval/run_evaluation.py

## Evaluation Limitations and Improvement
### Test Data Improvement
- **Current Limitations**: Only 2 test cases, limited scenario diversity...
- **Recommended Improvements**: Increase test case count to 10-20 cases...
```

## Best practices

### Evaluation design

-   **Start Simple**: Begin with basic functionality before testing edge cases
-   **Iterate Frequently**: Run evaluations regularly during development
-   **Document Assumptions**: Clearly document evaluation rationale and limitations
-   **Validate Results**: Manually review a sample of evaluation results for accuracy

### Agent preparation

-   **Self-Contained Code**: Ensure your agent directory has no external dependencies
-   **Tool Dependencies**: Document all required tools and their purposes

### Result interpretation

-   **Statistical Significance**: Consider running multiple evaluation rounds for reliability
-   **Failure Analysis**: Focus on understanding why failures occur, not just counting them
-   **Comparative Analysis**: Compare results across different agent configurations
-   **Stakeholder Alignment**: Ensure evaluation metrics align with business objectives

## Troubleshooting

### Common issues

**Issue**: “Agent directory not found” **Solution**: Ensure agent path is correct and directory is self-contained

**Issue**: “Evaluation script fails to run” **Solution**: Check that all dependencies are installed and agent code is valid

**Issue**: “Poor test case quality” **Solution**: Provide more detailed agent documentation and example usage

**Issue**: “Inconsistent evaluation results” **Solution**: Review evaluator configurations and consider multiple evaluation runs

### Getting help

-   **Agent SOP Repository**: [https://github.com/strands-agents/agent-sop](https://github.com/strands-agents/agent-sop)
-   **Strands Eval SDK**: [Eval SDK Documentation](/pr-cms-4519/docs/user-guide/evals-sdk/quickstart/index.md)

## Related tools

-   [**Strands Evaluation SDK**](/pr-cms-4519/docs/user-guide/evals-sdk/quickstart/index.md): Core evaluation framework and evaluators
-   [**Experiment Generator**](/pr-cms-4519/docs/user-guide/evals-sdk/experiment_generator/index.md): Automated test case generation
-   [**Output Evaluator**](/pr-cms-4519/docs/user-guide/evals-sdk/evaluators/output_evaluator/index.md): Custom rubric-based evaluation
-   [**Trajectory Evaluator**](/pr-cms-4519/docs/user-guide/evals-sdk/evaluators/trajectory_evaluator/index.md): Tool usage and sequence analysis
-   [**Agent SOP Repository**](https://github.com/strands-agents/agent-sop): Standard operating procedures for AI agents