Evaluate with AI

Overview

Point an AI assistant at your agent and it runs the whole evaluation for you: it plans the evaluation, generates test data, scores the agent with the Strands Evals SDK, and writes a report. This is the assistant driving the entire workflow, not an evaluator that happens to use an LLM as its judge. The workflow ships as the Eval SOP, an Agent SOP: a markdown standard for encoding agent workflows as natural-language instructions with parameterized inputs and RFC 2119 (MUST, SHOULD, MAY) constraints.

Designing agent evaluations by hand is slow and easy to get wrong. Choosing the right evaluators, covering edge cases and failure modes, writing the SDK code, and keeping results consistent across a team all take real effort. Eval SOP structures that work into four phases (Plan, Data, Eval, Report) that run the same way across different AI assistants, and distributes through MCP servers, Anthropic Skills, or direct integration with a Strands agent.

Installation and setup

Install strands-agents-sops

# Using pip
pip install strands-agents-sops

# Or using Homebrew
brew install strands-agents-sops

Setup evaluation project

Create a self-contained evaluation workspace:

mkdir agent-evaluation-project
cd agent-evaluation-project

# Copy your agent to evaluate (must be self-contained)
cp -r /path/to/your/agent .

Expected structure:

agent-evaluation-project/
├── your-agent/           # Agent to evaluate
├── evals-main/          # Strands Evals SDK (optional)
└── eval/                # Generated evaluation artifacts
    ├── eval-plan.md
    ├── test-cases.jsonl
    ├── results/
    ├── run_evaluation.py
    └── eval-report.md

Usage options

Option 1: MCP integration (recommended)

Set up MCP server for AI assistant integration:

# Download Eval SOP
mkdir ~/my-sops
# Copy eval.sop.md to ~/my-sops/

# Configure MCP server
strands-agents-sops mcp --sop-paths ~/my-sops

Add to your AI assistant’s MCP configuration:

{
  "mcpServers": {
    "Eval": {
      "command": "strands-agents-sops",
      "args": ["mcp", "--sop-paths", "~/my-sops"]
    }
  }
}

Usage with Claude Code

cd agent-evaluation-project
claude

# In Claude session:
 /my-sops:eval (MCP) generate an evaluation plan for this agent at ./your-agent using strands evals sdk at ./evals-main

The workflow proceeds through four phases:

1. Planning: /Eval generate an evaluation plan
2. Data Generation: yes (when prompted) or /Eval generate the test data
3. Evaluation: yes (when prompted) or /Eval evaluate the agent using strands evals
4. Reporting: /Eval generate an evaluation report based on /path/to/results.json

Option 2: Direct Strands agent integration

from strands import Agent
from strands.vended_tools import file_editor, shell
from strands_agents_sops import eval

agent = Agent(
    system_prompt=eval,
    tools=[file_editor, shell],
)

# Initial message to start the evaluation
agent("Start Eval sop for evaluating my QA agent")

# Multi-turn conversation loop
while True:
    user_input = input("\nYou: ")
    if user_input.lower() in ("exit", "quit", "done"):
        print("Evaluation session ended.")
        break

    agent(user_input)

You can bypass tool consent when running Eval SOP by setting the following environment variable:

import os

os.environ["BYPASS_TOOL_CONSENT"] = "true"

Option 3: Anthropic skills

Convert to Claude Skills format:

strands-agents-sops skills --sop-paths ~/my-sops --output-dir ./skills

Upload the generated skills/eval/SKILL.md to Claude.ai or use via Claude API.

Evaluation workflow

Phase 1: Planning

Eval analyzes your agent and creates an evaluation plan:

• Architecture Analysis: Examines agent code, tools, and capabilities
• Use Case Identification: Determines primary and secondary use cases
• Evaluator Selection: Recommends appropriate evaluators (output, trajectory, helpfulness)
• Success Criteria: Defines measurable success metrics
• Risk Assessment: Identifies potential failure modes and edge cases

Output: eval/eval-plan.md with structured evaluation methodology

Phase 2: Test data generation

Creates diverse test cases:

• Scenario Coverage: Generates tests for normal operation, edge cases, and failure modes
• Difficulty Gradation: Creates tests ranging from simple to complex scenarios
• Domain Relevance: Ensures test cases match your agent’s intended use cases
• Bias Mitigation: Generates diverse inputs to avoid evaluation bias

Output: eval/test-cases.jsonl with structured test cases

Phase 3: Evaluation execution

Implements and runs the evaluations:

• Script Generation: Creates evaluation scripts using Strands Evaluation SDK best practices
• Evaluator Configuration: Properly configures evaluators with appropriate rubrics and parameters
• Execution Management: Handles evaluation execution with error recovery
• Results Collection: Aggregates results across all test cases and evaluators

Output: eval/results/ directory with detailed evaluation data

Phase 4: Reporting

Generates insights and recommendations:

• Performance Analysis: Analyzes results across different dimensions and scenarios
• Failure Pattern Identification: Identifies common failure modes and their causes
• Improvement Recommendations: Provides specific, actionable suggestions for agent enhancement
• Stakeholder Communication: Creates reports suitable for different audiences

Output: eval/eval-report.md with analysis and recommendations

Example output

Generated evaluation plan

The evaluation plan follows a structured format with analysis and implementation guidance:

# Evaluation Plan for QA+Search Agent

## 1. Evaluation Requirements
- **User Input:** "generate an evaluation plan for this qa agent..."
- **Interpreted Evaluation Requirements:** Evaluate the QA agent's ability to answer questions using web search capabilities...

## 2. Agent Analysis
| **Attribute**         | **Details**                                                 |
| :-------------------- | :---------------------------------------------------------- |
| **Agent Name**        | QA+Search                                                   |
| **Purpose**           | Answer questions by searching the web using Tavily API... |
| **Core Capabilities** | Web search integration, information synthesis...            |

**Agent Architecture Diagram:**
(Mermaid diagram showing User Query → Agent → WebSearchTool → Tavily API flow)

## 3. Evaluation Metrics
### Answer Quality Score
- **Evaluation Area:** Final response quality
- **Method:** LLM-as-Judge (using OutputEvaluator with custom rubric)
- **Scoring Scale:** 0.0 to 1.0
- **Pass Threshold:** 0.75 or higher

## 4. Test Data Generation
- **Simple Factual Questions**: Questions requiring basic web search...
- **Multi-Step Reasoning Questions**: Questions requiring synthesis...

## 5. Evaluation Implementation Design
### 5.1 Evaluation Code Structure
./                           # Repository root directory
├── requirements.txt         # Consolidated dependencies
└── eval/                    # Evaluation workspace
    ├── README.md            # Running instructions
    ├── run_evaluation.py    # Strands Evals SDK implementation
    └── results/             # Evaluation outputs

## 6. Progress Tracking
### 6.1 User Requirements Log
| **Timestamp** | **Source** | **Requirement** |
| :------------ | :--------- | :-------------- |
| 2025-12-01    | eval sop    | Generate evaluation plan... |

Generated test cases

Test cases are generated in JSONL format with structured metadata:

{
  "name": "factual-question-1",
  "input": "What is the capital of France?",
  "expected_output": "The capital of France is Paris.",
  "metadata": {"category": "factual", "difficulty": "easy"}
}

Generated evaluation report

The evaluation report provides analysis and specific recommendations:

# Agent Evaluation Report for QA+Search Agent

## Executive Summary
- **Test Scale**: 2 test cases
- **Success Rate**: 100%
- **Overall Score**: 1.000 (Perfect)
- **Status**: Excellent
- **Action Priority**: Continue monitoring; consider expanding test coverage...

## Evaluation Results
### Test Case Coverage
- **Simple Factual Questions (Geography)**: Questions requiring basic factual information...
- **Simple Factual Questions (Sports/Time-sensitive)**: Questions requiring current event information...

### Results
| **Metric**              | **Score** | **Target** | **Status** |
| :---------------------- | :-------- | :--------- | :--------- |
| Answer Quality Score    | 1.00      | 0.75+      | Pass      |
| Overall Test Pass Rate  | 100%      | 75%+       | Pass      |

## Agent Success Analysis
### Strengths
- **Perfect Accuracy**: The agent correctly answered 100% of test questions...
- **Evidence**: Both test cases scored 1.0/1.0 (perfect scores)
- **Contributing Factors**: Effective use of web search tool...

## Agent Failure Analysis
### No Failures Detected
The evaluation identified zero failures across all test cases...

## Action Items & Recommendations
### Expand Test Coverage - Priority 1 (Enhancement)
- **Description**: Increase the number and diversity of test cases...
- **Actions**:
  - [ ] Add 5-10 additional test cases covering edge cases
  - [ ] Include multi-step reasoning scenarios
  - [ ] Add test cases for error conditions

## Artifacts & Reproduction
### Reference Materials
- **Agent Code**: `qa_agent/qa_agent.py`
- **Test Cases**: `eval/test-cases.jsonl`
- **Results**: `eval/results/.../evaluation_report.json`

### Reproduction Steps
source .venv/bin/activate
python eval/run_evaluation.py

## Evaluation Limitations and Improvement
### Test Data Improvement
- **Current Limitations**: Only 2 test cases, limited scenario diversity...
- **Recommended Improvements**: Increase test case count to 10-20 cases...

Best practices

Evaluation design

• Start Simple: Begin with basic functionality before testing edge cases
• Iterate Frequently: Run evaluations regularly during development
• Document Assumptions: Clearly document evaluation rationale and limitations
• Validate Results: Manually review a sample of evaluation results for accuracy

Agent preparation

• Self-Contained Code: Ensure your agent directory has no external dependencies
• Tool Dependencies: Document all required tools and their purposes

Result interpretation

• Statistical Significance: Consider running multiple evaluation rounds for reliability
• Failure Analysis: Focus on understanding why failures occur, not just counting them
• Comparative Analysis: Compare results across different agent configurations
• Stakeholder Alignment: Ensure evaluation metrics align with business objectives

Troubleshooting

Common issues

Issue: “Agent directory not found” Solution: Ensure agent path is correct and directory is self-contained

Issue: “Evaluation script fails to run” Solution: Check that all dependencies are installed and agent code is valid

Issue: “Poor test case quality” Solution: Provide more detailed agent documentation and example usage

Issue: “Inconsistent evaluation results” Solution: Review evaluator configurations and consider multiple evaluation runs

Getting help

• Agent SOP Repository: https://github.com/strands-agents/agent-sop
• Strands Eval SDK: Eval SDK Documentation

Related tools

• Strands Evaluation SDK: Core evaluation framework and evaluators
• Experiment Generator: Automated test case generation
• Output Evaluator: Custom rubric-based evaluation
• Trajectory Evaluator: Tool usage and sequence analysis
• Agent SOP Repository: Standard operating procedures for AI agents