## Overview

The `HelpfulnessEvaluator` evaluates the helpfulness of agent responses from the user’s perspective. It assesses whether responses effectively address user needs, provide useful information, and contribute positively to achieving the user’s goals. A complete example can be found [here](https://github.com/strands-agents/harness-sdk/blob/main/site/docs/examples/evals-sdk/helpfulness_evaluator.py).

## Key Features

-   **Trace-Level Evaluation**: Evaluates the most recent turn in the conversation
-   **User-Centric Assessment**: Focuses on helpfulness from the user’s point of view
-   **Seven-Level Scoring**: Detailed scale from “Not helpful at all” to “Above and beyond”
-   **Structured Reasoning**: Provides step-by-step reasoning for each evaluation
-   **Async Support**: Supports both synchronous and asynchronous evaluation
-   **Context-Aware**: Considers conversation history when evaluating helpfulness

## When to Use

Use the `HelpfulnessEvaluator` when you need to:

-   Assess user satisfaction with agent responses
-   Evaluate if responses effectively address user queries
-   Measure the practical value of agent outputs
-   Compare helpfulness across different agent configurations
-   Identify areas where agents could be more helpful
-   Optimize agent behavior for user experience

## Evaluation Level

This evaluator operates at the **TRACE\_LEVEL**, meaning it evaluates the most recent turn in the conversation (the last agent response and its context).

## Parameters

### `model` (optional)

-   **Type**: `Union[Model, str, None]`
-   **Default**: `None` (uses default Bedrock model)
-   **Description**: The model to use as the judge. Can be a model ID string or a Model instance.

### `system_prompt` (optional)

-   **Type**: `str | None`
-   **Default**: `None` (uses built-in template)
-   **Description**: Custom system prompt to guide the judge model’s behavior.

### `include_inputs` (optional)

-   **Type**: `bool`
-   **Default**: `True`
-   **Description**: Whether to include the input prompt in the evaluation context.

## Scoring System

The evaluator uses a seven-level categorical scoring system:

-   **Not helpful at all (0.0)**: Response is completely unhelpful or counterproductive
-   **Very unhelpful (0.167)**: Response provides minimal or misleading value
-   **Somewhat unhelpful (0.333)**: Response has some issues that limit helpfulness
-   **Neutral/Mixed (0.5)**: Response is adequate but not particularly helpful
-   **Somewhat helpful (0.667)**: Response is useful and addresses the query
-   **Very helpful (0.833)**: Response is highly useful and well-crafted
-   **Above and beyond (1.0)**: Response exceeds expectations with exceptional value

A response passes the evaluation if the score is >= 0.5.

## Basic Usage

Required: Session ID Trace Attributes

When using `StrandsInMemorySessionMapper`, you **must** include session ID trace attributes in your agent configuration. This prevents spans from different test cases from being mixed together in the memory exporter.

```python
import asyncio

from strands import Agent
from strands_evals import Case, Experiment
from strands_evals.evaluators import HelpfulnessEvaluator
from strands_evals.mappers import StrandsInMemorySessionMapper
from strands_evals.telemetry import StrandsEvalsTelemetry

# Setup telemetry
telemetry = StrandsEvalsTelemetry().setup_in_memory_exporter()
memory_exporter = telemetry.in_memory_exporter

# Define task function
def user_task_function(case: Case) -> dict:
    agent = Agent(
        trace_attributes={
            "gen_ai.conversation.id": case.session_id,
            "session.id": case.session_id
        },
        callback_handler=None
    )
    agent_response = agent(case.input)

    # Map spans to session
    finished_spans = memory_exporter.get_finished_spans()
    mapper = StrandsInMemorySessionMapper()
    session = mapper.map_to_session(finished_spans, session_id=case.session_id)

    return {"output": str(agent_response), "trajectory": session}

# Create test cases
test_cases = [
    Case[str, str](
        name="knowledge-1",
        input="What is the capital of France?",
        metadata={"category": "knowledge"}
    ),
    Case[str, str](
        name="knowledge-2",
        input="What color is the ocean?",
        metadata={"category": "knowledge"}
    ),
]

# Create evaluator
evaluator = HelpfulnessEvaluator()

# Run evaluation
experiment = Experiment[str, str](cases=test_cases, evaluators=[evaluator])

async def main():
    report = await experiment.run_evaluations_async(user_task_function)
    report.run_display()

asyncio.run(main())
```

## Evaluation Output

The `HelpfulnessEvaluator` returns `EvaluationOutput` objects with:

-   **score**: Float between 0.0 and 1.0 (0.0, 0.167, 0.333, 0.5, 0.667, 0.833, or 1.0)
-   **test\_pass**: `True` if score >= 0.5, `False` otherwise
-   **reason**: Step-by-step reasoning explaining the evaluation
-   **label**: One of the categorical labels (e.g., “Very helpful”, “Somewhat helpful”)

## What Gets Evaluated

The evaluator examines:

1.  **Previous Turns**: Earlier conversation context (if available)
2.  **Target Turn**: The user’s query and the agent’s response
3.  **Helpfulness Factors**:
    -   Relevance to the user’s query
    -   Completeness of the answer
    -   Clarity and understandability
    -   Actionability of the information
    -   Tone and professionalism

The judge determines how helpful the response is from the user’s perspective.

## Best Practices

1.  **Use with Proper Telemetry Setup**: The evaluator requires trajectory information captured via OpenTelemetry
2.  **Provide User Context**: Include conversation history for context-aware evaluation
3.  **Test Diverse Scenarios**: Include various query types and complexity levels
4.  **Consider Domain-Specific Needs**: Adjust expectations based on your use case
5.  **Combine with Other Evaluators**: Use alongside accuracy and faithfulness evaluators

## Common Patterns

### Pattern 1: Customer Service Evaluation

Assess helpfulness of customer support responses.

### Pattern 2: Information Retrieval

Evaluate if search or query responses meet user needs.

### Pattern 3: Task Assistance

Measure how well agents help users complete tasks.

## Example Scenarios

### Scenario 1: Very Helpful Response

```plaintext
User: "How do I reset my password?"
Agent: "To reset your password: 1) Click 'Forgot Password' on the login page, 2) Enter your email, 3) Check your inbox for a reset link, 4) Follow the link and create a new password. The link expires in 24 hours."
Evaluation: Very helpful (0.833) - Clear, complete, actionable steps
```

### Scenario 2: Somewhat Helpful Response

```plaintext
User: "How do I reset my password?"
Agent: "You can reset your password through the login page."
Evaluation: Somewhat helpful (0.667) - Correct but lacks detail
```

### Scenario 3: Not Helpful Response

```plaintext
User: "How do I reset my password?"
Agent: "Passwords are important for security."
Evaluation: Not helpful at all (0.0) - Doesn't address the question
```

### Scenario 4: Above and Beyond Response

```plaintext
User: "How do I reset my password?"
Agent: "I'll help you reset your password. Here are the steps: 1) Click 'Forgot Password' on the login page, 2) Enter your email, 3) Check your inbox for a reset link (check spam if not found), 4) Follow the link and create a new password. Tips: Use a strong password with 12+ characters, mix of letters/numbers/symbols. If you don't receive the email within 5 minutes, let me know and I can help troubleshoot."
Evaluation: Above and beyond (1.0) - Comprehensive, proactive, anticipates issues
```

## Common Issues and Solutions

### Issue 1: No Evaluation Returned

**Problem**: Evaluator returns empty results. **Solution**: Ensure trajectory contains at least one agent invocation span.

### Issue 2: Inconsistent Scoring

**Problem**: Similar responses get different scores. **Solution**: This is expected due to LLM non-determinism. Run multiple evaluations and aggregate.

### Issue 3: Context Not Considered

**Problem**: Evaluation doesn’t account for conversation history. **Solution**: Verify telemetry captures full conversation and `include_inputs=True`.

## Differences from Other Evaluators

-   **vs. FaithfulnessEvaluator**: Helpfulness focuses on user value, faithfulness on factual grounding
-   **vs. OutputEvaluator**: Helpfulness is user-centric, output evaluator uses custom rubrics
-   **vs. GoalSuccessRateEvaluator**: Helpfulness evaluates individual turns, goal success evaluates overall achievement

## Related Evaluators

-   [**FaithfulnessEvaluator**](/pr-cms-3708/docs/user-guide/evals-sdk/evaluators/faithfulness_evaluator/index.md): Evaluates if responses are grounded in context
-   [**OutputEvaluator**](/pr-cms-3708/docs/user-guide/evals-sdk/evaluators/output_evaluator/index.md): Evaluates overall output quality with custom criteria
-   [**GoalSuccessRateEvaluator**](/pr-cms-3708/docs/user-guide/evals-sdk/evaluators/goal_success_rate_evaluator/index.md): Evaluates if overall user goals were achieved
-   [**TrajectoryEvaluator**](/pr-cms-3708/docs/user-guide/evals-sdk/evaluators/trajectory_evaluator/index.md): Evaluates the sequence of actions taken

## Related pages

- [Coherence Evaluator](/pr-cms-3708/docs/user-guide/evals-sdk/evaluators/coherence_evaluator/index.md) (1 shared tag)
- [Conciseness Evaluator](/pr-cms-3708/docs/user-guide/evals-sdk/evaluators/conciseness_evaluator/index.md) (1 shared tag)
- [Goal Success Rate Evaluator](/pr-cms-3708/docs/user-guide/evals-sdk/evaluators/goal_success_rate_evaluator/index.md) (1 shared tag)
- [Interactions Evaluator](/pr-cms-3708/docs/user-guide/evals-sdk/evaluators/interactions_evaluator/index.md) (1 shared tag)
- [Output Evaluator](/pr-cms-3708/docs/user-guide/evals-sdk/evaluators/output_evaluator/index.md) (1 shared tag)
- [Trusted Message History](/pr-cms-3708/docs/user-guide/safety-security/trusted-message-history/index.md) (1 shared tag)
- [User Simulation](/pr-cms-3708/docs/user-guide/evals-sdk/simulators/user_simulation/index.md) (1 shared tag)
- [Context Management](/pr-cms-3708/docs/user-guide/concepts/context-management/index.md) (1 shared tag)
- [Prompt Engineering](/pr-cms-3708/docs/user-guide/safety-security/prompt-engineering/index.md) (1 shared tag)
- [Prompts](/pr-cms-3708/docs/user-guide/concepts/agents/prompts/index.md) (1 shared tag)