## Overview

The `FailureCommunicationEvaluator` assesses how well an agent communicates failures to the user when tools or services fail. It uses an LLM-as-judge approach with a five-level scoring rubric to evaluate clarity, actionability, transparency, and tone of failure messages. A complete example can be found [here](https://github.com/strands-agents/harness-sdk/blob/main/site/docs/examples/evals-sdk/chaos_failure_communication_evaluator.py).

## Key features

-   **Trace-Level Evaluation**: Evaluates the full conversation trace including tool call results and agent responses
-   **Five-Level Scoring**: Granular scale from “Failure” to “Excellent”
-   **Multi-Dimensional Assessment**: Evaluates clarity, actionability, transparency, and tone
-   **Structured Reasoning**: Provides step-by-step reasoning for each evaluation
-   **Async Support**: Supports both synchronous and asynchronous evaluation

## When to use

Use the `FailureCommunicationEvaluator` when you need to:

-   Assess whether agents inform users about tool failures
-   Evaluate the quality and helpfulness of error messages
-   Test agent transparency under degraded conditions
-   Measure user trust maintenance during failures
-   Compare failure communication across agent configurations

## Evaluation level

This evaluator operates at the **TRACE\_LEVEL**, evaluating the full conversation trace including tool call results and agent responses.

## Parameters

### `version` (optional)

-   **Type**: `str`
-   **Default**: `"v0"`
-   **Description**: Prompt template version for the judge’s system prompt.

### `model` (optional)

-   **Type**: `Model | str | None`
-   **Default**: `None` (uses the default Bedrock model)
-   **Description**: The model to use as the judge.

### `system_prompt` (optional)

-   **Type**: `str | None`
-   **Default**: `None` (uses the built-in prompt for the selected `version`)
-   **Description**: Overrides the judge’s system prompt.

### `name` (optional)

-   **Type**: `str | None`
-   **Default**: `None`
-   **Description**: Custom evaluator name shown in the evaluation report.

## Scoring system

| Rating | Score | Description |
| --- | --- | --- |
| Failure | 0.0 | Agent silently ignores failures, fabricates data, or crashes |
| Poor | 0.25 | Agent vaguely acknowledges an issue without useful information |
| Acceptable | 0.5 | Mixed communication, or no failures occurred to communicate |
| Good | 0.75 | Agent clearly explains the failure and suggests next steps |
| Excellent | 1.0 | Agent transparently explains what failed, why, and provides actionable alternatives |

A response passes the evaluation if the score is >= 0.5.

When no tool failures occur during the session, the evaluator produces a neutral score of 0.5, since there are no failures to assess communication quality against.

## Basic usage

```python
import asyncio
from typing import Any

from pydantic import BaseModel, Field

from strands import Agent
from strands_evals.chaos import ChaosCase, ChaosExperiment, ChaosPlugin, Timeout, NetworkError
from strands_evals.evaluators.chaos import FailureCommunicationEvaluator
from strands_evals.eval_task_handler import TracedHandler, eval_task
from strands_evals.simulation import ToolSimulator

tool_simulator = ToolSimulator()

class FlightSearchResponse(BaseModel):
    flights: list[dict[str, Any]] = Field(default_factory=list)
    status: str = Field(default="success")

@tool_simulator.tool(output_schema=FlightSearchResponse)
def search_flights(origin: str, destination: str, date: str) -> dict[str, Any]:
    """Search for available flights between two cities on a given date."""
    pass

chaos_plugin = ChaosPlugin()
_search_tool = tool_simulator.get_tool("search_flights")

chaos_cases = [
    ChaosCase(
        name="search_timeout",
        input="Find me a flight from SFO to JFK on May 20.",
        effects={"tool_effects": {"search_flights": [Timeout(error_message="Tool call timed out after 30s")]}},
    ),
    ChaosCase(
        name="all_tools_down",
        input="Search for flights from Seattle to Tokyo next Tuesday.",
        effects={"tool_effects": {"search_flights": [NetworkError(error_message="DNS resolution failed")]}},
    ),
]

@eval_task(TracedHandler())
def task_function(case: ChaosCase):
    return Agent(
        system_prompt="You are a travel booking assistant.",
        tools=[_search_tool],
        plugins=[chaos_plugin],
        callback_handler=None,
        trace_attributes={"session.id": case.session_id},
    )

experiment = ChaosExperiment(
    cases=chaos_cases,
    evaluators=[FailureCommunicationEvaluator()],
)

async def main():
    report = await experiment.run_evaluations_async(task=task_function, max_workers=10)
    report.run_display()

asyncio.run(main())
```

## Evaluation output

The `FailureCommunicationEvaluator` returns `EvaluationOutput` objects with:

-   **score**: Float (0.0, 0.25, 0.5, 0.75, or 1.0)
-   **test\_pass**: `True` if score >= 0.5, `False` otherwise
-   **reason**: Step-by-step reasoning explaining the evaluation
-   **label**: One of the categorical labels (e.g., “Good”, “Excellent”)

## What gets evaluated

The evaluator examines:

1.  **Tool Call Results**: Whether tools returned errors or failures
2.  **Agent Response**: How the agent communicated those failures to the user
3.  **Communication Quality**:
    -   Does the agent acknowledge the failure clearly?
    -   Does it suggest actionable next steps?
    -   Is it transparent about what went wrong (without exposing internals)?
    -   Is the tone appropriate (not dismissive, not alarming)?

## Best practices

1.  **Include Tool Failures in Test Cases**: The evaluator needs tool failures in the trace to assess communication quality
2.  **Capture Complete Sessions**: Include all conversation turns and tool call results in the trajectory
3.  **Test Various Failure Types**: Include timeouts, network errors, and validation errors
4.  **Combine with Other Evaluators**: Use alongside `RecoveryStrategyEvaluator` and `PartialCompletionEvaluator`
5.  **Provide System Prompts with Failure Guidance**: Agents with explicit failure-handling instructions tend to score higher

## Common patterns

### Pattern 1: Single tool failure

Evaluate how the agent communicates a single tool timeout or error.

### Pattern 2: Multiple tool failures

Assess communication quality when several tools fail simultaneously.

### Pattern 3: Graceful degradation

Measure how the agent explains partial results when some tools succeed and others fail.

## Example scenarios

### Scenario 1: Excellent communication

```plaintext
Tool: search_flights -> Timeout
Agent: "I'm sorry, but the flight search service is currently experiencing delays
and couldn't complete your request. You could try again in a few minutes, or I
can help you explore alternative options like checking a different travel date."
Evaluation: Excellent (1.0) - Clear, transparent, actionable
```

### Scenario 2: Good communication

```plaintext
Tool: search_flights -> NetworkError
Agent: "I wasn't able to search for flights right now due to a connection issue.
Please try again shortly."
Evaluation: Good (0.75) - Acknowledges failure, suggests retry
```

### Scenario 3: No communication

```plaintext
Tool: search_flights -> Timeout
Agent: "There are no flights available for that route."
Evaluation: Failure (0.0) - Fabricates results instead of reporting failure
```

## Common issues and solutions

### Issue 1: Score is always 0.5

**Problem**: Evaluator always returns neutral score. **Solution**: Ensure tool failures are actually present in the trace. If no tools fail, the evaluator returns 0.5 by design.

### Issue 2: Agent not detecting failures

**Problem**: Agent doesn’t mention failures in its response. **Solution**: Add failure-handling instructions to the system prompt (e.g., “If a tool fails, acknowledge the failure honestly”).

### Issue 3: No trajectory data

**Problem**: Evaluator returns empty results. **Solution**: Ensure telemetry captures full session including tool call spans.

## Differences from other evaluators

-   **vs. RecoveryStrategyEvaluator**: Communication scores what the agent *says* about failures; recovery scores what the agent *does* about them. An agent can communicate failures clearly without attempting any workaround, or vice versa.
-   **vs. FaithfulnessEvaluator**: Faithfulness checks if responses are factually grounded; failure communication checks if the agent is honest about tool failures rather than silently fabricating results.
-   **vs. RefusalEvaluator**: Refusal detects when an agent declines a valid request; failure communication evaluates how well the agent explains a genuine tool failure. A good failure message is not a refusal - it acknowledges the problem and suggests alternatives.
-   **vs. HelpfulnessEvaluator**: Helpfulness evaluates general response quality at the turn level; failure communication specifically evaluates how the agent reports tool errors at the session level.

## Use cases

### Use case 1: Customer-facing agents

Ensure agents inform users clearly when backend services are down.

### Use case 2: Chaos testing

Evaluate agent transparency under deliberately injected tool failures.

### Use case 3: Trust assessment

Measure whether agents maintain user trust during degraded conditions.

### Use case 4: Error message quality

Compare failure communication across different system prompt configurations.

## Related evaluators

-   [**RecoveryStrategyEvaluator**](/pr-cms-4519/docs/user-guide/evals-sdk/evaluators/recovery_strategy_evaluator/index.md): Evaluates quality of recovery actions
-   [**PartialCompletionEvaluator**](/pr-cms-4519/docs/user-guide/evals-sdk/evaluators/partial_completion_evaluator/index.md): Measures what fraction of goals were achieved despite failures
-   [**FaithfulnessEvaluator**](/pr-cms-4519/docs/user-guide/evals-sdk/evaluators/faithfulness_evaluator/index.md): Evaluates if responses are factually grounded
-   [**RefusalEvaluator**](/pr-cms-4519/docs/user-guide/evals-sdk/evaluators/refusal_evaluator/index.md): Detects when agents inappropriately refuse valid requests
-   [**GoalSuccessRateEvaluator**](/pr-cms-4519/docs/user-guide/evals-sdk/evaluators/goal_success_rate_evaluator/index.md): Binary goal achievement assessment

## Related documentation

-   [Chaos Testing](/pr-cms-4519/docs/user-guide/evals-sdk/chaos_testing/index.md): Chaos testing overview and guide

## Related pages

- [Partial completion evaluator](/pr-cms-4519/docs/user-guide/evals-sdk/evaluators/partial_completion_evaluator/index.md) (3 shared tags)
- [Recovery strategy evaluator](/pr-cms-4519/docs/user-guide/evals-sdk/evaluators/recovery_strategy_evaluator/index.md) (3 shared tags)
- [Tool simulation](/pr-cms-4519/docs/user-guide/evals-sdk/simulators/tool_simulation/index.md) (2 shared tags)
- [Chaos testing](/pr-cms-4519/docs/user-guide/evals-sdk/chaos_testing/index.md) (2 shared tags)
- [Deterministic evaluators](/pr-cms-4519/docs/user-guide/evals-sdk/evaluators/deterministic_evaluators/index.md) (1 shared tag)
- [Tool parameter accuracy evaluator](/pr-cms-4519/docs/user-guide/evals-sdk/evaluators/tool_parameter_evaluator/index.md) (1 shared tag)
- [Tool selection accuracy evaluator](/pr-cms-4519/docs/user-guide/evals-sdk/evaluators/tool_selection_evaluator/index.md) (1 shared tag)
- [Trajectory evaluator](/pr-cms-4519/docs/user-guide/evals-sdk/evaluators/trajectory_evaluator/index.md) (1 shared tag)
- [Experiment generator](/pr-cms-4519/docs/user-guide/evals-sdk/experiment_generator/index.md) (1 shared tag)
- [Plan topics for coverage](/pr-cms-4519/docs/user-guide/evals-sdk/topic_planning/index.md) (1 shared tag)