## Overview

The `MultimodalOutputEvaluator` assesses the quality of agent outputs for tasks that involve images or documents alongside text. It uses an MLLM as the judge and evaluates responses against a user-defined rubric, extending [`OutputEvaluator`](/pr-cms-3708/docs/user-guide/evals-sdk/evaluators/output_evaluator/index.md) with multimodal input support. A complete example can be found [here](https://github.com/strands-agents/harness-sdk/blob/main/site/docs/examples/evals-sdk/multimodal_output_evaluator.py).

## Key Features

-   **MLLM-as-a-Judge**: Uses a multimodal judge model to evaluate responses against the rubric and the media
-   **Data-Driven Dispatch**: Emits Strands SDK content blocks when the input carries media; falls back to a plain text prompt otherwise
-   **Automatic Reference Comparison**: Appends a reference suffix to the rubric when `expected_output` is provided on the case
-   **Multiple Media Sources**: Accepts file paths, base64 strings, data URLs, HTTP(S) URLs (auto-fetched via the stdlib), raw bytes, and PIL Images
-   **Built-in Subclasses**: `MultimodalOverallQualityEvaluator`, `MultimodalCorrectnessEvaluator`, `MultimodalFaithfulnessEvaluator`, and `MultimodalInstructionFollowingEvaluator` each default to a built-in rubric
-   **Async Support**: Supports both synchronous and asynchronous evaluation

## When to Use

Use the `MultimodalOutputEvaluator` when you need to:

-   Evaluate image or document understanding (VQA, chart QA, document QA, image captioning, OCR-style tasks)
-   Assess whether a text response is grounded in the visual content it describes
-   Score multimodal responses against a custom rubric that cannot be expressed as exact-match assertions
-   Compare multimodal agent configurations or prompts
-   Benchmark MLLM judges against text-only LLM judges on the same tasks

## Evaluation Level

This evaluator operates at the **OUTPUT\_LEVEL**, scoring a single agent response per case.

## Parameters

### `rubric` (required)

-   **Type**: `str`
-   **Description**: The evaluation criteria that define what constitutes a good response. Should include scoring guidelines (e.g., “Score 1.0 if …, 0.0 if …”). You can author your own or reuse a built-in (see [Built-in Rubrics](#built-in-rubrics)).

### `model` (optional)

-   **Type**: `Model | str | None`
-   **Default**: `None` (uses default Bedrock model)
-   **Description**: The multimodal judge model. Can be a model ID string or a `Model` instance. The default must support image input.

### `system_prompt` (optional)

-   **Type**: `str | None`
-   **Default**: `None` (uses the built-in `MLLM_JUDGE_SYSTEM_PROMPT`)
-   **Description**: Custom system prompt to guide the judge model’s behavior.

### `include_inputs` (optional)

-   **Type**: `bool`
-   **Default**: `True`
-   **Description**: Whether to include the user instruction in the evaluation prompt. Set to `False` to score the response in isolation.

### `reference_suffix` (optional)

-   **Type**: `str | None`
-   **Default**: `None` (uses the built-in default suffix)
-   **Description**: Text appended to the rubric when `expected_output` is present on the case. Lets you customize how the judge should use the reference (e.g., strict vs. lenient grading).

### `uses_environment_state` (optional)

-   **Type**: `bool`
-   **Default**: `False`
-   **Description**: Whether to include environment state in the evaluation prompt, enabling assessment of agent side effects alongside the output.

## Data-Driven Dispatch

The evaluator decides whether to call the judge as a multimodal or text-only model based on the shape of `case.input`:

-   If `case.input` is a `MultimodalInput` carrying non-empty media, the composer returns a list of content blocks (media first, then the evaluation text), and the judge is invoked as an MLLM.
-   If `case.input` is anything else (e.g., a plain string, or a `MultimodalInput` whose `media` is empty), the composer returns a plain text prompt, and the judge behaves exactly like `OutputEvaluator`.

This means the same evaluator can grade a mixed experiment containing both multimodal and text-only cases without any flags to set.

## `MultimodalInput`

`MultimodalInput` is the Pydantic model used for multimodal cases. It has three fields:

-   **`media`** (`ImageData | list[ImageData] | str`): one or more image sources. A plain string is accepted as shorthand for a single `ImageData(source=...)`.
-   **`instruction`** (`str`): the user’s question or request about the media.
-   **`context`** (`str | None`): optional additional context surfaced to the judge.

`MultimodalInput` round-trips through `Experiment.to_dict` / `from_dict`: when a saved experiment is reloaded, raw dicts are coerced back to `MultimodalInput` at the prompt-composer boundary.

## `ImageData`

`ImageData` normalizes image sources so the judge receives raw bytes in the correct format. Accepted forms for the `source` field:

| Source | Example |
| --- | --- |
| Local file path | `"/path/to/chart.png"` or `pathlib.Path(...)` |
| Base64 string | `"iVBORw0KGgo..."` |
| Data URL | `"data:image/png;base64,iVBORw0KGgo..."` |
| HTTP(S) URL | `"https://example.com/image.jpg"` |
| Raw bytes | `b"\x89PNG..."` |
| PIL Image | `PIL.Image.Image` instance (requires `Pillow`) |

HTTP(S) URLs are auto-fetched via `urllib.request` from the standard library, so no extra dependencies are required. The `format` field (`"jpeg"`, `"png"`, `"gif"`, `"webp"`) is auto-detected from the extension or data URL when omitted. S3 URIs and boto3 auto-fetch are not supported; pre-download S3 objects to bytes or a local path before constructing `ImageData`.

## Built-in Rubrics

The following rubric constants are available from `strands_evals.evaluators.prompt_templates.multimodal`:

-   `OVERALL_QUALITY_RUBRIC_V0`: Likert-5 scale (`0.0`, `0.25`, `0.5`, `0.75`, `1.0`) across visual accuracy, instruction adherence, completeness, and coherence
-   `CORRECTNESS_RUBRIC_V0`: strict binary (`1.0` / `0.0`) fact-check against the image
-   `FAITHFULNESS_RUBRIC_V0`: strict binary, catches hallucinations not grounded in the image
-   `INSTRUCTION_FOLLOWING_RUBRIC_V0`: strict binary, checks count/format/scope/order/completeness/style constraints

Each built-in rubric is the default for the matching subclass (see [Related Evaluators](#related-evaluators)).

## Scoring System

The score is a float between `0.0` and `1.0`; the granularity is determined by the rubric:

-   **Strict binary** (`CORRECTNESS_RUBRIC_V0`, `FAITHFULNESS_RUBRIC_V0`, `INSTRUCTION_FOLLOWING_RUBRIC_V0`): `1.0` on success, `0.0` on any violation. A response passes only if the score is `1.0`.
-   **Likert-5** (`OVERALL_QUALITY_RUBRIC_V0`): `0.0`, `0.25`, `0.5`, `0.75`, or `1.0`. A response typically passes if the score is `>= 0.75`.
-   **Custom rubric**: any granularity you define in the rubric text. Specify the pass threshold in the rubric itself (e.g., “Score 1.0 if …”).

## Basic Usage

### Reference-Free Evaluation

```python
import asyncio

from strands_evals import Case, Experiment
from strands_evals.evaluators import MultimodalOutputEvaluator
from strands_evals.evaluators.prompt_templates.multimodal import OVERALL_QUALITY_RUBRIC_V0
from strands_evals.types import ImageData, MultimodalInput
from strands_evals.types.evaluation_report import EvaluationReport


def task_function(case: Case) -> str:
    # Replace with a call to your multimodal agent. For illustration we return
    # a fixed response so the evaluator has something to score.
    return "The chart is a bar chart showing quarterly revenue for three product lines in 2024."


cases = [
    Case(
        name="chart-overview",
        input=MultimodalInput(
            media="/path/to/your/chart.png",
            instruction="What kind of chart is shown and what does it represent?",
        ),
    ),
]

evaluator = MultimodalOutputEvaluator(rubric=OVERALL_QUALITY_RUBRIC_V0)
experiment = Experiment(cases=cases, evaluators=[evaluator])

async def main():
    report = await experiment.run_evaluations_async(task_function)
    report.run_display()

asyncio.run(main())
```

### Reference-Based Evaluation

Setting `expected_output` on a case switches the evaluator into reference-based mode. The configured `reference_suffix` is appended to the rubric so the judge compares the response against the reference answer.

```python
from strands_evals import Case, Experiment
from strands_evals.evaluators import MultimodalOutputEvaluator
from strands_evals.evaluators.prompt_templates.multimodal import CORRECTNESS_RUBRIC_V0
from strands_evals.types import ImageData, MultimodalInput

# Reuses the task_function defined in the reference-free example above.

cases = [
    Case(
        name="chart-value",
        input=MultimodalInput(
            media=ImageData(source="/path/to/revenue_chart.png"),
            instruction="What is the Q3 revenue for Product A?",
        ),
        expected_output="$4.2M",
    ),
]

evaluator = MultimodalOutputEvaluator(rubric=CORRECTNESS_RUBRIC_V0)
experiment = Experiment(cases=cases, evaluators=[evaluator])

async def main():
    report = await experiment.run_evaluations_async(task_function)

asyncio.run(main())
```

### Multiple Images per Case

Pass a list of `ImageData` (or a list of source strings) to evaluate responses that reason over several images at once:

```python
Case(
    name="before-after",
    input=MultimodalInput(
        media=[
            ImageData(source="/path/to/before.jpg"),
            ImageData(source="/path/to/after.jpg"),
        ],
        instruction="What changed between the two photos?",
        context="Both photos were taken at the same location one year apart.",
    ),
)
```

## Evaluation Output

Like `OutputEvaluator`, `MultimodalOutputEvaluator` returns `EvaluationOutput` objects with:

-   **score**: Float between `0.0` and `1.0` (see [Scoring System](#scoring-system))
-   **test\_pass**: Boolean indicating if the test passed
-   **reason**: String containing the judge’s reasoning for the score
-   **label**: Optional label categorizing the result

## Best Practices

1.  **Use a Multimodal-Capable Judge**: The default Bedrock model must support image input. If you override `model`, confirm the model ID accepts image content blocks.
2.  **Keep Images at Reasonable Resolution**: Very large images increase latency and cost without improving judgment; resize to the minimum resolution needed for the task.
3.  **Write Rubrics That Reference the Image**: Phrases like “based on what is visible in the image” anchor the judge to the media rather than to its prior knowledge.
4.  **Use Reference Mode for Ground-Truth Tasks**: Chart QA and VQA benchmarks with known answers benefit from `expected_output` and the reference suffix.
5.  **Start with a Built-in Rubric**: The four built-in rubrics are the defaults that come with Strands Evals; customize only once you see where they fall short on your data.

## Related Evaluators

-   [**MultimodalOverallQualityEvaluator**](/pr-cms-3708/docs/user-guide/evals-sdk/evaluators/multimodal_overall_quality_evaluator/index.md): Likert-5 quality scoring
-   [**MultimodalCorrectnessEvaluator**](/pr-cms-3708/docs/user-guide/evals-sdk/evaluators/multimodal_correctness_evaluator/index.md): Strict binary factual correctness
-   [**MultimodalFaithfulnessEvaluator**](/pr-cms-3708/docs/user-guide/evals-sdk/evaluators/multimodal_faithfulness_evaluator/index.md): Strict binary hallucination detection against the image
-   [**MultimodalInstructionFollowingEvaluator**](/pr-cms-3708/docs/user-guide/evals-sdk/evaluators/multimodal_instruction_following_evaluator/index.md): Strict binary instruction compliance
-   [**OutputEvaluator**](/pr-cms-3708/docs/user-guide/evals-sdk/evaluators/output_evaluator/index.md): Text-only parent class with the same `evaluate()` contract

## Related pages

- [Multimodal Correctness Evaluator](/pr-cms-3708/docs/user-guide/evals-sdk/evaluators/multimodal_correctness_evaluator/index.md) (2 shared tags)
- [Multimodal Faithfulness Evaluator](/pr-cms-3708/docs/user-guide/evals-sdk/evaluators/multimodal_faithfulness_evaluator/index.md) (2 shared tags)
- [Multimodal Instruction Following Evaluator](/pr-cms-3708/docs/user-guide/evals-sdk/evaluators/multimodal_instruction_following_evaluator/index.md) (2 shared tags)
- [Multimodal Overall Quality Evaluator](/pr-cms-3708/docs/user-guide/evals-sdk/evaluators/multimodal_overall_quality_evaluator/index.md) (2 shared tags)
- [Google](/pr-cms-3708/docs/user-guide/concepts/model-providers/google/index.md) (1 shared tag)
- [Vercel](/pr-cms-3708/docs/user-guide/concepts/model-providers/vercel/index.md) (1 shared tag)
- [OpenAI](/pr-cms-3708/docs/user-guide/concepts/model-providers/openai/index.md) (1 shared tag)
- [Writer](/pr-cms-3708/docs/user-guide/concepts/model-providers/writer/index.md) (1 shared tag)
- [Amazon Nova](/pr-cms-3708/docs/user-guide/concepts/model-providers/amazon-nova/index.md) (1 shared tag)
- [Amazon Bedrock](/pr-cms-3708/docs/user-guide/concepts/model-providers/amazon-bedrock/index.md) (1 shared tag)