Skip to content

AgentCore evaluations

Amazon Bedrock AgentCore Evaluations is AWS’s managed service for evaluating agents. It scores agent traces with built-in and custom LLM-as-a-judge evaluators, evaluates production traffic continuously or past sessions on demand, and renders the results in the CloudWatch GenAI Observability dashboard. Strands is a supported framework: an agent instrumented with the Strands Harness SDK’s OpenTelemetry tracing emits traces that AgentCore Evaluations can score directly, with no evaluation code in your application.

Choose between the two based on where you want evaluation to run:

  • AgentCore Evaluations when you want managed, continuous evaluation of deployed agents, with results in a hosted dashboard.
  • Strands Evals SDK when you want evaluation in code: local experiments, CI gates, custom evaluators, simulators, and chaos testing.

The AWS documentation covers setup end to end:

Sending SDK results to the dashboard

Section titled “Sending SDK results to the dashboard”

Results produced by the Strands Evals SDK itself can land in the same dashboard. When the AGENT_OBSERVABILITY_ENABLED environment variable is true, Experiment writes each evaluation result to CloudWatch Logs, using the EVALUATION_RESULTS_LOG_GROUP environment variable as the destination log group. Run your evaluation script with ADOT auto-instrumentation configured for Bedrock AgentCore, as described in Add observability to your Amazon Bedrock AgentCore resources, and the scores appear in the GenAI Observability dashboard alongside your agent traces.

Section titled “Related documentation”