Build custom code-based evaluators in Amazon Bedrock AgentCore

TL;DR

• Amazon Bedrock AgentCore now supports custom code-based evaluators that utilize AWS Lambda functions to perform deterministic quality checks on agentic applications.
• The solution allows developers to bypass Foundation Model (FM) token costs by using objective code logic for specific validation tasks such as JSON schema validation, numerical accuracy, and PII detection.
• Evaluators operate at three distinct granularities: TRACE, TOOL_CALL, and SESSION, with results logged to Amazon CloudWatch for observability.
• The framework integrates seamlessly with both on-demand CI/CD pipelines and online production traffic monitoring, enabling real-time quality gating.
• It complements LLM-as-a-Judge evaluators by providing contract-verified reliability, ensuring agents adhere to strict business rules and data handling protocols.

Why It Matters

This capability is critical for enterprises moving AI agents from prototype to production, where "sounds right" is insufficient and strict compliance is required. By decoupling deterministic logic checks from LLM inference, organizations can significantly reduce evaluation costs while enforcing hard constraints on financial accuracy, security, and workflow integrity. It provides a standardized, framework-agnostic way to validate agent behavior across different agent implementations using OpenTelemetry spans.

Key Data

• Evaluation Granularity: Three specific levels are supported for code-based evaluators: TRACE, TOOL_CALL, and SESSION.
• Response Contract: Lambda evaluators must return a label (e.g., PASS or FAIL), with an optional numeric score ranging from 0.0 to 1.0.
• Data Standard: The evaluator payload consists of OpenTelemetry (OTel) spans, allowing compatibility with traces from different agent frameworks.
• Cost Efficiency: Custom code-based evaluators do not require Foundation Model (FM) tokens for each request, unlike LLM-as-a-Judge checks.
• Numerical Sensitivity: The system is designed to flag deviations as small as 0.1 percent in financial data, which is critical for trading decisions.

Technical Details

• Lambda Integration: The core engine is an AWS Lambda function registered with the AgentCore control plane. When an evaluation triggers, AgentCore assumes an IAM role, invokes the Lambda with a payload of OTel spans, and writes the response to Amazon CloudWatch Logs.
• Evaluator Payload Structure: The input object includes a schema version, evaluator ID/name, evaluation level, and an array of OTel spans. Trace-level evaluations include a specific target field, while session-level evaluations score the full conversation without a specific target.
• Deterministic Logic Categories: The approach supports four primary quality dimensions suited for code: tool response schema validation (catching structural issues), numerical accuracy against reference systems (preventing arithmetic errors), workflow contract compliance (verifying tool call sequences), and PII/secret detection (enforcing data withholding rules).
• Registration & Lifecycle: Developers must register evaluators separately for each granularity level (trace, tool call, session) even if they point to the same Lambda function. The response schema strictly requires a label on success, while errors must return specific error codes and messages.

Industry Insight

• Cost-Performance Balance: For high-volume production environments, replacing LLM-based checks with Lambda-based deterministic checks for objective metrics (like schema or price accuracy) will drastically lower inference costs while improving reliability.
• Compliance-First Design: The explicit support for PII detection and workflow contract compliance indicates a shift toward using evaluators as security gates, not just quality metrics, which is essential for regulated industries like finance and healthcare.
• Standardization via OTel: By basing evaluation payloads on OpenTelemetry spans, AWS is reducing lock-in to their specific agent frameworks, allowing enterprises to evaluate agents built on LangChain, AutoGen, or custom stacks, provided they emit standard telemetry.

FAQ

Q: Can the same Lambda function be used for multiple evaluation granularities?
A: Yes, you can point the same Lambda function to multiple evaluation levels, but you must register it separately for each level (TRACE, TOOL_CALL, or SESSION).

Q: How do code-based evaluators differ from LLM-as-a-Judge evaluators in terms of cost and determinism?
A: Code-based evaluators run objective logic without consuming Foundation Model tokens, ensuring identical results for identical inputs. This makes them ideal for deterministic checks like schema validation and numerical accuracy, whereas LLM-as-a-Judge is used for subjective qualities like clarity and usefulness.