Anthropic Adds Plugin Evals to Claude Code: 6 Grader Types, a No-Plugin Baseline, and a CI Gate for Skills
Anthropic introduced a `claude plugin eval` workflow for Claude Code that tests plugins against realistic prompts and grades outputs using 6 grader types The core metric is Δ (delta), measuring the difference between runs with and without the plugin loaded, proving whether the plugin actually contributed to the result Four graders (regex, tool_used, tool_order, file_exists) are free/computed locally, while two (llm, baseline) call a judge model and incur API costs A common failure mode is Δ near
Analysis
TL;DR
- Anthropic introduced a
claude plugin evalworkflow for Claude Code that tests plugins against realistic prompts and grades outputs using 6 grader types - The core metric is Δ (delta), measuring the difference between runs with and without the plugin loaded, proving whether the plugin actually contributed to the result
- Four graders (regex, tool_used, tool_order, file_exists) are free/computed locally, while two (llm, baseline) call a judge model and incur API costs
- A common failure mode is Δ near zero with a failing
tool_used: Skillgrader, indicating Claude isn't triggering the plugin on natural phrasing - The tool supports CI integration with flags like
--threshold,--max-cost-usd, and--trust-pluginto gate deployments
Why It Matters
This addresses a critical gap in agentic AI development: the inability to empirically measure whether a plugin actually improves outcomes versus what a bare model can already do. For AI practitioners building Claude Code plugins, this provides a structured, measurable evaluation framework that moves beyond syntax validation to behavioral validation. The industry needs reliable plugin evaluation as agentic workflows become production-critical, making this a foundational tool for the growing Claude plugin ecosystem.
Technical Details
- 6 Grader Types:
regex,tool_used,tool_order, andfile_existscompute locally from transcripts and disk (free);llmscores against prose criteria via a judge model;baselinecompares against a reference answer - Dual-Arm Evaluation: Every case runs twice—
with-arm(plugin loaded) andwithout-arm(plugin not loaded)—with Δ representing the plugin's isolated contribution - Case Structure: Eval suites live in an
evals/directory with subdirectories containingprompt.md(with frontmatter formax_turns,timeout_seconds,model,tags,allowed_tools) and agraders/folder with weighted markdown grader files - CI Integration: Command supports
--trust-plugin,--json,--threshold 0.8,--max-cost-usd 20,--model claude-sonnet-5, and--judge-model claude-haiku-4-5for automated pipeline gating - Auto-Generation:
claude plugin eval initreads the plugin, proposes cases and graders, and writes the suite; CI mode uses--bare <name>for blank templates
Industry Insight
- Plugin developers should adopt Δ as the primary success metric rather than absolute scores, since a plugin that merely replicates bare-model performance adds no value
- The
tool_used: Skillgrader failure with near-zero Δ is a diagnostic red flag for trigger design—developers must ensure natural language prompts actually invoke the plugin - CI gating with
--thresholdand--max-cost-usdenables cost-controlled regression testing, but teams should watch for usage-limit errors that can produce false regression signals
Disclaimer: The above content is generated by AI and is for reference only.