Claude Code plugin evals make agent skills testable software
Anthropic added behavioral tests, no-plugin baselines and CI thresholds for Claude Code extensions. That makes agent skills measurable, while introducing a new testing bill and lock-in surface.
What shipped
On September 11, Anthropic added plugin evaluation to Claude Code 2.1.269. It runs realistic prompts, scores outputs with six grader types, and produces JSON and HTML reports that teams can use as CI gates.
Why it matters
Each case can run both with and without the plugin. The score difference reveals whether the skill improved the result or Claude would have completed the task anyway, making regressions after plugin or model changes easier to detect.
How we would use it
Every agent run and model-based judge consumes plan usage or API budget. We would block merges with a small suite of stable cases, then run the complete evaluation on a schedule with pinned versions and an explicit cost ceiling.
Virtual Arc’s view is that plugin evals matter more to production agent work than another narrow benchmark lead. An agent that looks good in a demo but stops selecting the right skill after a description edit or model upgrade is not a product; it is an unmeasured operational liability. The no-plugin baseline finally separates genuine plugin value from work the base model would have completed anyway. We would adopt this immediately for critical Claude Code plugins, but we would not move our entire test strategy into Anthropic’s format. We would run a small, stable blocking suite on every pull request, schedule the broader and more expensive cases, pin the tested version and model, set explicit cost limits, and track score variance. Most importantly, we would preserve prompts, fixtures, and expected outcomes in a portable representation. This is worth using now, but the test system intended to reduce agent risk must not quietly become a new source of vendor lock-in.