claude plugin eval Explained: Using the With/Without Delta (Δ) to Judge Whether a Skill Helps — and Why Δ Never Fails Your CI

A Skill passing a test does not mean it is useful: Claude might do just as well without it. That is the design point of claude Plugin eval. Each test case runs with your plugin loaded and then again with no plugin at all, and the difference between the two scores is what the plugin actually contributed. The command needs Claude Code v2.1.269 or later, and git 2.31 or later if git is installed.

What a case looks like

The suite lives in an evals/ directory inside the plugin. Each case is a subdirectory holding a prompt.md (the sentence a user would type, plus run limits) and one or more files under graders/. The recommended starting point is claude plugin eval init, which opens an interactive session, reads your plugin, asks what a good result looks like, proposes prompts that should and should not trigger the plugin, designs graders and runs them once as a trial, then writes one case directory per prompt. To write one by hand, claude plugin eval init --bare first-case produces a blank template. The official prompt.md example, for a Skill that drafts commit messages:

---
max_turns: 10
allowed_tools: [Read, Glob, Grep, Skill]
---

Write me a commit message for this change: I renamed getUser to fetchUser and updated the three call sites.

Note that the prompt is written the way a user would naturally phrase it, not by naming the skill. A second grader checks that the answer really came from your skill (replace your-skill-name with the skill's directory name):

---
type: tool_used
tool: Skill
input_match: '"skill"\s*:\s*"(?:[\w-]+:)?your-skill-name"'
---

How scoring works

One run of a non-deterministic agent tells you little, so each case runs three times by default. A run's score is the fraction of graders that passed (weights optional), and the case's score is the mean of the three; a case passes when its score meets --threshold (default 1.0). With a baseline you get two scores: WITH (plugin loaded) and W/OUT (not loaded), and their difference is Δ. The official docs put it bluntly: if a case scores 1.0 with and without the plugin, the plugin is not what made it pass.

There are six grader types. regex, tool_used, tool_order and file_exists are computed from the transcript and files and cost nothing; llm and baseline call a judge model, and llm needs at least two of three votes to pass. There are no custom-code graders. Also, a tool_used: Skill grader ("the skill was invoked") can never pass without the plugin, so in a two-arm comparison it is excluded from the score (marked scored: false) and shown only as a pass/fail indicator in the with-arm.

The order to read results in

Each summary row reads CASE WITH W/OUT Δ RUNS COST NOTES, and a report.html is produced. The docs say the most common first finding is a Δ near zero together with a failing tool_used: Skill grader, which means Claude is not choosing your skill on natural phrasing; adjust the skill's description, rerun and compare. The opposite trap: the Skill grader passes but Δ is negative, in which case suspect the judge first, because a small judge model can mark a correct answer wrong when it is formatted differently. Rerun with --judge-model sonnet and write the rubric as concrete PASS and FAIL conditions.

To iterate on one case, claude plugin eval . --case <case-name> --runs 1 --ablation none runs one arm once, but the docs warn that a single run is noisy, so confirm any change at the default three runs.

Cost: one case is really six runs

One case is three runs with the plugin and three without, six agent runs in all, plus three short judge calls per llm or baseline grader per run. These use your own credentials and count against your plan usage or API bill; the cost the command reports is a list-price estimate. The official advice for keeping costs predictable: give quick every-change suites only graders that don't call a judge, use --ablation none when you don't need Δ (which halves the cost), and set a ceiling with --max-cost-usd. My own take, a recommendation and not official policy: iterate day to day with one arm and free graders, and run the full two-arm comparison only before a release or when the model changes.

Putting it in CI

The official CI example writes a JSON result for archiving, passes --trust-plugin so the job never waits at the trust prompt, pins both models so scores are comparable over time, keeps the report local, and sets a cost ceiling:

claude plugin eval . \
  --trust-plugin \
  --json results.json \
  --threshold 0.8 \
  --model claude-sonnet-5 \
  --judge-model claude-haiku-4-5 \
  --no-publish \
  --max-cost-usd 20

The exit codes: 0 means every case met the threshold and every case file loaded; 1 means a case scored below the threshold, a case file failed to load, no cases were found, a run could not start, the plugin directory is not trusted and --trust-plugin was not passed, or an option was invalid; 2 means a partial run (the cost ceiling was hit, or the credential was rejected); 130 means interrupted; 143 means terminated, for example by a CI timeout.

Two points are easy to misread. First, the threshold compares only the with-arm score. Second, Δ is reported but never changes the exit code. A fully green CI therefore does not mean the plugin is still adding value; if you want to Block a "Δ regression," you have to read the --json output and decide yourself. Also, the default threshold of 1.0 means any imperfect case exits 1, so you will usually lower it to fit your needs. In CI, init needs a terminal to ask its questions, so use init --bare instead.

Three easy misreadings

• A usage limit can turn scores into 0: if the suite hits your plan's usage limit or an API rate limit partway, later runs end with an error and usually score 0, and the suite is not marked partial, so it looks like a regression. Check the NOTES column or the error in the JSON before drawing conclusions.
• The test environment differs from your daily one: every run gets a clean temporary home and configuration with only the plugin under test loaded. Your user settings, CLAUDE.md, other plugins, memory and MCP servers are absent; the case definitions are hidden from the agent; and the Artifact tool is off.
• Passing is not the same as safe: it loads and runs the plugin's skills, hooks and agents on your machine as you, and the docs state plainly that a passing suite says nothing about whether the plugin is safe.

The docs also note that the skill-creator plugin, which iterates on one skill inside a conversation, runs a similar comparison but with its own evals/evals.json format, and neither tool reads the other's case files.