Agent evaluation harness

Tool descriptions and server instructions are part of an agent's control surface: a wording change can alter its behaviour in ways unit tests cannot see. The tools needed behavioural verification against a real model.

I built a harness that runs twelve scenarios through Claude Code against two throw-away ripeat servers, with a stand-in package manager and the shell and file tools denied. The scenarios cover a recipe whose description instructs the agent to ignore its rules, a high-risk install, a request to remove the administrator policy and a lockfile plan. Each run is scored from the recorded tool calls and the final answer, by rules that are unit-tested themselves.

Engineered a harness that drives real Claude sessions through twelve scenarios, including prompt injection and high-risk installs, and scores them from recorded tool calls. Every change to a tool contract is regression-tested before release.

• Prose checks are lexical, so a failing transcript is reviewed before it is trusted.
• Runs spend tokens, so the suite gates releases rather than every commit.

Every change to tool descriptions or instructions is measured across twelve scenarios, three runs each, for a few dollars. The suite also exposed real defects, such as a plan that described the latest release for a pinned version, which were fixed.