How coding agent evals catch what passing tests miss

A coding agent that passes the task's visible tests may still be unsafe to merge. It can delete an old behavior nobody put in the fixture, edit a generated file instead of its source, spend forty minutes exploring the wrong subsystem, or produce a patch that only makes sense in the repository where its authors trained it. A serious evaluation has to catch those outcomes, not congratulate the agent for finding green.

The useful unit is not a test run. It is a task attempt with a starting repository state, an instruction, a budget, an observable trajectory, a patch, and several independent judgments. Treating that whole attempt as evidence changes what teams measure and which failures they fix.

How should a coding agent task be defined?

A coding agent task needs a frozen starting commit, a realistic request, an explicit execution environment, and hidden acceptance checks. Without all four, two runs that appear to tackle the same issue may solve different problems.

Start with work that resembles what reaches your actual queue. A task can be a bug report, a small feature, a dependency migration, or an operational change. Preserve the ambiguity that a competent engineer would resolve by reading the repository, but remove ambiguity that only a departed employee could answer. "Fix invoice rounding when a credit line follows a taxable line" is fair if the code and existing behavior contain the answer. "Make finance happy" is not.

Pin more than the commit. Record the toolchain, dependency cache policy, environment variables available to the agent, network policy, test entry points, and resource limits. If the environment drifts, the score becomes a blend of agent quality and runner luck. Build images should have immutable identifiers, and fixtures should be versioned beside the evaluator.

Each task record should be inspectable. A compact specification might look like this:

{
  "task_id": "billing-credit-rounding-014",
  "repo_commit": "4f93c2a",
  "request": "Preserve tax rounding when a credit line follows a taxable line.",
  "visible_checks": ["test_billing_unit"],
  "hidden_checks": ["credit_after_tax", "mixed_currency_unchanged"],
  "budget": {"wall_seconds": 900, "model_cost_usd": 8.00},
  "allowed_paths": ["src/billing", "tests/billing"]
}

The hidden checks are not a trick. They stop the agent from optimizing directly against the complete answer sheet. Keep some behavioral checks outside the repository and rotate a portion of them, because agents can infer surprisingly much from test names and nearby fixtures.

Run each task several times. Agent behavior varies even when the model and prompt stay fixed. One lucky pass tells you the task is possible; repeated attempts tell you whether the system is dependable. Store the seed or sampling settings when the provider exposes them, but do not pretend they remove all variation.

Why passing tests are only one judgment

Tests answer whether selected observations matched expectations. They do not establish that the patch is scoped correctly, maintainable, secure, or compatible with behavior the suite never encoded.

Score task completion in layers. First, run the visible checks the agent could run. Second, run hidden tests that exercise neighboring inputs and the reported edge case. Third, run repository-wide checks, including lint, type checking, builds, and tests for downstream packages. Fourth, inspect properties that test frameworks rarely notice: forbidden file changes, dependency additions, public API changes, migrations, generated artifacts, and suspicious test deletion.

Keep those judgments separate instead of collapsing them immediately into one percentage. A patch that fixes the target but breaks an unrelated package differs from a patch that never fixed the target. Both fail release, but they point to different remedies. The first suggests weak impact analysis or inadequate repository exploration. The second suggests poor implementation or task understanding.

Patch inspection can be partly mechanical. Reject edits outside an allowlist when the task has a narrow boundary. Flag reductions in assertion count, new skips, broad exception handlers, and changes to snapshot files. These are review triggers, not automatic proof of cheating. A legitimate fix can update a snapshot or remove a stale assertion, so preserve the diff and the agent's explanation for human review.

Add a small rubric for qualities that machines cannot yet judge consistently. Reviewers can rate whether the patch follows local abstractions, puts validation at the right boundary, leaves dead code, or introduces a maintenance burden. Use anchored choices such as "matches an existing repository pattern" and "creates a parallel mechanism". Vague scores from one to five drift between reviewers and teach you very little.

The pass decision should remain strict: required behavior, regression checks, and repository health all pass. The diagnostic record should remain rich. Teams often ruin an eval by keeping only the final green or red bit, then have no evidence when a new agent version moves the score.

Previously fixed behavior belongs in the suite

A coding agent evaluation should replay bugs that your team already paid to understand. Those cases expose regression risk far better than a collection made only from fresh, tidy issues.

When a production bug is fixed, preserve three things: the repository state before the fix, the user-visible symptom, and an oracle that distinguishes the correct behavior. The oracle might be a focused test, a recorded request and response, a database state transition, or a command with normalized output. Remove secrets and unstable timestamps before storing traffic.

There are two distinct regression questions. "Can the agent solve an old bug from the broken state?" measures repair ability. "Does a new patch for another task reintroduce that old bug?" measures behavioral preservation. Teams blur these and end up with a benchmark that rewards bug fixing while saying nothing about collateral damage.

Build a cumulative behavior bank from closed incidents and difficult review findings. Tag each case by subsystem, failure mechanism, and consequence, then select relevant cases for every task plus a smaller repository-wide sample. The relevant set catches nearby breakage. The sample catches surprising coupling, such as a billing formatter changing a report export because both depend on the same rounding helper.

Do not run only the current default branch against the bank. Run the exact agent patch on its frozen base, because later human changes can hide or create failures. When a test fails for both the base and patched state, classify it as pre-existing noise. When it passes on the base and fails after the patch, you have a regression attributable to the attempt.

Flaky checks need quarantine with ownership and evidence, not silent retries until green. Record each individual run. A retry policy can tell you whether a patch is releasable under your current CI rules, but the raw outcomes tell you that the eval environment or product has nondeterminism. Mixing the two makes an agent look better without making its patch safer.

A growing behavior bank becomes expensive, so tier it. Run close, fast cases on every attempt; run broader replay before accepting a candidate agent release; run the slowest system cases on a schedule. The important constraint is monotonic coverage: a case leaves only when the behavior no longer exists or a stronger oracle replaces it.

Cost needs a denominator and a failure ledger

Cost per successful task is more useful than token spend per run. Cheap attempts that fail or require an engineer to repair the patch are not cheap work.

Capture model input and output charges, cached-token charges, tool compute, sandbox time, and any paid external service the run invokes. Keep engineering review time as a separate field rather than inventing a false dollar amount. You can still compare median review minutes across agent versions and task classes.

Use at least four views of cost:

• cost per attempt, which exposes runaway exploration;
• cost per accepted task, which includes failed attempts;
• cost by task class and repository, which prevents easy work from hiding expensive work;
• cost of wasted runs by failure category, which points to fixable orchestration defects.

Report distributions, not just averages. The median describes an ordinary run, while the 90th or 95th percentile catches the agents that loop through the same files, rebuild repeatedly, or dump an entire repository into context. A hard budget should stop those runs and mark them as budget exhaustion, not ordinary task failure.

Caching complicates comparisons. A warm dependency cache may be realistic for an internal CI worker, but a warm model prompt cache can make repeated benchmark tasks artificially cheap. Choose the condition that matches production, label it, and include a cold-cache slice. Never compare one version on warm tasks with another on newly created tasks.

The failure ledger matters because savings come from different changes. If most wasted spend comes from environment setup failures, changing the model will not help. If the agent repeatedly reads vendored code, improve repository guidance or tool filters. If it reaches a correct patch after long test cycles, improve test selection and incremental builds.

Cost limits also change behavior. An agent with a tight cap may implement the first plausible fix and skip broad verification. Evaluate quality at several budgets before declaring one configuration efficient. The useful operating point is where another unit of spend stops buying a meaningful increase in accepted tasks or reduction in severe regressions.

Latency should follow the critical path

Measure wall time as the user experiences it, then split that time into phases the team can act on. A single duration cannot tell you whether the delay came from model reasoning, tool startup, dependency installation, tests, or a congested runner.

Record timestamps for queue entry, sandbox readiness, first model response, each tool call, first patch, verification start, and final result. From those events, derive queue time, setup time, time to first edit, active agent time, verification time, and total time. Keep model latency and command duration separate.

Time to first edit is especially revealing. A very short time can mean the agent guessed before reading local conventions. A very long time can mean it wandered through irrelevant directories. Neither is automatically bad, so correlate it with patch acceptance and files inspected.

CI should also measure critical-path latency rather than summing parallel work. If unit tests and static analysis run together for eight minutes, the user waited eight minutes, not sixteen. Resource consumption may still total sixteen worker-minutes, which belongs in the cost record.

Set service objectives by task class. A one-file configuration correction and a cross-package schema change should not share a latency threshold. Compare an agent with the human workflow it replaces or assists: time to a reviewable patch, time to accepted merge, and time spent by the reviewer. The agent can be slower to produce a patch yet faster overall if its evidence makes review easier. It can also be fast and waste an afternoon in correction.

Timeouts deserve their own result. Do not score a timed-out run as equivalent to an incorrect patch. Preserve its last coherent state, tool trace, and budget usage. Repeated timeouts in one repository often reveal an evaluator problem, such as a test command waiting for an unavailable service, rather than weak code generation.

Run the latency benchmark on controlled workers, then separately observe shared CI. Controlled runs support model comparison. Shared runs show capacity planning and the experience developers actually get. Combining them produces a noisy number that answers neither question.

Unfamiliar repositories expose different failures

An agent evaluated only on repositories its designers know will inherit their assumptions. A codebase you did not write tests whether the agent can discover rules instead of receiving them through benchmark design.

The common failures start before code generation. The agent chooses the wrong test command, mistakes generated code for source, misses a second language in the build, ignores a local patching convention, or edits a shared module without finding its consumers. These attempts may compile and pass a focused test. They fail because the agent formed the wrong map of the system.

Select external or newly acquired repositories with legal permission and a reproducible build. Freeze them before task authors explore deeply. Ask one group to package the environment and another to create tasks from real issue history or observed defects. If the same person studies the code, writes detailed hints, and judges the result, the eval leaks their understanding into the instruction.

Measure discovery behavior without prescribing an ideal sequence. Useful signals include whether the agent reads repository guidance, identifies build entry points, searches call sites before changing a shared symbol, notices multiple implementations, and checks the diff before finishing. Do not award points for tool-call volume. Ten searches can reflect care or confusion.

Include repositories with awkward but genuine traits: mixed languages, custom generators, sparse tests, large fixtures, platform-specific scripts, and misleading directory names. Do not manufacture traps. The goal is to see whether the agent handles ordinary accumulated history, not whether it can solve a puzzle designed by the evaluator.

Contamination is hard to prove, so design around it. Use private code when you are authorized to do so, recent snapshots that could not have appeared in older training sets, and local transformations such as renamed business entities. Renaming alone does not create a new reasoning problem, but it reduces simple memorization. Strong evidence comes from comparable performance across known and genuinely unseen repositories, not from asking a model whether it remembers the code.

The trajectory explains failures the patch cannot

Store the agent's observable actions because the final diff cannot show how it found the answer, what it ignored, or why it exhausted its budget. A compact event trace makes failures reproducible without asking for private model reasoning.

For each model turn, retain the prompt version, response identifier, token usage, tool request, tool result status, timestamps, and files or commands touched. Redact secrets before persistence and put strict size limits on command output. The agent may print environment values or customer data while debugging, so access to traces should match access to the source repository.

Do not score hidden reasoning text or reward an agent for narrating the approach you expected. Providers expose different internal signals, and polished explanations can conceal bad decisions. Judge actions and artifacts: it read the build instructions, searched for callers, changed a file, ran a focused check, saw a failure, and revised the patch.

Classify the first decisive wrong turn. Later symptoms often cascade from it. If the agent edits a generated client, then fights the generator, then times out, "timeout" describes the terminal state but not the correct engineering response. The useful category is source-of-truth identification. Other categories might include environment discovery, task interpretation, dependency selection, impact analysis, implementation, and verification.

Record recovery as well as failure. An agent that notices a bad assumption after one failed check differs from one that repeats the same command six times. Useful trajectory metrics include repeated identical tool calls, time between a failing check and the next edit, fraction of inspected files outside the changed subsystem, and whether the final verification ran against a clean state. Interpret these signals with task outcomes; none is a quality score by itself.

Traces also expose harness interference. A tool may truncate the one compiler error that identifies the defect, a sandbox policy may block a normal repository command, or the orchestration layer may claim a command succeeded after killing its child process. Keep evaluator events distinct from agent events so the owner of each failure is visible.

Retention needs a deliberate policy. Keep the patch, normalized results, and aggregate metrics longer than raw command output. When you delete detailed traces, retain failure labels and evaluator versions so historical comparisons remain possible. An eval archive that quietly accumulates credentials and production snippets is itself a failed safety control.

The evaluator can fail before the agent does

An eval harness is production software. If its oracle is wrong, its sandbox leaks state, or its task fixture cannot build, the score measures evaluator defects.

Validate every task with two controls. The negative control is the untouched broken commit and should fail the target oracle. The positive control is the known human fix and should pass the target and regression oracles. A task that fails either control stays out of the scored set until repaired.

Then test isolation. Give every attempt a fresh worktree, clean process namespace, controlled clock where time matters, and unique service resources. A database left behind by the prior run can make the next patch appear correct. Shared dependency caches may be acceptable, but they must not contain mutable task outputs.

The runner should emit a compact, stable result that CI can retain:

$ ./eval-agent fixtures/billing-credit-rounding-014
task=billing-credit-rounding-014 outcome=regression
target=pass hidden=pass repository=fail
cost_usd=3.42 wall_seconds=286 review=required
artifacts=patch.diff,events.json,test-results.xml

Normalize nondeterministic values before comparing outputs. Sort unordered records, replace generated identifiers with stable placeholders, and compare structured data rather than screenshots or logs when possible. Every normalization rule risks concealing a defect, so keep the raw artifact beside the normalized one.

Version the evaluator, task, and oracle independently. When an oracle changes, retain enough metadata to reproduce old scores and recalculate when feasible. A model release should not look better merely because someone loosened a fixture in the same pull request.

Finally, sample evaluator failures manually. Inspect infrastructure errors, suspiciously fast passes, and clusters where every agent version fails identically. These are often broken tasks. Counting them as model failures may feel conservative, but it directs engineering effort to the wrong system.

A scorecard should preserve hard failures

The scorecard should make release decisions clear without averaging away a security regression or data corruption bug. Use gates for unacceptable outcomes and metrics for tradeoffs.

Start with eligibility gates: the agent must stay inside the allowed environment, avoid forbidden secret access, produce a reviewable patch, and pass all severe regression oracles. Any breach makes the candidate ineligible regardless of its average success rate. Define severity before running the comparison, or teams will relabel inconvenient failures after seeing results.

For eligible candidates, show a table by task class and repository. Include accepted-task rate, target-fix rate, regression-free rate, median and tail cost per accepted task, median and tail wall time, reviewer minutes, and failure categories. Provide counts beside percentages. Three successes out of four and seventy-five out of one hundred share a percentage but not confidence.

Avoid one weighted score unless an automated selection system requires it. Weights hide policy decisions and invite arguments about arithmetic. A release review can instead ask whether the candidate clears every gate, improves the metrics you care about, and keeps regressions within the declared tolerance.

Compare against useful baselines. Include the current agent configuration, a minimal agent with fewer tools, and a no-agent state where the evaluator applies no patch. Human results can help when tasks and working conditions are comparable, but do not use experienced maintainers on their own code as the universal benchmark for an unfamiliar agent.

Slice results before trusting the aggregate. Check language, repository size, test quality, task type, and whether the required change crosses subsystem boundaries. A candidate can improve the headline rate by getting much better at small TypeScript edits while getting worse at database migrations.

Treat reviewer overrides as data. If a reviewer accepts a patch that the harness rejects, or rejects one the harness passes, require a reason code and inspect the oracle. The reviewer may be wrong, but disagreement is where the scorecard learns.

CI rollout needs control groups and stop rules

Deploy a coding agent through CI as a measured change, with a fixed comparison window and stop conditions. Running it on every pull request immediately turns developers into unpaid eval infrastructure.

Begin in shadow mode on representative tasks. The agent receives the same repository state and request but cannot modify the real branch. Compare its patch and evidence with the outcome of normal engineering work. Shadowing reveals environment gaps and reviewer burden without putting its changes on the merge path.

Next, allow suggestions on low-consequence task classes with mandatory review. Randomly retain a control group on the prior agent version or normal workflow. Without a concurrent control, changes in task mix, repository activity, and CI load can masquerade as improvement.

Write stop rules before rollout. Pause automatic suggestions if a severe regression appears, secret boundaries are crossed, evaluator infrastructure errors exceed your tolerance, or tail cost grows beyond the assigned budget. A stop rule should name who can resume the rollout and what evidence they need.

Watch for adaptation. Developers may start writing unusually explicit issues for the agent, avoiding tasks it handles poorly, or rubber-stamping familiar patch shapes. Those changes affect apparent performance. Review a sample of requests and review comments, and keep a stable benchmark set outside the live queue.

The parity harness we use at CodeHero compares rewritten-system behavior with recorded production traffic, because a clean build cannot prove that decades of edge cases survived an architectural change. The same principle applies to a CI agent: preserve real behavior as executable evidence, then judge each patch against it.

Promotion should be reversible. Keep the old configuration available, tag every agent-authored patch with the evaluator version, and retain the task attempt long enough to investigate later regressions. If you cannot connect a production defect back to the prompt, repository state, patch, and checks that approved it, the eval record is incomplete.

Recheck the benchmark after any material change to the model, system prompt, tool permissions, context assembly, or runner image. Those components interact, so a model-only version label cannot identify the system that produced a patch. Keep a small canary set stable for rapid comparison and refresh the broader set as the live work changes. When refreshes retire tasks, run old and new sets for one overlap period so a difficulty shift does not masquerade as a quality shift.

An agent earns broader authority only when it repeatedly produces acceptable patches on the repositories where it will work, within a budget the team can defend. Green tests open the review. They do not finish it.