How to review AI-generated code you cannot read

A model can produce more code in an afternoon than a capable team can inspect in a week. Trying to preserve the old review ritual by adding more reviewers does not solve that mismatch. It creates a queue, encourages superficial approvals, and still misses faults distributed across hundreds of individually plausible functions.

The workable standard is evidence, not complete visual coverage. Reviewers should prove that the new system preserves the required behavior, obeys rules that must always hold, and contains no unacceptable decisions at the few points where human judgment cannot be automated. Reading still matters, but it moves from every line to the places where a line can change authority, money, data, or recovery.

Classify the risk before choosing the review method

The amount of code tells you almost nothing about the amount of review needed. Review effort should follow the consequence of a wrong decision and the ease with which tests can observe it. A generated parser with a precise grammar may deserve more automated cases and less line reading than a short authorization function whose intent lives in policy documents and exceptions.

Start by dividing the change into behavioral surfaces. A surface is a set of inputs, state, and outputs that can be tested as one contract: invoice calculation, account eligibility, file conversion, permission evaluation, batch restart, or database migration. Do not divide work by generated file count. Models often spread one decision across adapters, helpers, and types, while a large generated data mapper may contain almost no independent decisions.

For each surface, record four facts: the consequence of failure, the available reference behavior, the invariants that apply, and the code locations that exercise authority. Consequence determines how much independent evidence you need. Reference behavior tells you whether differential testing is possible. Invariants expose faults even when the reference implementation shares them. Authority locations identify where people must read.

A useful review matrix looks like this:

• Tax calculation: compare the old service with approved examples, replay cases, check conservation, and inspect rounding and rate selection.
• CSV import: use the format specification and production samples, generate malformed inputs, and inspect errors and resource limits.
• Role check: build a decision table from policy rules, test denials, and inspect every allow path.
• Batch restart: replay recorded job history, inject crashes, check idempotence, and inspect transaction boundaries.

This classification also answers whether model-written code is safe to merge. It is safe enough only when evidence matches the failure consequence and unresolved differences have owners. No model score, test count, or reviewer confidence can replace that decision. A low-risk formatter may pass with properties and samples. A payment or permission path needs independent specifications, adverse cases, and direct inspection.

Write the observable contract before reading the implementation

A reviewer needs an external description of correct behavior before opening generated code. Otherwise the implementation quietly becomes its own specification, and plausible structure gets mistaken for correct intent. The contract should describe what callers can observe, including returned values, state changes, errors, timing boundaries that matter, and side effects.

Extract the contract from sources that existed before generation: interface definitions, operator runbooks, production requests and responses, database constraints, accepted fixtures, regulations, and interviews with the people who handle exceptions. Old code is one source, not the only source. Comments may be stale, tests may encode accidents, and current operators may depend on behavior that nobody documented.

Write awkward cases explicitly. What happens to an empty input? Which timezone owns a cutoff? Does a retry repeat an email or reuse the earlier result? Is a missing field different from a zero value? Which decimal rule applies halfway between two representable amounts? Does ordering carry meaning even when an API claims it does not? These questions cause more migration failures than syntax or type errors.

The contract must also mark allowed change. Some rewrites should preserve byte-for-byte output. Others may normalize whitespace, replace an internal identifier, or return a clearer error while preserving the category. If reviewers do not define an equivalence relation, a comparison tool will report harmless noise or, worse, normalize away a real defect.

Keep contract statements testable. "Handles invoices correctly" is useless. "For an accepted invoice, the posted debit equals the sum of posted credits in the ledger currency" can become a property. "Retries are safe" is vague. "Repeating the same request identifier produces no additional external side effect" gives the harness something to measure.

Do this before reading because generated code is persuasive. It has names, branches, checks, and comments arranged in familiar shapes. Once a reviewer sees a tidy implementation, the reviewer starts explaining why it makes sense instead of asking whether it implements the required rule. The contract keeps the burden of proof outside the generated text.

Property tests cover rules across an input space

Property tests are best when you can state a rule that holds for many inputs but cannot afford to enumerate those inputs. They do not prove a program correct, and a weak property can bless nonsense. Their strength is forcing the reviewer to name relationships that examples hide.

Choose properties from the domain, not from the implementation. Round-trip properties work for encoders when decoding an encoded valid value must recover the original. Conservation properties work for money, inventory, and record counts. Idempotence applies to retries, normalization, and set-like updates. Monotonicity applies when adding an eligible item cannot reduce an aggregate. Metamorphic properties compare related inputs, such as reordering records that the contract says are unordered.

A compact Go fuzz test for a normalization boundary might look like this:

func FuzzNormalizeAccount(f *testing.F) {
    f.Add(" ab-123 ")
    f.Add("AB123")

    f.Fuzz(func(t *testing.T, raw string) {
        got, err := NormalizeAccount(raw)
        if err != nil {
            return
        }
        again, err := NormalizeAccount(got)
        if err != nil {
            t.Fatalf("normalized value rejected: %q", got)
        }
        if again != got {
            t.Fatalf("not idempotent: first=%q second=%q", got, again)
        }
        if strings.ContainsAny(got, " -\t\n") {
            t.Fatalf("separator survived: %q", got)
        }
    })
}

This test checks two real rules: successful normalization is idempotent, and the output contains no forbidden separators. It deliberately does not assert that every string must succeed. That would turn an unknown input policy into a fabricated requirement. Add a separate generator for valid account forms if the accepted grammar is known, then require success only for that generator.

Shrinking matters. When a generator finds a failure in a 4,000-character payload, the useful artifact is the smallest input that still fails. Store that reduced case as an ordinary regression fixture. The random search finds the hole; the fixed case prevents its return and makes review repeatable. Keep the seed when the framework exposes one, but never rely on a seed alone because generator or runtime changes can alter the sequence.

Watch for properties that merely echo the code. Testing that SortRecords returns the same result as another call to SortRecords says little. Testing that output is ordered, contains the same multiset of records, and is unchanged by a second sort checks independent facts. Mutation testing can expose weak suites here: if changing a comparison or deleting a validation branch leaves every property green, the suite has not earned confidence.

Differential testing finds drift the specification forgot

Differential testing runs the old and new implementations against the same inputs, then compares observable results under an explicit equivalence policy. For a legacy rewrite, it is usually the fastest way to discover hidden behavior because production has already explored combinations that a newly written test plan will miss.

Build the harness around a boundary, not around private functions. Capture a request or job input, relevant starting state, returned result, durable state changes, and external side effects. Then execute both systems from equivalent starting conditions. Replace clocks, random sources, identifiers, and remote services with controlled adapters so nondeterminism does not flood the comparison.

A comparison record should be inspectable rather than a green or red counter:

{
  "case_id": "replay-01842",
  "input_hash": "sha256:...",
  "old": {"status": "accepted", "total_minor": 1250, "events": ["invoice_posted"]},
  "new": {"status": "accepted", "total_minor": 1250, "events": ["invoice_posted"]},
  "normalizations": ["generated_id", "timestamp_within_1s"],
  "result": "equal"
}

Record every normalization. If the harness removes timestamps, sorts collections, masks identifiers, or maps error text into categories, that policy belongs in review. A broad normalizer can erase the exact difference you need to see. For example, sorting all arrays may hide a changed posting order that affects downstream processing, while comparing raw generated identifiers creates meaningless failures.

Production traffic needs careful capture. Remove or tokenize sensitive fields before they enter a general test environment. Preserve relationships that behavior depends on, such as the same customer appearing in several requests. A bag of disconnected, over-scrubbed examples may look safe while losing session, ordering, and retry semantics. In regulated environments, keep capture, execution, and artifacts inside the approved perimeter.

Coverage should describe behavior represented, not only the number of replays. Partition cases by operation, outcome, important flags, error category, data shape, and boundary condition. Ten thousand successful reads cannot compensate for having no failed update, duplicate submission, month-end transition, or restart after a partial write. Track empty partitions as explicit review debt.

Run the harness continuously during generation and after human edits. A final replay pass catches well-intended cleanup that changes behavior after the model work is done. Keep mismatches as durable records with a disposition: new defect, old defect intentionally preserved, old defect intentionally corrected, expected policy change, or harness fault. An unexplained mismatch is not a test failure to waive; it is an unfinished decision.

The old system is a witness, not an oracle

Matching the old implementation exactly can preserve defects, insecure defaults, and workarounds whose original reason has disappeared. Differential equality is evidence of compatibility, not evidence of correctness. You need an independent rule for any behavior with serious consequences.

This distinction becomes concrete when the old system accepts an impossible state. Suppose a batch can mark an invoice paid after writing the ledger entry but before confirming the payment reference. Production replay shows that sequence, so the rewrite reproduces it. A parity-only gate reports success. A conservation property may still pass because the money balances. Only an invariant requiring a confirmed reference for the paid state exposes the invalid transition.

Do not silently fix every oddity either. A defect may have become an interface. Downstream reports might expect an unusual rounding result, operators might use a particular error code to route work, or clients might retry only after a specific status. Correcting that behavior without a migration decision can cause a wider failure than preserving it temporarily.

Use a difference register with five fields: observed behavior, independent expected rule, affected consumers, chosen disposition, and owner. If the new version intentionally differs, add a test for the new rule and a release note or operational change where appropriate. If the defect must remain for compatibility, isolate it behind a named compatibility rule so future maintainers do not "clean it up" by accident.

The popular recommendation to make the new suite pass all old tests is therefore incomplete. Old tests are excellent witnesses for known behavior, but they carry the same blind spots and mistaken assumptions as the system around them. Treat them as one evidence set. Add properties derived from business rules, negative tests derived from threat analysis, and state-transition checks derived from data constraints.

A reviewer should be suspicious when parity reaches 100 percent too easily. It may mean the boundary is too narrow, the fixtures omit hard cases, or the comparator ignores too much. Inspect a sample of raw old and new traces, including failures, before trusting the aggregate. Good evidence remains legible when you open it.

Invariants must survive every entry and exit

An invariant is a condition that must hold across all valid system states, not merely an assertion attached to a happy path. Put invariant checks at boundaries where state enters, changes, and leaves: API handlers, message consumers, transaction commits, file imports, job checkpoints, and serializers.

Classify invariants by scope. Local invariants constrain one value, such as a quantity that cannot be negative. Aggregate invariants relate a collection, such as debits equaling credits. Temporal invariants constrain order, such as approval preceding disbursement. Authority invariants constrain actors, such as a user never approving a request they created. Recovery invariants constrain retries and restarts, such as one durable effect per idempotency token.

Database constraints provide strong, independent enforcement for some rules. A unique constraint can stop duplicate identifiers even if every caller makes the same mistake. A foreign key can prevent an orphan. A check constraint can reject an invalid status and field combination. Application tests should still exercise the resulting error path because a technically safe rejection can become an operational outage if the worker retries forever.

State machines deserve explicit transition tests. Generate valid sequences and assert that each transition keeps all invariants true. Then generate one invalid transition at every state and require rejection without partial state change. A final-state assertion alone misses transient damage, duplicate messages, and writes that survive an error response. Capture state before and after the attempted transition.

Place runtime assertions according to consequence. Cheap checks on untrusted input can run on every request. Expensive cross-table reconciliation may run at a commit boundary, in a shadow process, or as a deployment gate. Do not remove a useful invariant merely because it costs too much in the hottest path; move it to the nearest place where it still catches the fault before damage spreads.

Monitor invariant violations by identity, not as generic exceptions. The alert should name the rule, affected entity, operation, and release. Avoid logging the sensitive payload used to detect it. A count without identity cannot guide rollback or diagnosis, while a full payload may create a separate data exposure.

Reviewers should ask who owns each invariant. If everyone agrees that balances must reconcile but no test, constraint, runtime check, or scheduled reconciliation enforces it, the invariant exists only in conversation. Assign at least one executable control and one clear response for every rule whose violation matters.

Human review belongs at semantic choke points

People should read code where intent cannot be inferred from input-output examples alone or where a small decision controls a large consequence. These semantic choke points include authorization, cryptographic use, transaction boundaries, schema migration, concurrency control, data deletion, external side effects, error recovery, and every adapter that normalizes evidence for tests.

Read all allow paths in authorization code. A large denial suite helps, but policy meaning often depends on role inheritance, tenant boundaries, delegated authority, and default behavior when context is missing. Review the rule source beside the code. Verify that the default denies access, that cached decisions carry the right scope, and that logs do not reveal protected data.

Read transaction and retry boundaries as one unit. Find the point where an operation becomes durable, then follow every error that can occur before and after it. Check whether a retry repeats a write, sends a second message, or observes partially committed state. Generated code often handles each error locally and plausibly while missing the cross-function sequence that produces duplication.

Read the comparator and harness more skeptically than an ordinary test helper. The evidence is only as honest as the machinery that labels two runs equal. A model that wrote production code should not be the sole author and judge of its equivalence policy. A person should approve ignored fields, tolerances, canonical ordering, and error-category mappings.

Read dependency and generated configuration changes. Tests may never exercise an unsafe parser option, an overly broad network permission, a disabled certificate check, or an unbounded worker pool. Inspect lockfile movement, build scripts, container privileges, database grants, deserialization settings, timeouts, and resource limits. These choices can change the attack surface without changing ordinary functional output.

Sample ordinary code only after choke points are covered. Use risk-weighted sampling: inspect a complete vertical path for a few representative operations, plus code with high complexity, unusual model uncertainty, repeated manual repair, or weak test reach. Random line sampling creates the appearance of diligence but rarely follows a decision far enough to judge it.

Human review still ends with a written decision. The reviewer should name what was inspected, which evidence was trusted, what was excluded, and what residual risk remains. "Looks good" is not an approval record for a generated change that nobody could read completely.

Evidence needs its own chain of custody

Large generated changes require a review ledger that connects requirements, tests, replay partitions, mismatches, human findings, and the exact build under review. Without that connection, teams accumulate thousands of passing results that may belong to different commits, fixtures, normalizers, or dependency versions.

Give every artifact a stable identity. Record the source revision, generated revision, build inputs, harness version, fixture snapshot, random seeds where relevant, and environment configuration that affects behavior. Hash captured inputs after approved redaction so reviewers can tell whether two reports used the same corpus without retaining forbidden raw data in the report.

Map each contract rule to evidence. One rule may have a property test, a database constraint, a replay partition, and a manual inspection note. Another may have only a manual sign-off because automation cannot observe the business judgment. Empty mappings are useful: they show exactly where approval relies on assumption. A dashboard full of green counts hides that gap.

Quarantine flaky evidence instead of rerunning until green. Record the failing case, determine whether nondeterminism comes from the product or harness, and fix the cause. Repeated reruns change the meaning of the gate from "the build passed" to "one attempt passed," which is a much weaker claim. If a test cannot yet gate, label it non-gating and keep its failures visible.

Preserve counterexamples and mismatch decisions with the change. They explain behavior better than a generated comment because they contain an input, an observation, and an approved outcome. When a later rewrite changes the same surface, those artifacts become a compact institutional memory that does not depend on the original reviewer still being present.

CodeHero uses this principle during legacy rewrites by checking behavior against recorded production traffic with a parity harness, while the architecture changes instead of copying the old structure line by line. That claim is useful only when the replay corpus, comparison policy, and mismatch dispositions remain open to customer review.

The ledger should be reproducible by someone who did not generate the code. If a second engineer cannot run the specified checks and obtain the report, the team has a presentation, not evidence. Reproducibility also limits dependence on model explanations, which can sound coherent without matching the compiled build.

Evidence expires when the system around it changes. A replay report collected before a schema migration, dependency update, or comparator edit does not approve the later build. Define invalidation rules in the ledger: changes to contract code rerun affected properties, changes to normalization require review of earlier equalities, and changes to persistence rerun recovery sequences. This stops teams from treating a once-green report as permanent permission.

Keep evidence close to the owning surface. A giant release report makes it hard to tell which check protects which behavior, and it encourages reviewers to approve the bundle as one object. Surface records let a team replace one component, rerun its proof, and leave unrelated evidence alone. They also expose shared controls. If five surfaces depend on one clock adapter or comparison rule, that shared component deserves direct review because one mistake can corrupt five conclusions.

Review the evidence process after escaped defects. Ask which contract statement was missing, which generator could not produce the case, which replay partition was empty, which invariant failed to exist, or which human inspection skipped the decisive branch. Add the smallest durable control that would have caught that class of error. Do not respond by demanding that reviewers read more arbitrary lines. That reaction increases cost without repairing the blind spot.

An evidence retention policy should match the need to reproduce approval and the sensitivity of captured data. Keep minimized counterexamples and metadata when possible. Restrict or discard raw production payloads according to the customer rules that govern them. The ability to explain a decision never grants permission to retain every byte used to reach it.

Approval should state the residual risk

A generated rewrite is ready when the required behavior has independent evidence, dangerous decisions received human inspection, and remaining uncertainty fits the system's failure budget. Completion is not a percentage of lines read. It is a defensible statement about what can still go wrong and how the team will detect or contain it.

Set gates by consequence. For a low-impact internal converter, representative examples, parsing properties, and rollback may suffice. For money movement, access control, regulated records, or irreversible deletion, require independent contract sources, negative tests, invariant enforcement, differential coverage of important partitions, direct choke-point review, and an operational response for violations.

Do not combine all evidence into one score. A high replay match rate cannot cancel an unreviewed authorization default. Strong property coverage cannot cancel a comparator that masks ordering. A careful code read cannot cancel the absence of retry tests. Keep veto conditions visible so an attractive aggregate does not bury a severe gap.

Use a short approval record:

1. Name the build, contract version, and evidence snapshot.
2. List required gates and their results.
3. List every accepted mismatch and non-gating test.
4. Name the human-reviewed choke points and reviewers.
5. State residual risks, detection controls, rollback limits, and owners.

Stop the release when a high-consequence behavior has no independent oracle, invariant, or direct review. Sometimes the honest decision is to reduce scope: migrate read paths before write paths, shadow decisions without acting on them, or keep one hazardous operation on the old implementation until its contract is understood. That is engineering control, not a failure of ambition.

Model output changes the economics of writing code, but it does not change who carries the consequence of a bad release. Ask reviewers to approve evidence they can reproduce and risks they can name. Do not ask them to bless a volume of text that no person could responsibly absorb.