Whole codebase reading beyond the context window

A model that accepts a million lines is not thereby capable of understanding a million-line system. Capacity answers whether the text fits. It says nothing about whether the model will find the right definition, connect an indirect caller to a side effect, or notice that a JCL step changes the meaning of the COBOL program it launches.

This distinction matters most in legacy rewrites. A plausible local translation can compile and still break month-end close because the old behavior lives across source files, copybooks, database triggers, job control, screen validation, and production conventions. Feeding all of that into one prompt replaces an omission problem with a discrimination problem. The context contains the answer, but the model has no reliable structure for deciding which facts control the change.

Whole codebase reading is therefore a systems problem, not a prompt-size feature. It requires a durable model of the repository, explicit dependency traversal, staged reasoning, and executable evidence that the generated system behaves like the old one.

A context window measures capacity, not comprehension

A long context window tells you the maximum input a model can accept under specified conditions. It does not promise uniform recall, stable reasoning across the window, or correct joins between facts that sit far apart. Those are separate capabilities, and treating them as one metric is the first design error.

Nelson Liu and his coauthors made the positional problem concrete in the paper Lost in the Middle. On multi-document question answering and key-value retrieval, model performance often peaked when relevant material appeared near the beginning or end of the input and fell when the same material moved into the middle. The paper does not prove that every current model fails on every long program. It does prove that successful ingestion is a poor proxy for dependable use.

Code makes this harder than ordinary prose. A repository contains thousands of repeated shapes: getters, validators, record layouts, error branches, generated files, and near-identical batch steps. The model must distinguish facts that look alike while following relationships that may never appear as adjacent text. A CALCULATE-TAX paragraph and a CALCULATE-TAX-OLD paragraph can share most tokens and have different callers. Similarity helps find both; it does not decide which one governs the transaction.

Token counts also hide representation costs. Source text is only one layer. A useful analysis needs symbol identities, call edges, data flow, build metadata, schemas, configuration, tests, and observations from running software. If a vendor says its context window fits the repository, ask what was excluded, how files were ordered, how cross-language edges were encoded, and how the system tests claims made from that input. The window size alone answers none of those questions.

The practical acceptance test is simple: move the decisive file to a different position, add irrelevant but similar files, and repeat the task. If the answer changes, the system read a sequence, not a codebase.

Another test is compositional. Ask where a field originates, where it changes, and which external effect depends on its final value. Then ask the same questions separately and compare the assembled path. A reader that answers each local question but cannot preserve identity across the chain has search capability, not system comprehension. This failure is easy to miss because every individual answer can sound correct.

Retrieval misses relationships that vocabulary cannot express

Chunk retrieval works for questions whose answer resembles the query, but program behavior often depends on relationships rather than shared words. The caller may use a generic name, dispatch through a table, construct a procedure name, publish an event, invoke a stored procedure, or hand control to a scheduler. A vector search for the business concept can retrieve the callee and still miss the caller that supplies the decisive flag.

Consider a billing rule implemented in COBOL. The visible paragraph reads an account class and calculates a fee. A JCL step selects an alternate input dataset on the final business day. A copybook overlays two fields in the record, and a PL/SQL trigger suppresses the fee for migrated accounts. None of those artifacts needs to repeat the wording from the ticket. Retrieving the paragraph alone produces a clean, wrong TypeScript function.

SWE-bench captured a related fact at a smaller scale: real fixes often require coordinated changes across functions, classes, and files. Its original setup compared retrieval with access to the files changed by the human patch. That is a useful warning. Better generation cannot recover evidence the retrieval stage never supplied, and an oracle file set is not available on a live migration.

The field also blurs search recall with behavioral completeness. Search recall asks whether a known relevant item appears in the returned set. Behavioral completeness asks whether the system found every artifact needed to explain an observed outcome. You can measure the first with labeled chunks. You can establish the second only by tracing and testing the behavior.

A repository reader should combine several routes into the same evidence store:

• lexical search for exact identifiers, literals, record names, and error codes;
• semantic search for concepts expressed with different vocabulary;
• symbol resolution and call traversal for explicit program structure;
• data-flow and schema edges for values that cross procedure boundaries;
• runtime traces for dynamic dispatch, configuration, and external effects.

These are not five competing retrievers. Each reveals a different failure class. The system should preserve why it selected an artifact, which edge led to it, and what remains unresolved. A ranked bag of chunks without provenance invites the model to turn retrieval confidence into invented certainty.

Chunk boundaries create another silent loss. A procedure declaration may land in one chunk while its precondition, error handler, or adjacent data definition lands in another. Enlarging chunks keeps more local context but reduces retrieval precision and consumes more of the prompt. Overlap copies text without restoring program structure. Parse-aware chunks improve the tradeoff, yet even a complete function cannot reveal a scheduler condition or trigger effect elsewhere. Retrieval needs graph expansion after the first hit, with stopping rules based on resolved questions rather than an arbitrary top-k.

Attention dilution survives a larger prompt

Adding more relevant material can make an answer worse when the model must select among too many plausible facts. This is attention dilution in the engineering sense: the decisive evidence competes with boilerplate, duplicate implementations, dead branches, generated code, comments that describe an earlier release, and tests that encode obsolete behavior.

The common recommendation is to put the whole repository into the prompt and tell the model to inspect it carefully. It is popular because it removes a visible pipeline component. There is no index to tune and no retriever to blame. The simplicity is cosmetic. The model still performs selection, but now it does so inside an opaque inference pass where you cannot inspect missed candidates or replay a traversal.

Repository order then becomes accidental policy. Alphabetical file order favors some modules. Concatenating dependency order requires a dependency graph before prompting, which concedes the need for analysis. Putting likely files at both ends exploits a benchmark pattern rather than establishing comprehension. Repeating important files wastes capacity and can overweight stale duplicates.

LongBench v2 included repository understanding among tasks with very long inputs and found that direct answering remained difficult. Its more interesting lesson is methodological: longer reasoning and extra inference effort can matter as much as the advertised input size. A migration architecture should take that idea further by externalizing intermediate work. It should not ask one generation to discover the system, decide the target design, produce code, and certify parity in the same breath.

You can expose dilution with a small evaluation. Choose a change whose controlling facts span a caller, a configuration value, and a side effect. Run it with the minimal evidence set, then add ten groups of plausible distractors from the same repository. Record the claimed call path, cited symbols, patch, and test result for every run. The purpose is not to calculate a universal score. It is to discover whether additional context changes the system's account of behavior without any change in the program itself.

Prompt summaries do not cure this on their own. Summarization is a lossy decision about relevance made before the later task is known. A summary written for architecture discovery may omit rounding rules needed for a defect fix. Keep summaries as navigational aids, attach them to the source entities they describe, and let a task reopen the underlying evidence. Never allow a summary to become the only surviving account of a module.

The repository needs a typed map before generation

Whole-codebase analysis starts by building a typed, queryable representation of the system. A flat vector index is useful inside that representation, but it cannot be the representation. The durable unit is an entity with identity, location, language, and edges to other entities.

At minimum, the map should represent files, symbols, entry points, calls, reads and writes, schemas, jobs, screens, tests, configuration keys, and external interfaces. Edges need types because calls, loads dynamically, writes field, and runs after imply different reasoning. Confidence and origin belong on each edge. A compiler-derived call edge should not look identical to a model-inferred association.

Legacy repositories make language boundaries part of the semantics. JCL selects programs and datasets. CICS maps connect screens to fields. RPG programs depend on display files. VB6 forms contain event wiring outside ordinary procedures. Classic ASP mixes markup, script, session state, and database access. PL/SQL packages can hide effects behind triggers and synonyms. A parser for the main language sees only a fraction of the executable system.

The map also needs negative and unresolved facts. If a dynamic call target cannot be resolved statically, store the expression and candidate set instead of choosing one silently. If two copybooks define the same record name under different build options, retain both variants and the condition that selects them. Unknowns should become trace requests or parity cases, not prose guesses.

A compact evidence record might look like this:

{
  "claim": "late fee is suppressed for migrated accounts",
  "path": ["BILLJOB", "FEE-CALC", "ACCT_FEE_TRIGGER"],
  "evidence": ["jobs/bill.jcl:88", "src/fee.cbl:412", "db/account.sql:219"],
  "conditions": ["RUN_MODE=MONTH_END", "account.migrated=true"],
  "unresolved": ["dynamic dataset alias at BILLIN"]
}

That record is not a prompt trick. It is an inspectable claim that another stage can challenge. When generation changes FEE-CALC, the system can locate affected entry points, request the unresolved dataset binding, and select tests that exercise the condition. Without the map, each prompt must rediscover these facts and will rediscover them differently.

The representation must also be versioned. Generated code, source commits, schema snapshots, and recorded traces need a common revision boundary. Otherwise the graph can connect a caller from Tuesday to a callee replaced on Wednesday and describe a system that never existed. Incremental analysis should invalidate derived edges when their source changes, then recompute dependent claims. Cache reuse without provenance is fast until it certifies the wrong build.

Whole-codebase reading must be staged

A reliable reader separates discovery, behavioral modeling, target design, implementation, and verification. The stages can iterate, but each produces artifacts that constrain the next one. This prevents a fluent generation from erasing uncertainty discovered earlier.

Discovery inventories languages, build paths, entry points, schemas, jobs, and external boundaries. It parses what can be parsed, records failures, and connects cross-language artifacts. The output is a repository map plus an explicit list of blind spots.

Behavioral modeling starts from observable operations rather than files. For each entry point, the system traces conditions, state transitions, outputs, and side effects. It groups duplicate routes that implement the same rule and separates similar routes whose preconditions differ. The result is a set of behavior claims tied to evidence.

Target design then decides where those behaviors belong in Go services, Rust kernels, TypeScript clients, or Postgres. This is where a rewrite differs from transliteration. A paragraph-per-function conversion can preserve control flow while carrying forward global state, file coupling, and scheduler assumptions. The design should preserve behavior at the boundary while changing internal structure deliberately.

Implementation consumes bounded work packets derived from the map. A packet includes the target behavior, relevant source entities, upstream callers, downstream effects, constraints, unknowns, and required parity cases. The model may ask for more evidence. It should never fill a missing edge with a confident guess.

Verification runs continuously, not after a heroic final merge. Failures update the behavioral model or expose a target defect. That feedback matters because source analysis alone cannot decide whether an odd branch is dead code, a regulatory exception, or a path reached only by production data.

This staged design spends model attention where it has a defined job. Models remain good at interpreting messy code and proposing implementations. The surrounding system supplies identity, memory, traversal, and a judge that does not grade prose.

Human review should follow the same artifacts. Engineers should review disputed behavior and target contracts, not scroll through thousands of generated lines hoping to recognize a semantic change. A stage is ready to advance when its claims have evidence, its unknowns have owners, and its required checks exist. Approval of polished prose has no place in that gate.

Runtime evidence catches what static reading cannot

Static analysis describes possible behavior. Recorded execution shows behavior that actually occurred under particular inputs. A serious rewrite needs both, because each covers the other's blind spots.

Static analysis can enumerate branches that no recording reached. It can find a write hidden behind an error path, a batch job that runs quarterly, or a screen action absent from recent traffic. Runtime traces resolve dynamic calls, real configuration values, dataset aliases, SQL produced at execution, and the order of external effects. Neither source should be promoted to the whole truth.

Teams often propose test coverage as the substitute. Existing tests are useful evidence, but old systems frequently have narrow unit suites, environment-dependent integration scripts, or no automated tests around the behaviors that keep the business running. Passing those tests proves compatibility with the tests, not the production system.

A parity harness gives the comparison a concrete shape. Capture an input at a stable system boundary, replay it against the old and new implementations, normalize nondeterministic values, and compare outputs plus side effects. A result can be represented without interpretation:

case: invoice/month_end/migrated_account
old: status=200 body_sha256=7c... ledger_rows=0 notices=1
new: status=200 body_sha256=7c... ledger_rows=1 notices=1
verdict: FAIL side_effect.ledger_rows expected 0 got 1

That failure points back to the fee suppression claim and its evidence path. The team can inspect whether the new code missed the trigger behavior, whether replay setup failed to mark the account as migrated, or whether normalization hid a source difference. A generic assertion such as "outputs differ" would send the model hunting across the repository again.

Recorded traffic requires care. Secrets and personal data need handling appropriate to the environment, and destructive effects must be isolated or virtualized during replay. Coverage also needs an inventory: which entry points, business periods, error classes, and configuration variants appear in the corpus? A million repeated happy-path requests do not cover the rare close procedure.

Comparison rules require the same scrutiny as captured inputs. Timestamps, generated identifiers, unordered rows, and environment-specific paths may need normalization, but every normalization removes a possible difference from view. Store each rule as code, explain why it is safe, and test that it does not mask a meaningful change. If the old system emits records in an order consumed by another batch, sorting them before comparison would manufacture parity.

Parallel reading requires shared state, not isolated agents

Splitting a repository across agents saves wall-clock time only when they write into a shared, consistent model of the system. Ten independent summaries create ten vocabularies, duplicate conclusions, and gaps at the boundaries each worker assumed somebody else would inspect.

Parallel workers should claim explicit graph regions or questions. One may resolve job-to-program edges while another maps database effects and another classifies external endpoints. Each writes entities, typed edges, evidence, and conflicts into the same store. Entity identity must be stable so that CUSTOMER-REC in a copybook and the buffer passed through three programs can be recognized as the same layout under the same build condition.

Conflicts are useful output. If static parsing says one caller reaches a procedure but traces show a second dynamic route, the system should retain both and schedule resolution. If two agents attach different business meanings to the same field, a reviewer can see the disagreement before generated code embeds either interpretation.

Concurrency also needs dependency-aware scheduling. There is little value in generating a replacement for a downstream module while its input contract remains disputed. Work can proceed on independent components, but changes that cross unresolved edges should wait or carry explicit provisional interfaces.

The million-line claim becomes credible only under this model. CodeHero reads every language in the tree in parallel and uses a shared whole-system representation rather than treating each prompt as an isolated reading session. That is the architectural fact that matters; the number of concurrent agents by itself says nothing about comprehension.

The review surface should show how a conclusion was assembled: source locations, graph path, trace cases, competing interpretations, and the downstream code that consumes it. An executive progress bar is not enough for the engineer who must decide whether a payment rule survived the rewrite.

Shared state does not mean one enormous prompt shared by every worker. It means a transactional evidence store with stable identifiers and version checks. Workers can operate on bounded slices, then merge facts only when their source revision still matches. This avoids both extremes: isolated agents that forget one another and a global conversation that grows until nobody can tell which claim remains current.

Architecture quality and behavioral parity are separate gates

A rewrite can match observed behavior and still reproduce the old architecture badly. It can also look beautifully modern while changing the system at its boundaries. These are separate gates, and neither should be allowed to compensate for the other.

Behavioral parity evaluates requests, files, messages, database changes, timing-sensitive ordering where relevant, and error behavior. Architecture review evaluates module boundaries, ownership of state, target-language idioms, failure handling, observability, deployability, and removal of obsolete coupling. A weighted score that blends both can hide a fatal miss. Require each gate to pass.

This is another distinction that repository demos often blur. Generating syntactically correct Go from COBOL demonstrates translation. Replacing batch-coupled state with explicit services while preserving close behavior demonstrates modernization. The second requires the repository map, operational constraints, and parity evidence; code generation alone cannot establish it.

Set review criteria before implementation begins. For a service boundary, record allowed callers, request and response contracts, transaction ownership, retry behavior, and parity cases. For a Rust numeric kernel, record input domains, rounding, overflow, and reference vectors. For a TypeScript client, record validation ownership and server-side enforcement. These are engineering decisions, not details to infer from whichever source chunk happened to rank first.

Do not accept "the model saw the whole repository" as evidence for either gate. Ask for the path from entry point to changed behavior, the unresolved edges on that path, the target decision that replaced the old coupling, and the replay cases that passed. If the system cannot produce those four things, it has produced code without a defensible account of the system.

Operational equivalence may include more than response bodies. Batch cutoff times, locking order, retry boundaries, rounding modes, file encodings, and the timing of messages can all be contractual when another system depends on them. The behavioral inventory should mark which properties must match exactly, which can vary within a bound, and which are intentional changes approved by an owner. Calling every difference a bug blocks modernization; calling every inconvenient difference an improvement abandons parity.

Evaluate the reader by perturbing evidence

The best evaluation for a repository reader tests stability under changes that should not affect the answer. A single successful demonstration says little because the file order, query wording, and selected chunks may have favored that case.

Build a set of repository questions with answers supported by multiple artifacts. Include direct calls, dynamic dispatch, configuration-selected variants, database side effects, batch ordering, dead code that resembles live code, and cross-language boundaries. For each question, record the expected evidence path, not just a prose answer.

Then perturb the input in controlled ways:

1. Reorder files and graph traversal results without changing content.
2. Add near-duplicate dead implementations and stale comments.
3. Rename local identifiers while preserving structure and behavior.
4. Remove one required artifact and check whether the system reports uncertainty.
5. Add a runtime trace that contradicts the static hypothesis.

Score evidence recall, path correctness, uncertainty handling, generated change, and parity outcome separately. A reader that reaches the right answer through the wrong path is fragile. A system that refuses to conclude after evidence removal may be better than one that preserves its original answer.

Cost and latency belong in the evaluation, but not as substitutes for accuracy. Record parsing time, index update cost, graph traversal, model calls, replay time, and human review time. Incremental changes should update affected entities and tests rather than force a complete reread. That is how whole-codebase reading becomes an operational architecture instead of an expensive launch demonstration.

The acceptance question is not whether one model call can hold a million lines. It is whether the system can explain a behavior, survive irrelevant context, expose missing evidence, produce a deliberate target design, and prove the new implementation against the old one. A larger window may help at several points. It never removes the need for the machinery around it.