Can automated repository documentation be trusted?

Repository archaeology can recover a surprising amount of documentation, but it cannot recover intent merely by reading more files. A module map, a static call graph, a candidate data model, and much of a batch dependency graph are evidence-backed outputs. Labels such as "customer," claims about when a job is safe to rerun, and explanations of why a branch exists are hypotheses until another source confirms them.

That boundary matters because generated documentation tends to look equally confident on both sides of it. I have watched teams accept a polished diagram, plan a rewrite around it, and discover late that a scheduler rule or a dynamically selected program carried the behavior that mattered. The cure is not to reject automation. It is to make every generated statement carry its evidence, method, and known blind spots.

A repository can prove structure, not purpose

Automated repository documentation is trustworthy when it reports observable structure and states exactly how it observed it. Files, declarations, imports, build targets, SQL references, JCL statements, and literal configuration keys all leave inspectable traces. A tool can enumerate them, connect them, and point back to the lines that support each connection.

Purpose is different. A table named ACCT_MST might hold customer accounts, internal ledger accounts, or temporary reconciliation state. The name suggests an interpretation but proves none. A routine named VALIDATE may reject bad input, apply an authorization rule, or only check field widths. Comments can help, but stale comments are repository content too, not privileged truth.

I use three confidence classes in generated documentation:

• Observed means the repository contains direct evidence, such as an import, an EXEC PGM, or a foreign key declaration.
• Inferred means several observations support a conclusion, such as grouping programs into a billing module because they share tables and entry points.
• Unresolved means the repository does not settle the question, even if one interpretation looks likely.

Every node and edge should also cite its origin as a path plus a line or statement range. Without provenance, reviewers cannot distinguish a parser result from a model's guess. A generated sentence such as "INVOICE writes AR_LEDGER" is useful only when a reader can inspect the INSERT, stored procedure call, or record write behind it.

The distinction also prevents a common category error: completeness and correctness are separate. A parser may correctly find every direct call in the files it understands while missing calls made through configuration. Its output is correct within a declared scope but incomplete for the running system. Documentation should report both dimensions rather than roll them into a vague confidence score.

Build the module map from several kinds of edges

A credible module map combines directory structure with dependency and data access evidence. Treating top-level folders as modules works only in unusually disciplined repositories. Legacy trees often group files by deployment package, author habit, copybook location, or a migration that stopped halfway through.

Start with declared units: projects, packages, libraries, programs, forms, stored procedures, batch jobs, and build targets. Then collect typed edges between them. Useful edge types include imports, calls, includes, compiles_into, reads, writes, submits, and generates. Preserve the type. A shared table is weaker evidence of a module boundary than a build target, and a textual include is not the same relationship as a runtime call.

The first artifact should be a machine-readable inventory, not a picture. For example:

{"unit":"billing/post_invoice.cbl","kind":"cobol_program","declares":["POSTINV"],"includes":["ARREC"],"reads":["CUSTOMER"],"writes":["AR_LEDGER"],"evidence":["billing/post_invoice.cbl:18-146"]}

Generate diagrams and prose from that inventory. This makes changes reviewable: when a program moves or a parser improves, the source record changes first and every view follows. It also lets a team query the documentation instead of staring at a wall-sized graph.

Clustering needs restraint. Connected components, package declarations, naming prefixes, ownership files, and deployment units can propose boundaries. They should not silently manufacture them. If AR* programs share records and deploy together, call that an inferred billing cluster and list the rule that created it. A human can then accept, split, or rename it.

Generated module prose should answer practical questions: What enters this unit? What can it call? Which data does it own or merely touch? How is it built and deployed? What other unit would break if its interface changed? A colorful rectangle that answers none of these is decoration.

Static call graphs are useful and predictably incomplete

A static call graph can reliably capture calls whose targets the source resolves directly. It can also provide reverse edges, which are often more useful during change planning: instead of asking what a function calls, engineers ask who can reach the function they intend to replace.

The GNU cflow manual makes this exact direct-versus-reverse distinction for C. It also exposes symbol filtering and preprocessing controls. That qualification matters. A graph depends on the language front end, preprocessing configuration, build flags, and chosen entry points. Running a parser over every file with default settings is not equivalent to analyzing the program that production builds.

Dynamic dispatch creates the first large gap. Function pointers, reflection, dependency injection, COM dispatch, generated proxies, COBOL dynamic CALL, and program names assembled from data can hide the target. A source scanner may record the dispatch site and the expression used to select a target, but it should create an unresolved edge rather than guess one destination.

External execution creates another gap. Shell commands, job submission APIs, database triggers, message consumers, and files polled by another process cross boundaries that a language-specific graph rarely sees. The repository may contain both ends without containing a direct symbol edge between them.

For each edge, record the resolution mode:

• static when syntax and symbol resolution identify the target.
• configured when a manifest or setting names the target.
• observed when a runtime trace records the target.
• possible when dispatch analysis yields a bounded set.
• unknown when the call site exists but the destination does not resolve.

Do not delete unknown edges to make the picture cleaner. They are often the most useful items in the document because they identify where migration work needs tracing or an operator interview. A graph with 100 percent resolved calls in a reflective or configuration-heavy system is usually advertising its blindness.

The data model has three competing versions

The repository can yield a declared schema, an accessed schema, and an implied business model. These overlap, but teams get into trouble when documentation presents them as one thing.

The declared schema comes from DDL, migration files, ORM mappings, record definitions, copybooks, validation rules, and database metadata snapshots committed to the tree. It can identify tables, columns, types, indexes, declared keys, nullability, and constraints. PostgreSQL documents information_schema.columns as a portable view of column information while noting that PostgreSQL-specific types ultimately live in pg_catalog. That is a useful warning: even database metadata has a portable layer and a vendor-specific layer.

The accessed schema comes from code. SQL strings, query builders, file I/O, data access classes, screen bindings, and report definitions show which fields each program reads or writes. This view exposes tables with no declared foreign keys but consistent joins, and columns that exist in DDL but no longer appear in repository code.

The implied business model attaches meaning: an account belongs to a customer, a status of C means closed, or an effective-date pair models a policy period. Automation can propose these relationships from names, joins, checks, and repeated transformations. It cannot promote them to fact without a glossary, test, operator confirmation, or observed data.

A useful extraction keeps disagreements visible:

SELECT table_schema, table_name, column_name, data_type, is_nullable
FROM information_schema.columns
WHERE table_schema NOT IN ('pg_catalog', 'information_schema')
ORDER BY table_schema, table_name, ordinal_position;

Compare that output with repository references rather than choosing one as canonical. If code selects legacy_code but the captured schema lacks it, you may have a stale snapshot, conditional SQL, or a different production schema. If the DDL declares a foreign key that no code follows, the constraint still matters. The mismatch is a finding, not a nuisance to merge away.

Field-level lineage deserves the same caution. Direct assignments and named transforms can support a lineage edge. A stored procedure invoked through a generic gateway, a spreadsheet macro, or an operator-edited export breaks the chain. Mark the break. Do not draw a continuous arrow through missing evidence.

Batch dependencies live outside the JCL

A repository can derive much of a batch graph, but JCL or scripts alone rarely contain the production schedule. They show programs, steps, procedures, datasets, return-code branches, and explicit job submissions. Calendars, predecessor rules, resources, overrides, and recovery actions often live in a scheduler database or operations configuration.

IBM's documentation for Workload Scheduler describes job predecessors and successors, including conditions based on job status or return codes. Its JCL repository documentation also says the scheduler keeps a copy of JCL for jobs it submits in the current plan. Those facts expose an important boundary: the submitted JCL is an execution artifact, while the current plan carries orchestration state. A Git repository containing only one side cannot prove the whole dependency graph.

Within the repository, extract at least four edge classes: step order, program execution, dataset flow, and explicit condition. A producer-consumer edge inferred because one job writes a dataset and another reads it should remain inferred. Dataset names may be generation-based, symbolic, overridden at submission, or shared for reasons unrelated to ordering.

Represent the result in a form that can admit missing sources:

job: CLOSE_AR
steps:
  - exec: EXTRACT_AR
    writes: [AR.CLOSE.GDG(+1)]
  - exec: POST_AR
    when: EXTRACT_AR.RC <= 4
external_predecessors:
  - name: LOAD_RATES
    source: scheduler_export
unresolved:
  - "Symbolic HLQ is supplied by the submission profile"

That final field is part of the documentation, not an embarrassment. It tells the migration team which artifact to request next.

Control cards and scheduler exits deserve special attention. A one-line JCL step may feed hundreds of lines of parameters from a dataset maintained outside source control. A scheduler exit may rewrite variables or select a procedure library. Treat referenced but absent control data as an external dependency with an owner and retrieval task.

Runtime evidence changes the answer

Static extraction describes what the repository permits. Runtime evidence shows what selected executions actually did. Neither view should impersonate the other.

Recorded traces, database statement logs, job histories, message metadata, file catalogs, and production traffic can confirm dynamic targets and prioritize paths. They can show that a configurable dispatcher chose three of twenty possible programs during the capture window. They cannot prove the other seventeen are dead. Absence in a trace means "not observed in this sample," not "unreachable."

The strongest documentation stores static and observed edges separately, then offers their intersection and differences. Consider a call site with a configured target list of RATEA, RATEB, and RATEC. A month-end trace sees RATEA and RATEC. The correct record keeps all three possible targets, marks two as observed, and records the capture period and environment. Deleting RATEB from the graph would turn limited evidence into a false claim.

Production traffic also helps verify behavior during a rewrite. Inputs and outputs can become parity cases, provided the capture removes or protects sensitive data and preserves the variables that drive behavior. A passing parity case proves agreement for that case. It does not establish general equivalence, so the documentation should state coverage by entry point, branch, data shape, and error class where those measures are available.

This is where repository analysis becomes more than a prettier index. A static graph tells you where to place probes. Traces tell you which unresolved edges deserve attention. Differences between old and new executions reveal undocumented behavior, and those findings can flow back into the evidence store.

Never let a runtime overlay erase the static base. Quiet quarterly jobs, failure handlers, regulatory extracts, and disaster procedures may not appear during an ordinary capture. Teams often call them dead because the normal trace is silent, then learn their purpose during the one event when they run.

Generated prose needs citations and expiration

Generated prose becomes trustworthy when a reviewer can challenge each material claim without reverse-engineering the generator. Put source references next to claims and add the extraction revision, tool version, configuration, and generation time to the document metadata.

The repository commit is the document's effective date. If the main branch changes, the generated document is stale even when its prose still sounds plausible. Regenerate it in continuous integration or label it clearly with the commit it describes. I prefer failing a freshness check over serving a silent mixture of old diagrams and new code.

Claims need different citation shapes. A structural claim can cite source lines. A runtime claim should cite a trace set or job-history export plus its observation window. A business definition should cite an approved glossary, rule, test, or named reviewer. When no citation exists, label the statement as a question or inference.

Use a small review ledger rather than burying uncertainty in prose:

ID       CLAIM                                  CLASS       EVIDENCE
DOC-041  POSTINV writes AR_LEDGER               observed    post_invoice.cbl:88
DOC-042  AR_LEDGER is the accounting system     inferred    table name, 6 writers
DOC-043  CLOSE_AR may be safely restarted       unresolved  no recovery rule found

The output shape matters because it changes reviewer behavior. If all three claims become fluent paragraphs, readers tend to accept them together. A ledger forces the weak claim to remain weak.

Expiration should be selective. A module inventory can regenerate on every merge. A business meaning approved by an operator should persist until its evidence changes, but the system must keep the approval and source. A runtime claim expires when its observation window no longer represents current use. One global "last updated" stamp cannot express these differences.

Mixed-language repositories need a common evidence model

No single parser can document a system that crosses COBOL, JCL, PL/SQL, shell, Java, and spreadsheet macros. Each language needs a front end that understands its own declarations and resolution rules, while the combined result needs one vocabulary for units, entry points, data assets, and edges.

Text search still has a role, but it should find candidates rather than assert relationships. Searching for a table name can locate embedded SQL, comments, copied definitions, test fixtures, and unrelated fields with the same spelling. A language-aware extractor can classify some of those hits. A later resolution pass can connect a call to a declaration under the correct build configuration.

Normalize identities without erasing native names. POSTINV, a source filename, a load-module name, and a scheduler operation might refer to one executable at different stages. Keep each identifier and add an evidence-backed alias relation. If the alias comes only from a naming convention, mark it inferred. Prematurely merging identities causes false edges that become hard to untangle later.

Cross-language connections usually appear at protocols and artifacts rather than symbols. A COBOL job writes a flat file that a Perl script reads. A VB6 client invokes a COM interface implemented in Delphi. A stored procedure writes a queue table polled by a service. Model the file, interface, table, or message as a first-class node. Connecting both programs directly would hide the contract that actually couples them.

Generated source needs two records: the generator input and the emitted artifact used by the build. Analyzing only templates misses emitted behavior; analyzing only emitted files makes ownership and regeneration obscure. The documentation should show which file can be edited and which will be overwritten.

Large repositories add a scale problem, not a different truth problem. Parse files incrementally, cache content-addressed results, and recompute affected edges when declarations or configurations change. Do not reduce scope by sampling directories and call the result a system map. A million-line tree can be processed in parts, but its cross-boundary references still need resolution against the whole inventory.

Verification can be sampled without becoming superficial

A team can test generated documentation without manually rereading the whole repository. Verification should sample by risk and edge type, then use automated invariants to catch broad classes of extraction failure.

Begin with parser fixtures. Give each language extractor small examples of direct calls, aliases, conditional compilation, dynamic dispatch, includes, malformed input, and comments containing code-like text. Assert both the edges it must emit and the tempting false edges it must reject. Keep real reduced failures as regression cases.

Run repository-wide invariants after extraction. Every cited file and line must exist at the analyzed commit. Every resolved call target must have a declaration or an explicit external identity. Every module member must exist in the inventory. Every edge type must use allowed source and target kinds. These checks do not prove meaning, but they catch broken joins and stale locations before a reviewer sees them.

Then choose review samples unevenly. Inspect all unknown edges on critical entry points, all cross-module writes, all scheduler conditions, and a random slice of ordinary static calls. Also sample negative space: select known dynamic mechanisms and confirm the document shows their uncertainty. Accuracy measured only on easy direct calls rewards the wrong pipeline.

A compact acceptance report can carry useful counts without turning them into a quality score:

Analyzed commit: 7c41e2f
Parsed files: 18,442 of 18,517 discovered
Skipped files: 75 (list attached to the evidence store)
Resolved call edges: 91,208
Unknown dispatch sites: 613
Broken citations: 0
Scheduler sources: repository JCL only; current-plan export absent

The numbers are an example of the output shape, not a benchmark. The important lines are the denominator, the skip list, and the absent scheduler source. Reporting "18,442 files analyzed" without saying that 75 were skipped lets a failed parser disappear inside a large total.

Review corrections should update rules or evidence, not just the rendered paragraph. If a reviewer identifies a false alias, add a constraint that prevents the merge next time. If an operator confirms a business definition, store the approval as a separate source. Otherwise regeneration will faithfully recreate every corrected mistake.

Trust breaks at dynamic and human boundaries

Automated documentation stops being trustworthy at boundaries where the repository lacks the deciding information. The main boundaries are dynamic selection, external state, generated or missing source, operational intervention, environment-specific configuration, and business intent.

You can turn that statement into a review checklist:

1. Resolve every referenced artifact. Find included files, generated sources, procedure libraries, control cards, schemas, and deployment manifests. Record anything absent.
2. Compare build reality with repository layout. Capture compiler flags, conditional symbols, generated code steps, and the exact deployable units.
3. Overlay runtime evidence without treating it as exhaustive. Keep the sample window and environment beside every observed edge.
4. Ask operators about restart, cutoff, override, and exception paths. These rules often live in runbooks, scheduler consoles, or memory.
5. Require a named source for business labels. A plausible expansion of an eight-character field name is still a guess.

One popular recommendation is to have a language model read the repository and write a complete architecture handbook in one pass. It is popular because the first result is fast and coherent. It is wrong because coherence removes the visible seams between parsed facts, interpretations, and omissions. Use a model to explain a graph, group evidence, and draft questions, but keep the evidence graph as the authority.

Security and access limits can create another blind spot. An analyzer that cannot read production scheduler exports, encrypted configuration, or database catalogs should say so at the top. The absence of access must not become the absence of a dependency.

The practical acceptance test is simple: select claims at random and follow their citations. If reviewers cannot reproduce the structural claims, the pipeline is not ready. If they can reproduce them but disagree with the prose, fix the inference rule or wording without discarding the extracted evidence.

Documentation should drive the rewrite plan

Repository documentation earns its cost when it changes sequencing, testing, and scope. A module map should identify independently replaceable units and shared-state knots. A reverse call graph should reveal callers that need compatibility coverage. The data model should identify ownership disputes and hidden coupling. The batch graph should expose cutoffs and recovery paths that a service rewrite must preserve.

For migration planning, query the evidence instead of reading it front to back. Ask which entry points reach a candidate module, which tables cross the proposed boundary, which batch jobs invoke it, and which edges remain unresolved. An unresolved edge attached to a daily settlement path deserves work before a fully mapped reporting utility, even if the utility has more lines.

Architecture modernization also requires a behavior baseline. Transliterating each old program into a new language preserves accidental boundaries and makes the generated diagrams look familiar, but familiarity is a poor design criterion. Use observed entry points, data contracts, side effects, and ordering constraints to define compatibility. Then design target services around coherent ownership.

CodeHero uses this combination when rewriting legacy systems: its platform reads the whole mixed-language tree, and a parity harness compares the replacement with recorded production traffic. That does not turn inferred purpose into fact. It gives structural extraction and behavioral evidence separate jobs, which is the discipline a rewrite needs.

Before approving a generated document, demand an answer to one concrete question: which statements would change if the scheduler export, runtime trace, or operator interview arrived tomorrow? If the document cannot identify them, it has hidden uncertainty instead of managing it. A repository can produce an excellent map, but the blank areas must remain visible until evidence fills them.

Keep the evidence inventory after the rewrite ships. It becomes a regression oracle for dependency changes, a source for operational documentation, and a check against new accidental coupling. The prose may age, but reproducible facts tied to commits can be regenerated whenever the system changes.