How moving VSAM to Postgres breaks hidden key contracts

A VSAM KSDS key and a Postgres primary key can identify the same business record while enforcing different contracts. Treat them as interchangeable and the online screens may look fine, right up to the point when an overnight program browses records in an order nobody documented, an alternate index returns duplicates differently, or a rewritten key changes which record comes next.

Moving the bytes is the easy part. The migration succeeds only when you discover every access path and reproduce its observable behavior before you improve the schema. That includes awkward details: fixed-width padding, EBCDIC ordering, partial-key starts, duplicate alternate keys, updates that maintain several indexes, and end-of-file behavior after concurrent changes. I have seen teams test CRUD calls and declare victory while the batch estate still depended on the physical personality of the KSDS.

A KSDS key is an access contract, not a column

A Postgres primary key expresses uniqueness and row identity inside a relational table. A KSDS primary key also drives placement in the logical sequence, keyed access, browse positioning, and status behavior expected by application code. Those jobs overlap, but they are not the same.

In a KSDS, the key is a fixed field at a declared byte offset and length in each record. VSAM compares that field according to the data and environment presented to it. Programs often construct the key in working storage, pad it, move display or packed values into it, and pass it through COBOL file operations or CICS commands. The bytes are part of the interface even when a copybook gives them a friendly business name.

A Postgres PRIMARY KEY requires every value to be non-null and unique, and Postgres backs it with a unique B-tree index. That says nothing about whether '00123 ' equals '00123', whether bytes encoded in EBCDIC sort like text encoded in UTF-8, or whether a caller may start a browse with only the leading part of a composite business key. A clean relational definition can therefore be wrong for the migrated application.

Keep three identities separate during discovery. The record key is the exact legacy byte sequence. The business identity is what the organization says the record means, such as account plus effective date. The database identity is the key you choose for references and updates in Postgres. Sometimes all three can converge. Do not force that result before the evidence supports it.

The IBM documentation for key-sequenced data sets describes records as ordered by the key field and accessed directly or sequentially. PostgreSQL documentation describes a primary key as a uniqueness and non-null constraint that also creates an index. Read together, those manuals expose the gap: VSAM documents access behavior around a record; Postgres documents a relational constraint around a value. Your compatibility design must cover the missing behavior.

Inventory the calls, not just the cluster catalog

The catalog tells you which base clusters, alternate indexes, and paths exist. It cannot tell you which programs rely on them, which key shapes those programs build, or what they do after a failed start. Build the inventory from code, JCL, transaction definitions, copybooks, and production observations.

Search COBOL for READ, START, READ NEXT, READ PREVIOUS, REWRITE, and DELETE against each file description. Search CICS code for READ, STARTBR, READNEXT, READPREV, RESETBR, ENDBR, WRITE, REWRITE, and DELETE, then record DATASET, RIDFLD, KEYLENGTH, GENERIC, GTEQ, and browse tokens. Find JCL steps that use IDCAMS, SORT, or utilities to copy, unload, merge, and validate the cluster. Include Assembler, PL/I, REXX, and scheduler scripts if they touch the same data.

For each operation, capture five things:

• The path used: base cluster or named alternate-index path.
• The exact key bytes and the declared or supplied key length.
• The operation and positioning rule, including exact, greater-than-or-equal, next, and previous.
• The expected result shape, status, and duplicate behavior.
• The downstream consumer, especially a batch step or extract.

Do not collapse several callers into a single row because they name the same file. A CICS inquiry that reads an exact customer number and a nightly COBOL job that starts at a branch prefix exercise different contracts. The latter may also depend on the sequence of duplicates beneath an alternate key, even if nobody intended that sequence to be public.

Recorded production traffic helps with online commands, but traffic alone misses scheduled behavior and rare recovery branches. Pair it with static call discovery and at least one complete batch cycle. If month-end processing uses a different JCL route, capture that too. The aim is a finite access-contract table that can drive tests, not a prose architecture document people will interpret differently.

Byte equality and text equality part company quickly

Preserve the original key bytes until you have proved that a decoded value has identical equality and ordering behavior. Character conversion is a semantic change, not housekeeping.

Consider a ten-byte customer key containing an uppercase region code, a zero-padded number, and trailing spaces. A loader might decode EBCDIC, trim the spaces, parse the number, and store (region text, customer_no integer). The new representation looks better. It also discards distinctions the old program may create, changes comparisons for malformed values, and loses the exact bytes needed to explain a parity failure.

Collation adds another trap. Postgres text ordering follows the selected database or column collation, while a legacy browse observes the ordering produced by its encoded key bytes and VSAM environment. Letters, digits, spaces, punctuation, lowercase data that slipped past validation, and national characters can land in a different order. ORDER BY key_text is not a compatibility claim unless tests prove it against representative and adversarial keys.

A conservative landing table keeps the raw identity beside parsed fields:

CREATE TABLE customer_landing (
    legacy_key       bytea PRIMARY KEY,
    record_image     bytea NOT NULL,
    region_code      text,
    customer_no      bigint,
    loaded_at        timestamptz NOT NULL DEFAULT clock_timestamp(),
    CHECK (octet_length(legacy_key) = 10)
);

CREATE UNIQUE INDEX customer_business_identity
    ON customer_landing (region_code, customer_no)
    WHERE region_code IS NOT NULL AND customer_no IS NOT NULL;

The raw key gives you lossless lookup and a stable diagnostic handle. Parsed columns support the intended model. The partial unique index tests a business hypothesis without pretending that every historical record is clean. If the hypothesis fails during load, you have found data that needs an explicit rule, not an inconvenience to suppress with ON CONFLICT DO NOTHING.

For a text-compatible raw key, you may use a fixed normalization function and a binary collation, but document every transformation and test its inverse. For mixed or zoned data, bytea is usually the honest first representation. You can retire it later after the parity suite proves that no caller observes the distinction. Removing evidence before parity is backward.

Alternate indexes carry their own semantics

An alternate index is not merely a secondary Postgres index. It is another access path with its own key extraction, uniqueness rule, duplicate handling, and browse order, exposed to programs through a path.

IDCAMS can define an alternate index with unique or nonunique keys. With nonunique alternate keys, multiple base records share the same alternate value. A caller can position on that value and browse the matching records. A plain Postgres index on surname makes lookup fast, but it does not define a deterministic order among equal surnames. The planner may return ties in an order that changes after vacuuming, index rebuilds, plan changes, or data movement. SQL promises no order without an ORDER BY that resolves ties.

Model each alternate path explicitly. Suppose the legacy application browses policies by agent code and, within an agent, has historically observed base-record key order. Encode both parts in the compatibility index and query:

CREATE INDEX policy_by_agent_legacy
    ON policy (agent_key_bytes, legacy_key);

SELECT legacy_key, record_image
FROM policy
WHERE (agent_key_bytes, legacy_key) >= ($1::bytea, $2::bytea)
  AND agent_key_bytes = $1::bytea
ORDER BY agent_key_bytes, legacy_key
LIMIT $3;

Do not assume the secondary order. Measure it against the source path, including duplicate groups created in different sequences and records whose alternate key changed after creation. If the observed order depends on an internal VSAM detail that you cannot or should not reproduce, make that incompatibility explicit and change the consumer with a controlled release. An unexplained ORDER BY added during migration merely replaces one hidden dependency with another.

Updates deserve special attention. A REWRITE that changes an alternate-key field makes VSAM maintain the alternate index according to how it was defined and upgraded. In Postgres, a generated column, trigger, or application write path must update the corresponding value atomically with the record. If one service writes the parsed field and another imports raw images, centralize key derivation in one tested function. Two implementations will diverge on padding or invalid bytes eventually.

Also inventory paths that exist but appear unused. Some are recovery tools or audit extracts invoked only after a failure. Mark them dormant with evidence; do not silently drop them because thirty days of online traces showed no calls.

Browse state must become an explicit cursor rule

A VSAM browse is stateful from the caller's point of view. SQL queries are sets, so the replacement must define positioning and continuation rather than relying on connection state or offset pagination.

START or STARTBR may request an exact key, a generic prefix, or the first key greater than or equal to supplied bytes. Subsequent next and previous calls advance relative to that position. Applications care about boundary behavior: whether the start returns a record, merely establishes position, reports not found, or places the browse at end of file. Your adapter should reproduce the contract that each caller actually uses.

Use keyset pagination with the complete legacy ordering tuple. For a forward browse on (alternate_key, base_key), return both values in an opaque cursor and resume with a strict comparison:

SELECT alternate_key, legacy_key, record_image
FROM customer
WHERE (alternate_key, legacy_key) > ($1::bytea, $2::bytea)
ORDER BY alternate_key, legacy_key
LIMIT $3;

For a greater-than-or-equal start, use >= only on the initial request. Continuations use > so the last row does not repeat. Reverse browsing flips both the comparison and the order. Offset pagination is wrong here because inserts and deletes before the offset shift the window, and because a large offset makes Postgres walk rows the application already consumed.

Partial keys need a byte-level boundary, not a casual LIKE 'ABC%'. For a fixed binary prefix, compute a lower bound equal to the prefix padded with the minimum suffix and an exclusive upper bound equal to the next possible prefix. If no next prefix exists because every byte is maximal, use only the lower bound and verify the prefix on returned keys. Put this logic in one adapter and test empty, all-zero, all-maximum, and embedded-space cases.

Concurrency forces a policy choice. A long VSAM browse and a sequence of stateless SQL requests may see inserts or deletes differently. Decide whether the replacement holds a repeatable-read transaction, uses a materialized work list, or accepts a moving view with keyset continuation. Match the source behavior required by the business process, not a theoretical image of VSAM. Holding a database transaction across an operator's screen session is usually a poor trade, while materializing keys for a bounded batch often works well.

The overnight batch is where ordering becomes business logic

Batch programs frequently treat sorted input as control flow. The program detects a key break, flushes totals, opens a new report group, carries a prior record forward, or matches two files by advancing the lower key. Change the order and you change the calculation even though every record arrived.

A typical failure starts innocently. The migration exports all rows from Postgres, the counts and checksums match, and the online API passes exact-key tests. At 1:00 a.m., a job reads policies through an alternate path by branch. Equal branch keys arrive in a different base-key order. The program pairs each policy with the next transaction record using a one-pass merge. One record now compares lower than the saved transaction key, falls into an exception branch, and prevents the control total from balancing. The database is available; the batch is still broken.

Another failure comes from implicit order in SQL. A developer tests SELECT ... FROM policy WHERE status = 'A' and sees primary-key order because the chosen plan happens to scan an index. Production statistics later favor a sequential scan. The same rows arrive in heap order. PostgreSQL has always been clear that rows have no guaranteed order without ORDER BY; the test accidentally blessed a plan, not a contract.

Turn every consumed sequence into an explicit data product. Record the source path, complete sort tuple, encoding, duplicate tie rule, and snapshot boundary. Then make the extract query state all of them. If the batch historically reads a frozen generation created by an upstream step, do not point it at live tables and hope transaction isolation gives the same cutoff. Create a run-scoped staging table or export under a declared snapshot.

A useful parity artifact compares ordered streams, not only unordered hashes:

run_id: 2026-08-14-nightly
path: POLICY.BY.BRANCH
snapshot_cutoff: 2026-08-14T01:00:00Z
record_count: 184203
first_key_hex: C1F0F0F0F0F0F0F1
last_key_hex: E9F9F9F9F9F9F9F9
rolling_digest: sha256:<digest>
first_mismatch_position: <none|integer>
source_key_hex: <hex when mismatched>
target_key_hex: <hex when mismatched>

The format shows where equality stopped, which is what an engineer needs at night. Generate a digest over length-prefixed key and record bytes so field boundaries cannot collide. Keep per-group counts and totals where the batch uses control breaks. A single whole-file checksum tells you that something changed; it does not tell you which access contract failed.

Mutation behavior matters as much as reads

Writes must preserve key immutability, duplicate rejection, index maintenance, and status mapping as one transaction. A migration that copies data correctly can still corrupt tomorrow's browse when the first online update follows different rules.

Many KSDS applications do not change the primary record key during REWRITE; they delete and write a new record instead. A relational API may casually allow UPDATE ... SET id = .... Decide whether the compatibility layer rejects key changes or implements the legacy delete-and-create consequences, including alternate indexes and audit behavior. Do not let an ORM choose.

Map expected failures deliberately. Duplicate primary key, duplicate unique alternate key, missing record, stale update, and end-of-browse are application outcomes, not generic internal errors. Postgres SQLSTATE values are useful inside the adapter, but leaking them to COBOL-era callers changes branches that may drive retries or operator messages. Build a small mapping table and test the exact status returned for each operation.

Locking also differs. A source transaction might read for update, modify the record, and rewrite it under a unit of work. The target needs an equivalent rule, commonly SELECT ... FOR UPDATE followed by an update with a version check. For disconnected clients, an optimistic version column can prevent a stale screen from overwriting a newer record. That is a modernization, but it must feed the legacy-visible status expected by the caller until the caller changes.

Dual writing to VSAM and Postgres is a popular safety recommendation, and I argue against making it the default. Two systems with different constraints and ordering semantics create a third reconciliation problem, especially when a partial failure changes an alternate key on only one side. A captured change log with replay and measured lag can be justified for cutover, but an open-ended dual-write period hides ownership. Prefer one writer at a declared cutover boundary, with a rehearsed rollback that restores the previous writer and replays accepted changes.

Build parity around operations, sequences, and failures

Record-level comparison is necessary and insufficient. The parity harness must issue the operations applications issue, observe ordered sequences, and compare failure behavior.

Create fixtures that are rude to both systems: minimum and maximum key bytes, leading and trailing spaces, numeric text with leading zeroes, duplicate alternate keys, missing optional fields, invalid historical encodings, keys adjacent to a prefix boundary, and records rewritten so an alternate key moves. Add a small random generator after the deliberate cases, but never let random coverage replace named fixtures that explain a failure.

For each access contract, run this sequence:

1. Load identical record images into the source fixture and target landing model.
2. Execute exact, greater-than-or-equal, prefix, next, and previous operations that the inventory says are valid.
3. Compare returned key bytes, record bytes, status, order, and end conditions.
4. Apply writes, rewrites, alternate-key changes, and deletes, then repeat the reads.
5. Run the consuming batch and compare its reports, control totals, rejects, and restart data.

Treat recorded production traffic as another fixture set, after removing or protecting sensitive fields under the customer's rules. CodeHero verifies rewritten behavior with a parity harness against recorded production traffic, and its whole-codebase analysis is useful here because the contract crosses COBOL, JCL, utilities, and copybooks rather than living in one repository folder. That does not remove the need for synthetic boundary cases or complete batch runs.

Define acceptance per path. An exact lookup may require byte-identical output and status. A modernized report may permit formatting changes but require identical group membership and totals. A corrected defect needs an approved difference, not a hidden exception in the comparator. Store those decisions beside the test so a later schema cleanup cannot erase a compatibility rule by accident.

Run performance tests with the same access shapes. A query that returns the correct first page after scanning the whole table will fail under batch load. Inspect EXPLAIN (ANALYZE, BUFFERS) for representative sizes, verify that composite indexes match comparisons and ordering, and test skewed alternate keys where one value owns many records. Correctness comes first, but a browse that misses its nightly window is operationally incorrect.

Restart state is part of the data contract

A batch cutover must preserve where work can restart, not only which rows the new database contains. Legacy jobs often checkpoint with a last key, a record count, a control total, a generation name, or a scheduler flag whose meaning depends on the source sequence. If the target changes that sequence, the same checkpoint can skip rows or process them twice.

Start by tracing restart data through the full job stream. A COBOL program may write the last completed branch and policy key into a small file, while JCL disposition rules decide whether that file survives an abend. A later step may delete the checkpoint only after reports are copied and control totals pass. Moving the primary data into Postgres without reproducing those commit boundaries turns a recoverable failure into an uncertain rerun.

Never translate a source record position into a Postgres offset. Relative byte addresses and control-interval positions belong to the VSAM implementation, and SQL offsets are unstable when rows change. Convert restart state to the complete logical ordering tuple, such as (branch_key_bytes, policy_key_bytes), plus the batch run identity and snapshot boundary. Resume with a strict comparison after the last committed tuple. If the old program intentionally re-reads the checkpoint record and detects it as a duplicate, preserve that rule in the adapter instead of silently changing >= to >.

The checkpoint write and the business effects it protects need one atomic boundary. When a batch updates Postgres directly, store its result rows, totals, and next cursor in the same transaction where practical. When it produces files for later steps, write run-scoped output and publish it only after the database commit. A cursor advanced before an output file reaches its final location creates a gap; an output published before the cursor commit creates a duplicate on restart.

Test interruption rather than merely discussing it. Kill the target process after the first row, in the middle of a duplicate alternate-key group, immediately before a group total, after the database commit but before output publication, and during the final checkpoint cleanup. Restart from the captured state and compare the final artifacts with an uninterrupted source run. The expected result is not always byte-identical intermediate work, but the accepted records, rejects, totals, and published output must match the declared contract.

Cutover rollback uses the same discipline. Record a high-water mark for accepted changes, stop or fence writers, drain in-flight work, and prove that the chosen system owns writes before the batch window opens. If rollback returns ownership to VSAM, replay only changes beyond its confirmed high-water mark and verify alternate paths afterward. A vague instruction to switch traffic back does not cover queued transactions, partially published extracts, or checkpoints created against the target order.

Give operations a runbook with concrete evidence: writer ownership, source and target high-water marks, active batch run IDs, last committed ordering tuple, output publication status, and the command or query that verifies each value. Rehearse it with the same scheduler dependencies used in production. The worst time to discover that a restart file contains a trimmed character key is after the first target-side abend.

Modernize only after the compatibility boundary holds

The first safe Postgres model may look less elegant than the final one. Raw keys, record images, explicit compatibility indexes, and adapters preserve evidence while you prove behavior. Once parity holds, modernize behind that boundary in measured changes.

A useful target often has three layers. The landing layer stores lossless imported bytes and source metadata. The compatibility layer implements legacy reads, browses, statuses, and ordered extracts. The domain layer exposes typed relational tables and APIs for new code. These layers can share a database, but their contracts differ. New services should not parse raw copybook bytes, and legacy callers should not gain direct access to tables whose keys are still changing.

Choose the durable primary key based on ownership. If the legacy key is stable, compact, and genuinely identifies the entity, keeping it can be sensible. If it embeds mutable attributes, overloaded type codes, or presentation padding, use a surrogate database key and preserve the legacy key under a unique constraint. Foreign keys should point to the identity that remains stable when the business corrects a code. This is a data-model decision, not a blanket rule that natural or surrogate keys always win.

Retire compatibility one dependency at a time. Change a consumer to a domain API with documented ordering, run both paths through parity, then remove its legacy path from the inventory. Only after no caller needs byte ordering should you consider dropping raw-key indexes or record images. Storage is cheap compared with reconstructing why a year-end job once sorted 9 before A.

The migration plan should name the overnight batch as a first-class consumer with an owner, test window, restart procedure, and rollback boundary. If the plan says only that the KSDS becomes a table, it has not described the work. A trustworthy plan says which byte contracts remain, which behaviors intentionally change, and which test proves each decision before the old cluster stops being the system of record.