Can a zero downtime migration stay reversible?

A zero downtime migration is possible only when the old system remains a valid place to serve traffic until the new one has proved that it can take over. That sounds obvious, yet many migration plans quietly destroy that option. They copy data, deploy a replacement, schedule a maintenance window, and call the outage "the cutover." The outage is not a law of nature. It is the result of coupling four separate actions: changing the route, changing the writer, changing the data authority, and removing the old path.

The safer design separates those actions and makes each one observable and reversible. Requests move through a routing seam. Writes carry stable identities and can be replayed. A backfill has a defined position in the change stream. Reconciliation compares business meaning, not byte shapes. The final route change then becomes a small configuration update rather than a leap across a gap.

This does not mean users will never see an error. A system can stay available while an individual request fails for the same reasons it failed before the migration. The promise is narrower and more useful: the migration itself does not require a period when the service refuses all work, and operators can send traffic back without reconstructing yesterday's database.

Zero downtime is a routing property

The application stays available when every request has a valid destination throughout the migration. Data replication helps, but replication alone does not provide availability. If clients connect directly to a server you plan to replace, the routing decision lives in hundreds of clients and you do not control the cutover.

Put one owned decision point in front of both implementations. It may be an API gateway, a reverse proxy, a load balancer rule, a message consumer assignment, or an adapter inside the existing process. The technology matters less than the contract: operators must be able to change the destination for a narrow slice of work without deploying every caller.

Choose the slice from a stable business boundary. Route account lookup separately from account mutation, invoice export separately from invoice creation, or one tenant separately from the rest. Avoid slicing by whichever controller file happens to be easiest to replace. A route that mixes reads, writes, scheduled jobs, and callbacks will give you a misleading green canary while an unobserved path still changes old data.

A minimal route record should be boring enough to review during an incident:

{
  "capability": "invoice.read",
  "cohort": "tenant-042",
  "destination": "new",
  "fallback": "old",
  "revision": 17,
  "changed_by": "change-1842"
}

The capability names the behavior, not a URL. The cohort limits exposure. The fallback states where the router may send a request when the new side is unhealthy. The revision makes concurrent edits visible, and the change reference tells the incident lead why the route moved. Store this record in a system with an audit history and make the router keep the last valid configuration if the control plane disappears.

Health checks must test the capability being routed. A process that answers /health can still lack a database migration, a decryption key, or a live downstream dependency. Run a small read or a synthetic operation that exercises the same chain as real traffic, with data reserved for that purpose. Keep automatic fallback conservative for writes. Retrying a read on the old side is usually harmless; retrying an accepted write can create a second order, payment, or case.

The strangler seam must own every entrance

Strangler routing works only when the seam intercepts all ways a capability enters the system. Martin Fowler's description of the Strangler Fig pattern emphasizes gradual replacement around the old system. Teams often remember "gradual" and forget "around." If a nightly job, desktop client, file drop, or message queue bypasses the seam, two implementations can become active without a shared traffic policy.

Inventory entrances from runtime evidence, not the architecture diagram. Inspect access logs, queue bindings, scheduler definitions, firewall flows, batch control language, stored procedure callers, and outbound callbacks that later return as inbound work. Legacy systems commonly expose the same operation through an HTTP endpoint, a terminal transaction, and a file imported after midnight. Treat those as one capability with several adapters.

The seam should normalize identity and context before dispatch. Both sides need the same request ID, actor, tenant, authorization result, deadline, and idempotency key. If each implementation derives those fields independently, reconciliation will blame business logic for differences created at the edge. Preserve the original request body for audit where policy permits, but pass a versioned canonical envelope to both paths.

Do not use percentage routing as the first control for stateful work. A ten percent random canary can send the first step of a workflow to the new side and its follow-up to the old side. Prefer a deterministic key such as tenant, account, case, or workflow ID. The router should produce the same answer for that key until an operator changes the cohort rule.

Authentication is another entrance. If the old system creates sessions that the new system cannot validate, the first routed request becomes a forced logout. Either validate the existing session at the seam and pass a short-lived signed identity assertion, or teach both sides to accept a shared session format during coexistence. Do not migrate password hashes by asking users to reset them unless the business has consciously accepted that disruption.

Finally, make route evaluation visible in every trace and log line. Record the rule revision, chosen destination, fallback decision, and cohort key. When a user reports that yesterday's invoice differs from today's, you need to know which implementation served each request. A dashboard that shows only aggregate traffic percentages will not answer that question.

Dual writes need one durable intent

The safe form of dual writing records one durable business intent and lets independent workers apply it to each data model. The unsafe form makes the request thread call database A, then database B, and hopes both calls succeed. That arrangement has an unavoidable gap: the first commit can succeed while the second times out, and the caller cannot tell whether a retry will duplicate work.

Use the current authority to commit the business change and an outbox event in the same local transaction. A relay publishes the event, and a consumer applies an idempotent projection to the new store. During an early migration phase, the old store remains authoritative even if the new projection is seconds behind. Later, after the route moves, the direction may reverse so that rollback remains possible.

The write envelope needs enough information to reject duplicates and detect ordering mistakes:

{
  "event_id": "01J7M6R2K8N4T3Q9V5X1",
  "aggregate_type": "invoice",
  "aggregate_id": "inv-90318",
  "aggregate_version": 44,
  "operation": "invoice.adjusted",
  "occurred_at": "2026-08-14T10:42:31Z",
  "payload": {"line_id": "ln-8", "amount_minor": 1250}
}

The consumer stores event_id in a processed-event table under a unique constraint. It applies version 44 only after version 43, or it parks the event until the missing version arrives. The payload uses business units such as minor currency units rather than formatted strings. A successful duplicate delivery returns the previous result; it does not perform the operation twice.

Microsoft's retry pattern documentation makes the relevant point plainly: a service can finish work and lose the response, so a retry may repeat a non-idempotent operation. Migration traffic creates this condition routinely because relays restart and network paths change. An idempotency key is not merely an API convenience. It is the evidence that two deliveries represent one intent.

Some writes cannot be replayed safely, particularly calls to an external party that lacks idempotency support. Keep that side effect behind the existing authority while coexistence lasts. Replicate the resulting state, not the command that caused it. Sending the same payment instruction from both systems is not dual writing; it is issuing two payments.

Watch the outbox age, not only its row count. A small queue containing one event blocked for six hours can be worse than a large queue draining normally. Expose the oldest unpublished event, oldest unapplied event per aggregate type, retry count, dead-letter reason, and version gap. Those signals tell operators whether rollback data is actually current.

Backfill must meet a moving change stream

A backfill is complete only when its snapshot has joined the live change stream at a known position. Copying every row while production continues creates a moving target. If the copier reads an account before an update and writes it after the live consumer has already applied that update, the old snapshot can overwrite newer state.

There are two sound patterns. Take a consistent snapshot tied to a log position, load it, then apply changes after that position. Or make every backfill upsert conditional on source version so it cannot replace a newer projection. Database change data capture products differ in syntax, but the invariant does not: every copied record and every live event must have a comparable order.

PostgreSQL logical replication illustrates both the help and the limits. Its documentation says the initial table snapshot is followed by changes in publisher order within one subscription. The same documentation warns that schema definitions and DDL are not replicated in commonly deployed versions, and sequence state has historically required separate handling before a switchover. Treat the manual for your exact database version as part of the runbook. "Replication is caught up" does not prove that a target can accept new writes.

Throttle the backfill against production latency rather than a fixed rows-per-second guess. Read in primary-key ranges, record the completed range and snapshot position, and make each chunk restartable. Large transactions retain logs, increase lock time, and make progress hard to inspect. Tiny transactions waste overhead. Measure the effect on the source and choose a chunk size that can finish and retry comfortably inside your operational limits.

Transformations need explicit failure storage. If a legacy status contains a value the new enum does not accept, do not coerce it to UNKNOWN and continue silently. Save the source key, source version, transformer version, raw value, and error. The business owner can then decide whether the value maps to an existing state, needs a new state, or exposes old corruption.

Never backfill derived totals without defining who recomputes them. If the new system derives invoice balance from entries while the old system stores a mutable balance column, compare the entries and the final business balance, but do not keep copying the mutable column forever. Otherwise two authorities survive inside the new schema.

Reconciliation compares invariants, not rows

Reconciliation should prove that both systems make the same business claims, even when their schemas differ. Row counts and checksums catch missing data, but they fail as the main acceptance test after an architectural rewrite. A normalized Postgres model will not have the same rows as a COBOL record layout, and a TypeScript client will not serialize fields in the same order as a desktop binary.

Start with invariants the business already relies on: every posted entry belongs to one account, debits and credits balance for a ledger boundary, an invoice total equals its lines plus tax, a closed case has a closing event, and an external reference remains unique. Write each invariant against both sides and compare results by stable business ID.

For fields that should match, canonicalize only differences that lack business meaning. Normalize timestamps to one precision, Unicode to one agreed form, empty strings and nulls according to the old behavior, and money to integer minor units. Do not lowercase identifiers or round numbers merely to make a report green. Every normalization rule can hide a defect, so keep the rules versioned and reviewable.

A practical comparison query returns discrepancies rather than a pass count:

SELECT account_id, old_balance_minor, new_balance_minor,
       old_balance_minor - new_balance_minor AS delta_minor
FROM migration_account_balance
WHERE old_balance_minor <> new_balance_minor
ORDER BY ABS(old_balance_minor - new_balance_minor) DESC;

The output shape matters: an operator needs the business ID, both values, and the delta. Store a reconciliation run ID, the source and target positions, query version, start and finish times, discrepancy count, and a bounded sample. A report without positions cannot be reproduced because both databases keep changing beneath it.

Classify discrepancies by cause. Transport gaps mean an event never arrived. Ordering gaps mean it arrived too early or late. Transformation defects produce the wrong target state from the right source. Expected differences come from deliberate architecture changes such as removing trailing spaces. Unexplained differences block expansion of the cohort even if the total is small.

Do not demand a universal zero-difference result when time itself changes the answer. An expiring quote or a live stock level can differ between sequential reads. Freeze the relevant clock, compare at matched log positions, or assert a business tolerance that the owner has approved. "Close enough" chosen by the migration team is not an acceptance criterion.

Shadow reads expose behavior before authority moves

Shadow reads let the new implementation process real requests while the old response still goes to the caller. They find semantic defects that data checks miss: default sort order, authorization edge cases, rounding, locale formatting, missing records, and different error handling. Because the new result is discarded, shadowing is safe only for operations with no side effects.

Clone the normalized request at the seam and set a hard deadline shorter than the user's request budget. A slow shadow must never delay the authoritative response. Remove or substitute secrets that the new path does not need, and mark the request so downstream code cannot send email, mutate caches, extend sessions, or emit billable calls.

Compare structured meaning rather than raw response bytes. Ignore trace IDs and generated timestamps. Compare status class, ordered versus unordered collections according to the contract, authorization decision, selected fields, error category, and business totals. Save a redacted sample for each new mismatch signature, not every response. Otherwise the comparison store becomes a second copy of sensitive production data.

Shadowing writes by executing and rolling back a database transaction is usually unsafe. External calls, sequence allocation, queue publication, and triggers may escape the transaction. Validate a write with recorded traffic in an isolated harness, or execute its pure decision logic against a snapshot and suppress the commit adapter. Be precise about what you tested.

Performance comparisons need the same care. A shadow request that runs after the old request may benefit from a warm cache, while parallel execution can add load that neither system sees alone. Measure each side's latency and resource use, but do not publish a winner until the test controls for cache state, request mix, and extra shadow load.

The useful exit criterion combines coverage and clean behavior. Track which operations, authorization roles, data shapes, and error paths the shadow has exercised. Ten million repeats of the common account lookup do not prove the rare reversal path. Route a cohort only when its actual behavior set has been observed or deliberately tested.

Cutover is a state machine, not an event

A reversible cutover moves one capability through named states with guarded transitions. A calendar entry may authorize a transition, but the clock must not decide whether the transition is safe. Operators should see the current authority, route, replication direction, lag, and rollback action on one page.

Use states that describe facts rather than project sentiment:

1. old_only: the old side serves and writes; the new side may be empty.
2. old_authority: both sides receive current data; users still reach the old side.
3. new_canary: a deterministic cohort reaches the new side; old writes remain current.
4. new_primary: all eligible traffic reaches the new side; the old side stays rollback ready.
5. new_only: the rollback window has closed and old mutation paths are disabled.

Each transition needs machine-readable guards. Moving to new_canary might require zero unexplained reconciliation differences, no unapplied event older than the lag budget, successful shadow coverage for every critical operation, and a tested route reversal. Moving to new_primary should add capacity headroom, background job ownership, support readiness, and confirmation that non-HTTP entrances follow the same authority.

Keep the transition command small and idempotent. It should update a versioned route record, not deploy code, run schema changes, flush queues, and restart workers. If five actions must happen in a precise minute, one will be late and the rollback procedure will be ambiguous.

Separate a stop from a rollback. A stop freezes cohort expansion while both sides continue in their current roles. A rollback sends traffic to the old side because a defined guard failed. A data repair corrects state after operators understand the fault. Automatically copying target rows back to the source during an alarm can spread corruption faster than people can diagnose it.

Practice the reverse transition under production conditions before the broad cutover. Send a canary cohort to the new side, create and update representative records, return the cohort to the old side, and confirm those records remain correct and accessible. A rollback document that has never moved real state is a theory.

Rollback expires when the old side stops learning

The old system remains a rollback target only while it receives every state change needed to resume authority. Keeping its servers powered on is irrelevant if its database fell behind after the first new-side write. Define the rollback window in data terms: which operations replicate back, what lag is acceptable, and which changes cannot be represented in the old model.

Reverse replication becomes difficult when the new architecture allows states the old schema cannot express. Delay those features during coexistence or add a compatibility representation before cutover. For example, if the new system supports multiple adjustments where the old record has one amount field, you cannot promise rollback after users create the second adjustment unless the old path can preserve it.

Use an expand-and-contract schema sequence. Add fields and readers that tolerate both forms. Populate the new form. Switch writers. Observe. Remove the old form only after the rollback window closes. Destructive schema changes, reused enum values, and shortened fields erase your return path even while request routing still appears reversible.

Ownership of scheduled work must be singular. A route can send interactive traffic to the new side while both schedulers generate statements or close cases. Give every job a lease or authority flag governed by the same migration state, and record which implementation claimed each run. On rollback, transfer the lease before the route if the job can mutate data that the old request path will read.

CodeHero treats this coexistence contract as part of the rewrite: the new Go, Rust, or TypeScript system is checked against recorded production traffic with a parity harness, and the project is delivered in under 30 days. That short delivery promise does not remove the need for customer-owned acceptance guards or a rollback policy; it makes those decisions impossible to hide inside a long program.

Set an explicit condition that ends reversibility. It might be the first use of a new-only state, deletion of reverse replication, execution of a destructive schema change, or expiry of the operational agreement to support both sides. Have the accountable owner approve it. If nobody can name that point, the team will discover it during the incident when rollback is already gone.

The last route change should be uneventful

The final cutover is safe when it changes only the default route for a capability whose cohorts have already run on the new side. At that point, code, data, jobs, access controls, dashboards, on-call instructions, and rollback mechanics are already in production. The remaining risk is scale, so capacity and queue behavior deserve more attention than feature correctness.

Before changing the default, capture a decision record with the exact route revision, replication positions, reconciliation run, open exceptions, approver, and rollback threshold. Check that the old side can absorb the full load immediately. Scaling it down before the rollback window closes saves little and turns a reversible route edit into a capacity restoration exercise.

Change the route in stages that match your failure domain. One tenant can expose tenant-specific data. One region can expose dependency placement. A percentage may be appropriate only after workflow affinity is guaranteed. Pause between stages long enough to see the slowest relevant job or callback, not an arbitrary five-minute graph.

Watch symptoms users feel: error category, tail latency, queue age, failed business invariants, authorization denials, and support contacts tied to the cohort. CPU and memory can look calm while invoices disappear from search because an index consumer is stalled. Tie every rollback threshold to a named signal and an observation window.

Keep one operator responsible for the transition command and another responsible for reading the guards. The second person should have authority to stop the move without negotiating in the incident channel. Record both decisions automatically. This division catches stale dashboards, misunderstood cohort rules, and the familiar mistake of treating silence as approval.

When a threshold trips, execute the pretested route reversal and preserve evidence. Do not improvise a forward fix while new errors accumulate. Once traffic is stable on the old side, freeze the relevant target writes if needed, record positions, and diagnose. Reversibility buys time for careful work only if operators use it.

After the rollback window closes, remove the machinery deliberately. Disable old writers, revoke credentials, stop reverse consumers, archive reconciliation evidence according to policy, and keep route history with the change record. A forgotten dual-write relay can resurrect stale data months later. A forgotten fallback can send a small class of requests into an application nobody monitors.

The migration has succeeded when the old system is no longer needed, not when the first request reaches the new one. Until that point, treat routing state, event position, and reconciliation evidence as production data. If any of those three is missing, the cutover is relying on memory, and memory is weakest when the pager is loudest.