A rollback plan must survive the first write

A rollback plan is credible only if the old system can accept the new system's writes without losing, duplicating, or misordering them. Switching a load balancer back is the easy part. The hard part begins with the first order, payment, case, journal entry, or status change committed after cutover.

That makes rollback a data and authority problem, not a deployment feature. Before cutover, the team must know which system owns writes at every instant, how later writes return to the old model, which outside effects cannot be undone, and who may call the reversal. If those answers live in someone's head or depend on code that has never run, there is no rollback plan. There is only a hope that the new system fails before anything interesting happens.

Cutover transfers write authority

A safe cutover gives write authority to exactly one system at a time. Both systems may serve reads, compare results, consume copied events, or run calculations in shadow mode. They must not independently accept authoritative changes to the same business record. Two writers create conflicts that a traffic switch cannot resolve.

Write authority includes more than the main database connection. A legacy estate often accepts changes through batch files, message queues, operator screens, scheduled jobs, partner transfers, stored procedures, and direct support updates. I have seen teams freeze the web application while an overnight job quietly posts adjustments through a different account. The migration looked stable until the two ledgers disagreed the next morning.

Build a write inventory around business operations, not tables. For each operation, record the entry point, identity, transaction boundary, generated identifier, timestamp source, downstream effects, and system that owns it before and after the switch. "Update customer" is too broad. "Change postal address, emit the compliance notice, and enqueue the print job" is specific enough to expose what rollback must preserve.

The routing control also needs one owner. DNS alone is a poor emergency control because resolvers and clients cache it beyond the moment when operators think the change occurred. Prefer a control at a gateway, proxy, queue consumer, connection broker, or another point where the team can observe the active route. Record the control's current value and the command that changes it. A screenshot of a console does not give the incident commander a repeatable action.

Define the authority states in plain terms: OLD_WRITES, DRAINING, NEW_WRITES, and ROLLING_BACK are enough for many systems. Each state should permit a known set of writers. The transition must reject an unexpected writer rather than merely log it, because a warning discovered after rollback cannot unwrite a transaction.

Preconditions need pass or fail answers

Cutover should begin only when every rollback precondition has a named test, a current result, and an owner who can stop the change. A document that says "replication healthy" leaves room for an argument during an incident. A test that says the replay position equals or exceeds a captured position, with the exact query and output attached, gives the team a fact.

Use a short readiness contract. It is a gate, not an aspiration:

• The old release can read every schema change introduced for cutover.
• The old system is deployed, reachable, patched, and able to authenticate to its dependencies.
• The capture or replication path has processed a production shaped load without falling behind its agreed ceiling.
• All writers in the inventory obey the authority control, including batch and operator paths.
• The team has restored a recent backup into an isolated environment and proved that it starts.

The last item catches a common substitution: backup success is not restore success. A green backup job proves that bytes were copied somewhere. It does not prove that the encryption key is available, the archive is complete, the engine can read it, or the application can start against it. Keep the restore command, duration, checksum, and application smoke result with the cutover record.

Capture a data checkpoint immediately before authority changes. In PostgreSQL, a team using physical streaming replication can compare a primary's current WAL location with the standby's replay location. The PostgreSQL manual defines an LSN as a position in the write ahead log and says pg_last_wal_replay_lsn() returns the last location replayed during recovery. A minimal evidence capture looks like this:

/* On the primary */
SELECT pg_current_wal_lsn();
 pg_current_wal_lsn

 7A3/91F2C6D0
(1 row)

/* On the standby */
SELECT pg_last_wal_replay_lsn();
 pg_last_wal_replay_lsn

 7A3/91F2C6D0
(1 row)

Treat the value as an example output shape, not a magic threshold. Equality at one instant does not prove that every business write is reversible, and a byte difference does not translate directly into elapsed time. The check proves one narrow fact: the standby replayed through the captured log position. Your evidence also needs application counts, invariants, and sampled records that make sense in the domain.

Keep the old system warm until data can return

The old system should stay warm until the team either proves it can ingest every postcutover change or formally gives up rollback and moves to forward recovery. A fixed number of days sounds decisive, but it ignores transaction volume, delayed jobs, settlement cycles, and schema changes. Set the period from observable exit conditions, then add a calendar limit for staffing and cost control.

"Warm" means runnable under incident pressure. The old application has compute capacity, current configuration, valid secrets, network access, dependency contracts, storage headroom, monitoring, and operators who still know how to use it. A powered off virtual machine with an expired certificate is an archive. It is not a rollback target.

Keep it warm through at least one complete business cycle that can expose delayed behavior. That cycle might include an overnight posting run, a billing boundary, a partner file, a weekend schedule, or a period close. Do not copy a generic seven day or thirty day rule. A claims system that receives late documents and a point of sale service that settles every night have different evidence windows.

The exit conditions should cover behavior and recoverability. Require the new system to complete the delayed jobs, reconcile external acknowledgements, hold error and latency limits under real traffic, and produce a change stream the old system can consume. Also require a successful rollback rehearsal against data captured after a simulated switch. If the team removes the reverse path, applies a destructive schema change, or lets an old dependency contract expire, record the exact moment when rollback ends.

Keeping the old system warm has a cost and a risk. Unpatched services, duplicate schedulers, and live credentials expand the failure surface. Reduce that surface deliberately: block user traffic, disable every scheduler except the one needed for a tested rollback, restrict operator access, and monitor any connection attempt. Warm does not mean casually left running.

Postcutover writes determine the strategy

Every postcutover write needs one of four treatments: reproduce it in the old system, preserve it for later replay, compensate for its effect, or accept that it makes rollback impossible. Labeling all four as "data sync" hides the decisions that matter. The right treatment follows business semantics, especially ordering and side effects.

Reverse replication works when the target model can express the new change and the replication tool preserves the required transaction boundaries. It becomes dangerous when the modern system splits one legacy row into several records, replaces mutable status with events, changes identifier generation, or strengthens validation. A row copier can deliver syntactically valid data that the old application interprets incorrectly.

An append only change journal is often easier to reason about. Give each accepted command an immutable operation ID, business key, source sequence, schema version, actor, acceptance time, payload, and result. The rollback importer records the operation ID it applied, so retries do not repeat the business action. Idempotence belongs at the business boundary: "set address to X" can be retried, while "add 10 to balance" needs a unique operation identity and duplicate rejection.

Dual writing from application code is popular because it appears immediate. I argue against it for most cutovers. The request can commit to the new database and time out before committing to the old one, leaving the caller unsure and the systems split. Reversing the order only changes which system wins the failure. A transactional outbox or database change stream ties capture to the authoritative commit and lets a separate consumer retry delivery.

Some writes should not flow backward automatically. If the new system permits a state that the old schema cannot represent, quarantine the operation and expose the count before cutover. The same applies when validation rules differ. Do not coerce the value, drop a field, or stuff new meaning into an old free text column merely to make a reconciliation counter reach zero.

The runbook must move data and traffic

An executable rollback runbook freezes new writes, establishes a final boundary, drains captured changes into the old model, verifies business state, and only then returns traffic. Reversing traffic first invites users to create more changes while the importer is still catching up. That makes the boundary move during the incident.

A practical sequence has explicit aborts and evidence:

1. Declare ROLLING_BACK, reject new mutating requests, pause consumers, and record the time plus the last accepted operation ID. Reads may continue only if they cannot trigger hidden writes.
2. Wait for in flight transactions to finish or terminate them under a documented rule. Capture the source log position, queue offsets, and counts of unprocessed journal entries.
3. Apply the reverse stream through the recorded boundary. Stop if an operation lacks a mapping, violates an old invariant, or produces a different external reference.
4. Run reconciliation queries and sampled business reads on the old system. Route a synthetic transaction through every critical entry point, but keep outside notifications in a controlled sink.
5. Restore old write authority, release traffic gradually, resume only the old schedulers, and watch duplicate, rejection, and backlog counters.

Write expected outputs beside every command. 0 rows can mean success for an orphan query and disaster for an order count. State which meaning applies. Put credentials and approval procedures near the commands without embedding secrets in the runbook. If the person executing it must search a password vault path, identify the entry and verify access during the rehearsal.

Time each stage separately. The useful measure is not only total recovery time. The team needs to know how long writes remain frozen, how fast the reverse consumer drains at peak volume, and which validation dominates the pause. If a backlog of 500,000 operations takes longer to replay than the business can tolerate, the plan fails even if a small rehearsal passes. Test at the expected high water mark, with headroom.

Compatibility preserves the return path

Rollback depends on backward compatibility across databases, messages, APIs, files, and authentication. The old binary must run against the cutover state. If a migration drops a column, reuses an enum value, changes a message meaning, or rotates a credential beyond the old client's support, the return path can disappear before anyone notices.

Danilo Sato's Parallel Change description separates an incompatible change into expand, migrate, and contract phases. That model is useful because rollback belongs in the expanded phase, while both old and new consumers still work. Teams get into trouble when they treat contract as housekeeping and remove the old field or endpoint as soon as new traffic looks healthy. Contract is the deliberate end of rollback, and it deserves a change record of its own.

Additive schema work is necessary but insufficient. A nullable column can still break old code if a trigger changes, a query uses positional inserts, or a stored procedure returns a new result shape. Test the actual old release against a clone of the cutover schema. Exercise reads and writes, including rare values, empty batches, maximum field sizes, and error paths.

Messages need a compatibility rule too. Consumers should ignore fields they do not understand, but they must reject a changed meaning disguised under an old field name. Keep old event types available through the warm period or provide a tested down converter. Version the converter and store the original payload so an operator can reproduce a disputed mapping.

Authentication failures are especially embarrassing because they are preventable. Retain the old service identities, certificate chains, cipher compatibility, and network routes for as long as rollback remains an option. Test them from the old runtime, not from an administrator's workstation.

External effects need compensation

A rollback cannot retract an email already delivered, a bank file already accepted, a label already printed, or a partner instruction already acted upon. These are external effects, and the plan needs a policy for each one before the switch. Database parity does not settle them.

Assign an idempotency key to every outbound action and keep the provider's acknowledgement with it. During rollback, the old system must learn which actions already occurred so it does not send them again. If the receiver supports idempotent requests, reuse the same key. If it does not, place the action behind an internal ledger that rejects a second send.

Compensation is a new business action, not deletion. A posted payment may require a reversal entry. A dispatched warehouse instruction may require a cancellation that can itself fail. A customer notification may need a correction written by a person. Record who authorizes each compensation, the deadline, and what happens when the outside party cannot reverse it.

Scheduled work creates a quieter duplicate risk. When both estates stay warm, two schedulers may collect the same due rows and emit the same action. Authority control must cover jobs as firmly as HTTP requests. During normal operation, the inactive scheduler should be unable to acquire its lease or production credential. During rollback, operators transfer that lease only after the new worker stops and its last completed item is known.

Reconcile effects by business identifier and acknowledgement, not by local queue depth. An empty queue can mean that every item succeeded, every item failed into an unmonitored dead letter store, or a filter selected nothing. The useful report joins intended actions, attempts, provider responses, and compensations into one row per business event.

A rehearsal must force the ugly failure

A rehearsal proves rollback only when it uses postcutover writes, breaks something on purpose, and returns service through the same controls the production team will use. A meeting where people read the runbook aloud tests prose. A staging switch with no representative data tests routing. Neither demonstrates recovery.

Use a production shaped clone or isolated replay environment with scrubbed records and recorded traffic. Start the old and new releases, enable the real authority control, and load enough history to make migrations and indexes behave credibly. CodeHero uses a parity harness against recorded production traffic when rewriting legacy systems, which is useful here because the same traffic corpus can expose behavior differences before a rollback rehearsal.

Then force a failure after the new system has accepted a mixed set of operations. Include an update that arrives twice, two commands against the same account in order, a batch boundary, a rejected value, an operation with an outside effect captured in a sink, and a job that was running during the freeze. Kill the reverse consumer halfway through and restart it. The operation IDs should prevent duplicates, and the boundary should remain stable.

One familiar failure deserves special attention. The reverse importer reports no backlog, traffic returns to the old system, and basic counts match. Hours later, a partner rejects the day's file because the modern system generated identifiers in a format the old export truncates. The rollback moved rows correctly but lost behavior at a file boundary. A proper rehearsal runs the export, parses it with the receiving contract, and compares identifiers end to end.

Collect evidence that someone other than the migration author can judge: accepted operation IDs, source and applied positions, invariant query results, duplicate counts, quarantined mappings, outside effect ledger entries, stage durations, and the final authority state. Store the failed rehearsal too. A plan that passes only after an engineer edits data by hand needs that repair written as a controlled step or removed through a code change.

Observability must expose correctness

Rollback signals must report business correctness, not merely whether processes are alive. A new service can return fast responses while assigning duplicate invoice numbers, skipping a posting rule, or leaving an export permanently queued. Infrastructure graphs help locate a fault, but they rarely tell the decision owner whether continued writing is safe.

Define invariants from the legacy behavior before cutover. An invariant might state that every accepted payment has one journal posting, every shipment references an accepted order, every closed case has a closing reason, or every outbound file has a matching control total. Express each one as a query or report that both systems can produce. When schemas differ, compare a canonical business projection instead of forcing table equality.

The projection should normalize differences that do not change behavior and preserve differences that do. Converting timestamps to one zone before comparison is sensible. Ignoring cents because one system stores decimal values and another stores integer minor units is not. Decide those rules with the people responsible for the business record, then version them alongside the migration code. Otherwise an engineer can make a red comparison green by broadening a filter during the incident.

Counts need denominators and boundaries. "Twelve mismatches" means little without the number examined, the operation types, and the accepted time range. A zero mismatch count can also lie when the comparison job stopped at yesterday's partition. Every reconciliation result should state the source boundary, target boundary, number selected, number compared, mismatch count, oldest unmatched operation, and completion time.

Use freshness signals that follow the actual replay path. Queue depth alone misses a stuck partition if other partitions keep draining. Consumer liveness misses a poison message that retries forever. Report the oldest unapplied operation and its age, the latest source sequence seen, the latest target sequence committed, quarantine count, retry count by operation, and drain rate. A stable backlog under constant traffic may be healthy during normal operation but fatal to a rollback that requires reaching zero inside a write freeze.

Sampling still matters because invariants cannot encode every odd legacy rule. Select records by deterministic criteria so the rehearsal and production cutover inspect comparable cases: largest transactions, records with maximum field lengths, reopened cases, reversed payments, nondefault currencies, and operations crossing a date boundary. Include the business result and the output artifact, not just the stored row. An invoice that balances in the database but renders with a blank tax identifier is a behavior mismatch.

Alerts should map directly to declared decisions. An invariant breach may freeze writes immediately. A rising replay age may start a timer. A transient latency spike may call for observation without rollback. Write that mapping before cutover and attach it to the alert. During an incident, an alert named "migration unhealthy" only starts an argument about what unhealthy means.

Preserve the telemetry after the warm period ends. It explains why the team retired the return path and gives forward repair a trusted baseline. The final record should show that delayed work completed, comparison boundaries advanced past the agreed business cycle, quarantines were resolved, and no unexplained external effects remained. That evidence is stronger than a meeting note that says the migration looked good.

Make the comparison job independent from the route it judges. If the new application's database connection, cache, or serialization code also powers the verifier, one defect can corrupt both the result and its supposed check. Run critical reconciliations through a separately deployed reader with restricted credentials, and keep the query text plus result checksum in the cutover record. Independence does not require a second platform, but it does require a failure path that cannot silently agree with itself.

Test missing telemetry as a failure too. Stop the journal consumer, withhold one source partition, and make the outside effect sink reject an acknowledgement. The dashboard should show an aging boundary and an incomplete comparison, while the authority control keeps the team from declaring success. If a blank panel looks the same as zero mismatches, fix the panel before cutover.

The rollback decision needs a clock

The rollback decision should follow declared triggers, an owner, and a time budget set before cutover. Without them, engineers spend the reversible window diagnosing a problem while new writes accumulate and external effects spread. By the time leadership asks to go back, replay may take longer than a forward fix.

Choose triggers that describe user or accounting harm: rejected critical operations, invariant violations, unexplained reconciliation drift, an unbounded queue, missing external acknowledgements, or inability to close a required batch. CPU or latency can support the decision, but infrastructure symptoms alone rarely say whether data remains safe. State the observation period and the source of each signal.

Give one person authority to order the rollback and one alternate. The database lead, application lead, operations lead, business owner, and incident commander may advise, but consensus is too slow when writes are arriving. Also name who can declare rollback unavailable because a destructive boundary has passed. That declaration should switch the response plan to forward repair and compensation, not leave the team debating an obsolete option.

The clock needs two limits. The decision deadline says how long the team may investigate before freezing writes or committing to forward recovery. The execution budget says how long the business can tolerate the write pause and replay. Derive both from transaction arrival, backlog drain rate, outside deadlines, and staffing, then test those assumptions in rehearsal.

A team ready for cutover can hand the runbook to an operator who did not write it, inject a failure after real writes, and recover the old system within that budget. If the exercise needs an unwritten query, a special engineer, or a manual data edit, postpone cutover. Production will not make those dependencies kinder.