Migrating a microservices system is rarely a single database command or one deployment.

A real migration may change several things at the same time:

Which service owns a business capability
Which database stores the data
How APIs communicate
Which events are published and consumed
How existing records are transformed
How traffic moves from old code to new code
How the team detects errors and recovers

The difficult part is not creating the new table, endpoint or service. The difficult part is keeping the system correct while old and new versions run together.

This guide explains the main best practices for planning and performing microservice migrations with minimal downtime and a practical recovery path.

Quick Answer

The safest microservice migrations are small, backward-compatible and observable.

Use this general sequence:

Discover the real dependencies and define the migration boundary.
Expand the system by adding compatible schemas, APIs or events.
Synchronise old and new paths while existing data is copied.
Verify data, behaviour, performance and security continuously.
Shift traffic gradually using flags, routing rules or canary releases.
Stabilise the new path while retaining a tested recovery option.
Contract by removing obsolete fields, endpoints, topics and code only after all consumers have moved.

This is often called expand and contract or parallel change. The important idea is that a breaking change is divided into several compatible releases instead of being performed all at once. Martin Fowler describes the three broad stages as expand, migrate and contract in his explanation of Parallel Change.

Risk	Safer practice
Big-bang replacement	Incremental strangler migration
Renaming or deleting a column immediately	Add, backfill, switch and remove later
Copying millions of rows in one transaction	Small resumable batches
Writing to a database and broker separately	Transactional outbox or CDC
Assuming messages arrive once	Idempotent consumers and deduplication
Changing an API response without checking clients	Backward-compatible changes and contract tests
Deploying all traffic to new code	Canary, percentage rollout or routing by tenant
Treating rollback as “deploy the old image”	Design data-aware rollback or roll-forward
Declaring success after deployment	Reconcile data and monitor business metrics

What Does “Migration” Mean in Microservices?

The word migration can describe several different operations.

Table of Contents

1. Service Migration

A business capability is moved from a monolith or an existing service into a new microservice.

Example:

Monolith order module
        ↓
New Order Service

2. Database Schema Migration

The structure of a service-owned database changes.

Examples include:

Adding a column
Creating an index
Splitting one table into several tables
Changing a data type
Adding a constraint
Removing an obsolete table

3. Data Migration

Existing records are copied, transformed or reassigned to a different service.

Example:

monolith.customers
        ↓
customer_service.customers

4. API Migration

Consumers move from an old HTTP, gRPC or GraphQL contract to a new one.

5. Event Migration

Producers and consumers move to a new event name, payload, topic or schema version.

6. Infrastructure Migration

The service moves to a new cluster, region, cloud account, runtime or deployment platform.

These migrations often overlap. Extracting an Order Service, for example, can involve a new database, a historical backfill, a new API, new events, traffic routing and changes to several consumers.

That is why the migration should be managed as a sequence of controlled state transitions rather than one deployment ticket.

Why Microservice Migrations Are Difficult

In a monolith, code and database changes can sometimes be released together. In a microservices environment, independent deployment creates a mixed-version period:

Old application instances may still be serving traffic.
New instances may already use the new schema.
Some consumers may understand a new event field while others do not.
A mobile application may continue using an old API for months.
Delayed messages may contain an older event format.
Backfill jobs may still be copying historical records.
Cached data may represent the old ownership model.

The system must remain correct throughout this period.

A migration can therefore fail even when every individual component appears healthy. Typical failure modes include:

Lost writes during data copying
Duplicate messages during retries
Old consumers rejecting a new payload
Long database locks causing request timeouts
Partial data movement between services
Two databases disagreeing about the same entity
Rollback code being unable to understand data written by the new version
A silent business error that produces no infrastructure alert

The objective is not merely “the deployment completed.” The objective is that the new system is correct, recoverable and measurably better or at least equivalent.

1. Define the Boundary and Success Criteria First

Do not start with migration scripts. Start with the reason and the boundary.

Write a short migration definition containing:

The business capability being moved
The present owner and future owner
The source and destination data stores
All known API consumers
All event producers and consumers
Expected traffic and data volume
Availability and latency targets
Acceptable data lag during transition
Recovery point objective and recovery time objective
Compliance, audit and retention requirements
Clear completion and abort conditions

Example success criteria:

- 100% of order-creation traffic reaches Order Service.
- Reconciliation mismatch stays below 0.01% for seven days.
- No unexplained missing or duplicate orders exist.
- p95 creation latency remains below 400 ms.
- Error rate remains below 0.5%.
- Every known consumer has stopped reading the legacy order tables.
- The rollback or roll-forward runbook has been tested.

“Move orders to a microservice” is not a testable success criterion. Precise thresholds create an objective go/no-go decision.

2. Discover Dependencies Before Changing Ownership

Documentation is useful, but the real dependency graph may be larger than the documented one.

Inspect:

Source-code references
Database queries and stored procedures
Foreign keys, views and triggers
Scheduled jobs and ETL pipelines
Reports and business-intelligence tools
API gateway and reverse-proxy routes
Event topics and consumer groups
Cache keys
Search indexes
Data exports and partner integrations
Support and administrative tools
Manual operational procedures

Combine static discovery with runtime evidence:

Distributed traces
Database query logs
API access logs
Broker consumer information
Network telemetry
Audit logs

For every dependency, record:

Item	Example
Owner	Billing team
Dependency	Reads `orders.status` directly
Interface	Shared PostgreSQL table
Criticality	Required for invoice generation
Migration action	Replace with Order API or event projection
Deadline	Before legacy table becomes read-only
Verification	Compare invoice count and total value

Hidden database readers are especially dangerous. A service cannot truly own its data while unrelated services continue to query its tables directly.

3. Give Each Service Clear Data Ownership

The usual target is not necessarily one physical database server per service. The important rule is logical ownership:

One service controls the schema.
Other services do not write its tables.
Other services obtain information through published contracts.
Schema changes are made by the owning team.

Depending on scale and risk, services may use:

Separate database servers
Separate database instances
Separate databases on the same server
Separate schemas with strictly controlled permissions

The last option can be an intermediate step, but it requires access controls that prevent accidental cross-service queries.

Avoid a Distributed Monolith

Moving code into several repositories while retaining one freely shared database does not create independent services. It can produce the operational cost of microservices without the isolation benefits.

Common warning signs are:

Service A joins Service B’s tables.
Several services run migrations against the same schema.
A column cannot be changed without coordinating many teams.
Business rules are enforced by undocumented cross-service database triggers.
One database outage stops every service.

During an incremental migration, temporary sharing may be unavoidable. Make it explicit, time-limited and measurable. Track every temporary dependency to removal.

4. Prefer Incremental Migration Over a Big-Bang Rewrite

A full rewrite creates a long period in which the new system receives little production feedback. It also concentrates data, code and operational risk into one cutover.

The strangler fig pattern replaces capabilities gradually. A routing layer sends selected requests to the new service while the rest continue to use the legacy system. AWS describes this as a way to reduce transformation risk and business disruption during monolith-to-microservice migration in its Strangler Fig guidance.

A typical progression is:

Stage 1: All traffic → Legacy system
Stage 2: Selected operation or tenant → New service
Stage 3: Most traffic → New service, fallback retained
Stage 4: All traffic → New service
Stage 5: Legacy capability removed

Choose a first migration slice that has:

A clear business boundary
Manageable data volume
Few synchronous dependencies
Measurable outcomes
Real value even if later migrations pause

Do not automatically start with the smallest module. A technically small module with ten hidden consumers may be riskier than a larger but well-isolated capability.

5. Use Expand-and-Contract Database Changes

A destructive schema change should normally span multiple releases.

Suppose customers.full_name must become display_name.

The unsafe approach is:

ALTER TABLE customers
RENAME COLUMN full_name TO display_name;

Old application instances may still query full_name and fail immediately.

Use the following phases instead.

Phase A: Expand

Add the new structure without removing the old one:

ALTER TABLE customers
ADD COLUMN display_name VARCHAR(200) NULL;

Deploy code that can tolerate both schemas. Depending on the application and database, it may temporarily write both fields or use a compatibility layer.

Phase B: Backfill

Copy existing values in small batches:

UPDATE customers
SET display_name = full_name
WHERE id > :last_id
  AND id <= :batch_end_id
  AND display_name IS NULL;

Make the operation resumable. Store progress, limit the batch size and monitor database load.

Phase C: Switch Reads

Deploy code that reads display_name. Keep the old field available while older instances and consumers are retired.

Phase D: Verify

Check for missing or conflicting values:

SELECT COUNT(*) AS missing_values
FROM customers
WHERE display_name IS NULL;

If the two columns are expected to match during transition:

SELECT COUNT(*) AS mismatched_values
FROM customers
WHERE display_name IS DISTINCT FROM full_name;

Use the equivalent null-safe comparison for the selected database engine.

Phase E: Contract

Only after all code versions and consumers have moved should the old column be removed:

ALTER TABLE customers
DROP COLUMN full_name;

The contract stage may be days or weeks later. That delay is intentional.

6. Keep Schema Changes Backward-Compatible During Rolling Deployments

During a rolling deployment, old and new instances coexist. Both versions must work with the current schema.

Usually safer changes include:

Adding a nullable column
Adding a table that old code ignores
Adding an optional response or event field
Adding a non-unique index using the database’s online or concurrent mechanism

Potentially breaking changes include:

Dropping or renaming a column
Changing a field’s meaning
Changing a data type in place
Adding NOT NULL before data is populated
Replacing a default while old code depends on it
Tightening a constraint before invalid historical records are repaired

GitLab’s zero-downtime migration guidance highlights a subtle example: older processes can retain stale schema information and depend on a database default while a post-deployment migration removes it. The later insert can then fail. This illustrates why compatibility must be evaluated against every live application version, not just the new one. See Avoiding downtime in migrations.

Adding a Required Column Safely

Do not add a mandatory field and expect every existing row and old instance to satisfy it immediately.

Use this sequence:

Add the column as nullable or with a compatible temporary default.
Deploy writers that populate it.
Backfill historical rows.
Verify that no invalid rows remain.
Add the constraint using an engine-appropriate low-lock method.
Remove a temporary default only after old writers are gone.

Creating Indexes Carefully

Index creation on a large busy table can block writes or consume significant I/O and CPU.

PostgreSQL supports:

CREATE INDEX CONCURRENTLY idx_orders_customer_id
ON orders (customer_id);

The PostgreSQL documentation states that CREATE INDEX CONCURRENTLY can create an index without locking out writes, although it takes more work and has restrictions. Check the exact behaviour and failure recovery for the database version in use. See the official CREATE INDEX documentation.

Do not copy PostgreSQL syntax into MySQL, SQL Server or another engine. Each database has different online DDL capabilities and locking behaviour.

7. Separate Schema Migration from Large Data Backfills

A schema migration should normally be short and predictable. Updating millions of records inside the deployment migration can cause:

Long locks
Transaction-log growth
Replica lag
Increased I/O
Request latency
Deployment timeout
Difficult recovery after partial failure

Use a background migration for large data changes.

A good backfill worker should be:

Batched: processes a limited number of records at a time
Resumable: continues from a checkpoint after interruption
Idempotent: safely processes a record more than once
Throttled: slows down when database health degrades
Observable: exposes progress, failures and estimated completion
Auditable: records the migration version and transformation rule
Cancellable: can stop without corrupting partially processed data

GitLab recommends batched background migrations when data migrations exceed its normal migration time limits. Its batched background migration documentation is a useful production example.

Example Backfill Loop

repeat:
    claim the next ID range
    update no more than 1,000 rows
    commit
    record the checkpoint
    measure latency and replica lag
    pause if safety limits are exceeded
until no rows remain

Prefer a stable indexed cursor such as the primary key over deep OFFSET pagination. Offset queries become increasingly expensive and can behave unpredictably while rows are inserted or deleted.

Do not assume one batch size is correct for every environment. Tune it using production-like data and real monitoring.

8. Run Migrations as a Controlled Deployment Step

Avoid allowing every application replica to run migrations automatically at startup. If 20 pods start together, they may compete for locks, repeat work or leave the deployment in an unclear state.

A safer pattern is one controlled migration job that:

Acquires an advisory or migration lock.
Checks the expected current schema version.
Runs a bounded migration.
Records success or failure.
Releases the lock.
Allows the application rollout to proceed only when appropriate.

In Kubernetes, a Job represents a one-off task that runs to completion, according to the official Kubernetes Job documentation. Teams commonly use a dedicated Job or an equivalent release phase for migrations.

An init container can also run setup before an application container, but attaching a database migration to every pod startup can create concurrency and rollout problems. Choose the mechanism based on deployment guarantees, not convenience.

Additional controls should include:

One migration owner per service
Explicit timeouts
Lock timeouts
Statement timeouts
Preflight checks
An append-only migration history
Checksums or immutability for released migration files
Alerts on failed or unusually slow migrations

Never silently edit a migration that has already run in production. Add a new corrective migration.

9. Move Data with Snapshot, Change Capture and Reconciliation

Copying a table while the application is writing to it creates a moving target.

A robust live-data migration often uses four stages.

Stage 1: Initial Snapshot

Copy the historical records from the source to the destination.

Preserve stable identifiers where possible. Record the snapshot boundary, such as a log sequence number, timestamp or change-stream offset.

Stage 2: Capture Ongoing Changes

While the snapshot runs, capture inserts, updates and deletes that occur in the source.

Possible methods include:

Change data capture from the database log
A transactional outbox
Existing domain events, if their completeness has been proven
A temporary compatibility layer

Stage 3: Catch Up and Reconcile

Apply captured changes until lag is acceptably low. Then compare source and destination.

Useful checks include:

Total row count
Count by tenant, date or status
Sum of important monetary fields
Minimum and maximum identifiers
Null and constraint violations
Hashes of canonical record representations
Random and risk-based samples
Business outcomes, such as number and value of completed orders

A single total row count is not enough. Two datasets can have the same number of rows and different contents.

Stage 4: Cut Over Ownership

Move reads and writes to the destination gradually. Continue monitoring changes and discrepancies during a defined confidence period.

The cutover plan must state which system is authoritative at every stage. Avoid a period in which teams cannot answer, “If the two values disagree, which one wins?”

10. Avoid Unsafe Dual Writes

Consider this code:

save order in database
publish OrderCreated event

Two failures are possible:

The database commit succeeds but event publication fails.
The event is published but the database transaction later fails.

Retrying the whole operation can also create duplicates.

This is the dual-write problem. The AWS transactional outbox guidance explains how separate database and message operations can leave distributed systems inconsistent.

Transactional Outbox Pattern

Write the business record and an outbox record in the same local database transaction:

BEGIN;

INSERT INTO orders (
    id,
    customer_id,
    status,
    total_amount
) VALUES (
    :id,
    :customer_id,
    'created',
    :total_amount
);

INSERT INTO outbox_events (
    event_id,
    aggregate_type,
    aggregate_id,
    event_type,
    payload,
    created_at
) VALUES (
    :event_id,
    'order',
    :id,
    'OrderCreated',
    :payload,
    CURRENT_TIMESTAMP
);

COMMIT;

A separate relay or CDC connector publishes committed outbox rows to the broker. Debezium provides an Outbox Event Router specifically for this pattern.

The outbox improves reliability, but consumers must still expect duplicates. The relay may publish an event and fail before recording completion.

Idempotent Consumer

Give every event a stable ID. The consumer records processed IDs in the same transaction as its local state change:

BEGIN;

INSERT INTO processed_messages (
    consumer_name,
    event_id,
    processed_at
) VALUES (
    'billing-service',
    :event_id,
    CURRENT_TIMESTAMP
)
ON CONFLICT DO NOTHING;

-- Continue only if the insert affected one row.
-- Apply the business update here.

COMMIT;

The exact SQL differs by database. The important property is that deduplication and the local business update succeed or fail together.

11. Replace Cross-Service Transactions with Explicit Workflows

Once each service owns its database, a single ACID transaction normally cannot cover the complete business process.

For example:

Create order → Reserve stock → Authorise payment → Arrange shipment

Use a saga when several local transactions form one distributed business workflow.

A saga defines:

The forward steps
The responsible service for each step
Retry rules
Timeouts
Idempotency behaviour
Compensating actions
The point after which compensation is no longer appropriate
The final failure states visible to users and operators

Example compensation:

If payment fails after stock is reserved:
    release the reservation
    mark the order as payment_failed

Compensation is a business action, not a database rollback. Refunding a payment is a new auditable transaction; it does not erase the original charge.

The Azure Saga pattern guidance notes that retryable saga operations should be idempotent so a temporary failure does not create repeated side effects.

Use choreography for small, understandable event flows. Consider orchestration when the process has many participants, deadlines, complex compensation or a strong need for central visibility. Neither style removes the need for service autonomy, tracing and failure handling.

12. Evolve APIs Without Breaking Consumers

Service providers and consumers are rarely deployed at exactly the same time.

Prefer additive changes:

Add an optional field.
Add a new endpoint or operation.
Accept both old and new input representations temporarily.
Preserve existing status codes and field meanings.
Use tolerant readers where appropriate.

Treat these as breaking changes unless proven otherwise:

Removing or renaming a field
Changing a field from optional to required
Changing units, timezone or semantic meaning
Returning a different error code
Changing empty data from [] to null
Changing pagination behaviour
Reusing a field for a different purpose

Version Only When Necessary

API versioning can help manage unavoidable breaking changes, but it is not a replacement for compatibility discipline.

When a new version is necessary:

Publish a deprecation policy.
Identify every consumer.
Measure usage of the old version.
Provide a migration guide.
Set a realistic removal date.
Alert owners who still use the old contract.
Keep both versions until the exit criteria are met.

Use Consumer-Driven Contract Tests

Contract tests verify the behaviour that each consumer actually depends on. They can reject a provider deployment that breaks a known consumer before production.

Pact describes itself as a code-first consumer-driven contract testing tool and generates contracts from consumer tests. See the official Pact introduction.

Contract tests complement—not replace—unit, integration, security, performance and end-to-end tests.

13. Treat Event Schemas as Long-Lived Contracts

An event can remain in a topic, retry queue, archive or replay system long after the producer deployment.

Every event should have a clear envelope, for example:

{
  "event_id": "01JXYZ...",
  "event_type": "OrderCreated",
  "schema_version": 2,
  "occurred_at": "2026-08-06T08:30:00Z",
  "correlation_id": "req-12345",
  "producer": "order-service",
  "aggregate_id": "ord-9081",
  "payload": {
    "customer_id": "cus-42",
    "currency": "MYR",
    "total_amount": "129.90"
  }
}

Best practices include:

Use stable event IDs.
Include an event occurrence time, not only publication time.
Preserve correlation and causation information.
Add optional fields instead of changing existing meanings.
Never reuse an event name for different semantics.
Do not renumber or reuse removed Protobuf field numbers.
Validate compatibility in CI.
Test old consumers against new producer schemas.
Retain readers for historical formats when replay is required.

Confluent’s schema evolution documentation distinguishes backward, forward and full compatibility. Select the compatibility rule based on deployment and replay requirements rather than relying blindly on a registry default.

Events Should Describe Facts

Prefer:

OrderCancelled

over an ambiguous database-shaped event such as:

OrdersTableRowUpdated

A domain event should express what happened. Exposing internal table structure couples consumers to the producer’s storage model and makes future migrations harder.

14. Make Retries Safe

Retries are normal in distributed systems. Timeouts do not prove that an operation failed; the remote service may have completed it before the response was lost.

For commands that cause side effects, accept an idempotency key:

POST /payments
Idempotency-Key: checkout-874-payment-1

The receiving service stores the key and the result. A retry returns the original result or safely resumes the operation instead of creating another payment.

The same principle applies to:

Backfill batches
Event consumers
Saga steps
File imports
Webhook handlers
Traffic replay

Azure’s microservice assessment guidance describes deriving and storing idempotency keys so retried work can be detected and skipped safely. See Microservices assessment and readiness.

Do not claim “exactly once” merely because the broker offers an exactly-once feature. End-to-end behaviour also includes databases, external APIs and application side effects. Design the business operation itself to tolerate repetition.

15. Control Cutover with Flags and Routing

A migration should have a control plane that allows operators to change behaviour without an emergency rebuild.

Useful controls include:

Feature flag by tenant or user cohort
Percentage-based traffic routing
Route by endpoint
Route by region
Read-source selection
Write-path selection
Event-consumer enablement
Shadow traffic
Emergency kill switch

A sensible rollout might be:

Internal users → 1% → 5% → 25% → 50% → 100%

Promotion should depend on measurements, not a fixed timer alone.

At each stage, compare:

Error rate
Latency percentiles
Saturation
Data mismatch rate
Event lag
Duplicate rate
Business conversion or completion rate
Support incidents

Shadow Traffic

Shadowing sends a copy of production requests to the new service but does not use its response for the user.

It is useful for checking:

Compatibility
Performance
Result differences
Unexpected input shapes

Shadow requests must not create real side effects. Disable writes or direct them to an isolated destination. Remove or protect personal and secret data according to the organisation’s security requirements.

16. Build Observability Before Cutover

Do not wait for a migration failure to decide what should be measured.

Technical Metrics

Request rate, error rate and duration
Database CPU, I/O, locks and connection usage
Query latency
Replica lag
Queue depth and consumer lag
Retry and dead-letter counts
Backfill throughput and remaining records
CDC lag
Cache hit rate
Resource saturation

Migration Metrics

Percentage of traffic on the new path
Records copied
Records remaining
Source-destination mismatches
Writes handled by each path
Old API or event-schema usage
Number of unknown consumers
Compensation rate

Business Metrics

Orders created
Payments completed
Inventory reserved
Invoices generated
Total monetary value
User completion rate

A service can return HTTP 200 while calculating the wrong total. Business invariants often detect migration defects faster than infrastructure metrics.

Use consistent correlation IDs across HTTP calls, events, logs and saga steps. Distributed traces help reveal which path processed a request, but traces should be combined with metrics, logs and reconciliation reports.

17. Design Rollback Before Deployment

“Redeploy the previous image” is not a complete rollback plan.

After a migration begins, the new version may have:

Written data the old version cannot understand
Published events old consumers cannot parse
Moved ownership to another database
Triggered external side effects
Applied an irreversible schema change

For each phase, decide whether the response is:

Traffic rollback: route users back to the old path
Code rollback: deploy the previous compatible version
Data restore: restore a verified backup or point-in-time copy
Compensation: perform a business reversal
Roll-forward: fix the new path while preserving already committed state

Many production migrations are safer to roll forward after data has changed. The runbook should state the point of no return and the approved response after it.

A Useful Rollback Matrix

Phase	Failure	Response
Expand schema	Migration lock timeout	Stop migration; application remains on old schema
Backfill	High replica lag	Pause worker and resume from checkpoint later
Shadow traffic	Result mismatch	Keep responses non-authoritative; investigate
5% cutover	Error rate exceeds limit	Route traffic back to legacy path
New writes authoritative	Consumer defect	Fix or roll forward; replay retained events
Old column removed	Old code requested	Restore only if recovery plan supports it; otherwise roll forward

Test the recovery path in a production-like environment. An untested rollback plan is an assumption.

18. Secure the Migration Path

Temporary migration components often receive broad access and may be forgotten after cutover.

Apply normal production security controls:

Least-privilege source and destination credentials
Separate identities for schema migration, backfill and application runtime
Encrypted connections and storage
Secret rotation
Audit logging
Network restrictions
Masked or synthetic non-production data
Access expiry for temporary tools
Validation and safe parsing of imported records
Approval for destructive operations

Do not place credentials in migration scripts, container images or command history.

If personal data moves to a new service, update:

Data inventory
Retention and deletion processes
Data-subject request workflows
Encryption-key ownership
Backup and restore scope
Regional residency controls
Audit evidence

Migration completion includes removing temporary accounts, firewall rules, topics, buckets and data copies.

19. Test the Migration as a State Machine

Testing only the final architecture misses the riskiest part: intermediate states.

Test at least these states:

Old code + old schema
Old code + expanded schema
Old and new code + expanded schema
New code + partially backfilled data
New code + completed backfill
New code + contracted schema

Important test categories include:

Migration Tests

Upgrade from a realistic previous production version
Re-run an already completed migration
Interrupt and resume a backfill
Apply the migration to production-sized data
Verify lock and duration limits
Verify downgrade or roll-forward behaviour

Contract Tests

Old consumer against new provider
New consumer against old provider during the transition
Old event reader against new producer schema
Replay historical events through the new consumer

Failure Tests

Database unavailable during a batch
Broker publish timeout
Duplicate event delivery
Out-of-order event delivery
Consumer crash after business commit but before acknowledgement
CDC pause and recovery
Network partition during cutover
Partial destination outage

Reconciliation Tests

Intentional missing row is detected
Intentional field mismatch is detected
Duplicate record is detected
Monetary totals use correct precision and currency
Deleted records or tombstones are handled correctly

Use anonymised production-like volume and data distribution. A migration that completes in seconds on 1,000 test rows may behave very differently on 500 million production rows.

20. Create a Migration Runbook

The runbook should be executable by an engineer who did not design the migration.

Include:

Purpose and scope
Owners and communication channel
Architecture before and after
Dependency inventory
Preconditions
Backup and restore verification
Exact deployment sequence
Flags and routing controls
Validation queries
Dashboards and alert links
Go/no-go thresholds
Pause, abort and roll-forward instructions
Escalation contacts
Expected duration for each stage
Cleanup tasks
Evidence to retain

Conduct a rehearsal. Record actual timings and revise the runbook instead of assuming the first plan is accurate.

For a high-risk migration, assign explicit roles:

Migration lead
Database operator
Service owner
Observability lead
Business validator
Incident commander if thresholds are breached
Communications owner

This reduces confusion during a time-sensitive cutover.

Complete Example: Extracting an Order Service

Assume a monolith owns order creation, and the goal is a new Order Service with its own database.

Step 1: Discover

The team finds these dependencies:

Checkout creates orders.
Billing reads order totals.
Warehouse reads new orders from a scheduled query.
Customer support searches the monolith database.
Finance exports completed orders nightly.

The warehouse job and support search were not present in the original architecture diagram.

Step 2: Establish the Contract

Create:

POST /orders
GET /orders/{id}
OrderCreated
OrderStatusChanged

Add contract tests for checkout and billing. Design event fields as backward-compatible schemas.

Step 3: Prepare the Destination

Create the Order Service database and schema. Set service-specific credentials. Add dashboards, alerts and audit logs before sending production traffic.

Step 4: Backfill History

Copy orders in stable ID batches. Record each checkpoint and the source change-stream position.

For every batch:

Insert or update idempotently.
Compare count and monetary total.
Log invalid records separately.
Stop automatically if error or database-load thresholds are exceeded.

Step 5: Capture New Changes

Use CDC or a transactional outbox to transfer changes that occur after the snapshot boundary.

Measure:

Change lag
Failed records
Duplicate deliveries
Source-destination mismatch

Step 6: Shadow Reads

The production response still comes from the monolith, but the application also queries Order Service asynchronously and compares canonical results.

Differences are categorised instead of merely counted:

Expected formatting difference
Timing or eventual-consistency difference
Missing record
Incorrect status
Incorrect financial value
Unknown

Step 7: Move Consumers

Move billing, warehouse, support and finance to the published API or event projections. Deny new direct database integrations.

Monitor legacy access logs until no known consumer reads the old tables.

Step 8: Shift Reads

Move internal users first, followed by 1%, 5%, 25%, 50% and 100% of eligible traffic. Hold each stage until the defined error, latency and mismatch thresholds pass.

Step 9: Shift Writes

Make Order Service the authoritative writer. Ensure commands are idempotent and events use the transactional outbox.

Do not allow both systems to accept unrelated authoritative writes. If a transitional write path is needed, define one owner and one replication direction.

Step 10: Stabilise

Run at 100% while retaining the ability to route reads or commands according to the approved recovery plan. Continue reconciliation for the agreed confidence period.

Step 11: Contract and Clean Up

After exit criteria pass:

Remove legacy writes.
Revoke old database access.
Disable temporary CDC or replication if no longer required.
Archive or remove obsolete tables according to retention policy.
Remove old flags, routes and code.
Update architecture and operational documentation.
Record final evidence and lessons learned.

The migration is complete only after cleanup. Permanent compatibility code and unused data pipelines become future failure points.

Common Microservice Migration Mistakes

1. Performing the Schema and Code Change in One Release

This assumes every instance and consumer changes simultaneously. Rolling deployments and independent consumers make that assumption unsafe.

Use compatible stages.

2. Moving Code Without Moving Ownership

If the new service still depends on direct writes to the monolith’s tables, the old coupling remains.

Define a real system of record and remove cross-service writes.

3. Running a Huge Backfill During Deployment

Long data updates make deployment duration unpredictable and can overload the database.

Use a separate resumable background process.

4. Trusting Dual Writes Without Failure Analysis

Two writes to two systems are not atomic. A success response from one does not guarantee the other succeeded.

Use outbox, CDC or another design with explicit consistency guarantees.

5. Assuming Messages Are Delivered Once and in Order

Retries, rebalances and network failures can produce duplicates or reordering.

Use stable IDs, idempotency, version checks and ordering only where the business requires it.

6. Testing Only with Empty Databases

Empty-database tests do not reveal lock duration, invalid historical data, volume, skew or backfill behaviour.

Test upgrades using realistic data.

7. Monitoring Only CPU and HTTP Errors

Infrastructure can be healthy while orders, invoices or balances are wrong.

Monitor business invariants and reconcile data.

8. Keeping the Old Path Forever

Temporary routes, flags and dual-read code create permanent complexity when no removal date exists.

Set exit criteria and a cleanup owner before the migration begins.

9. Deleting Old Data Too Early

Early deletion eliminates recovery and comparison options.

Retain data according to a documented recovery, compliance and cost decision—not an improvised cleanup.

10. Calling a Backup a Rollback Plan

A backup is useful only if it is complete, recent, restorable and compatible with the intended recovery point.

Test restoration and measure how long it takes.

Migration Readiness Checklist

Planning

Business reason and scope are documented.
Current and future system of record are identified.
API, event, database, batch and reporting dependencies are inventoried.
Success, pause and abort thresholds are measurable.
Data volume and expected migration duration are estimated.
Security, privacy and retention requirements are reviewed.

Design

Changes are backward-compatible during mixed-version operation.
Expand, migrate and contract releases are separated.
Large backfills are batched, resumable and idempotent.
The dual-write problem is addressed.
Consumers tolerate duplicates and required reordering scenarios.
API and event contracts have a compatibility strategy.
The authoritative owner is clear at every phase.

Testing

Migration scripts run against production-like data volume.
Intermediate schema and application combinations are tested.
Contract tests cover known consumers.
Historical events can be replayed where required.
Backfill interruption and resumption are tested.
Reconciliation detects injected errors.
Rollback or roll-forward procedures are rehearsed.

Deployment

Migration execution is controlled and locked.
Backups and restoration are verified.
Flags or routing controls are ready.
Dashboards and alerts exist before cutover.
Correlation IDs connect requests, events and logs.
Owners and escalation paths are available.

Completion

Traffic is fully on the intended path.
Reconciliation meets the agreed threshold.
Old consumers and direct database readers are gone.
The confidence period has completed.
Obsolete schemas, routes, events and flags are removed.
Temporary credentials and infrastructure are revoked.
Documentation and incident lessons are updated.

Frequently Asked Questions

What is the safest migration strategy for microservices?

For most live systems, the safest general approach is an incremental migration using expand-and-contract changes, gradual traffic shifting, continuous reconciliation and explicit recovery controls. The exact tools depend on the database, broker and deployment platform.

What is an expand-and-contract migration?

It divides a breaking change into compatible stages. First, expand the system by adding the new field, endpoint or schema. Next, migrate consumers and data. Finally, contract the system by removing the obsolete interface after it is no longer used.

Can microservices share one database?

They can share physical infrastructure, especially during transition, but each service should have clearly controlled schema and data ownership. Other services should not directly write the owner’s tables. Separate credentials and permissions help enforce the boundary.

Should database migrations run when the application starts?

Small systems sometimes use this approach, but it becomes risky when many replicas start simultaneously. A controlled, single migration job with locking, timeouts and recorded status is usually easier to reason about.

How can a large table be migrated without downtime?

Add compatible destination structures, copy data in resumable batches, capture ongoing changes, reconcile source and destination, shift traffic gradually, then remove the old structure only after the new path is stable.

Is dual writing to old and new databases safe?

Not automatically. One write can succeed while the other fails. If temporary duplication is required, define the authoritative source, failure behaviour, repair process and reconciliation. Prefer patterns such as transactional outbox and CDC where they fit.

What is the difference between CDC and a transactional outbox?

CDC captures committed changes from a database log. A transactional outbox deliberately stores domain-event records in an outbox table within the same transaction as the business update; a relay or CDC tool then publishes them. Raw CDC exposes storage-level changes, while an outbox can publish intentional domain contracts.

How do you prevent duplicate event processing?

Give each event a stable ID and make consumers idempotent. Record the processed ID atomically with the local business change, or design the state transition so repeating it has no additional effect.

Should every breaking API change create a new version?

Not every change is breaking. Prefer additive, backward-compatible evolution. Create a new version when compatibility cannot reasonably be preserved, then publish a deprecation plan and measure old-version usage until all consumers migrate.

How should event schemas be versioned?

Choose explicit backward, forward or full compatibility based on producer and consumer rollout order and replay requirements. Enforce the rule in CI or a schema registry, retain stable field meanings and test older consumers with newer schemas.

What should be monitored during cutover?

Monitor technical signals such as errors, latency, locks and event lag; migration signals such as traffic percentage, backfill progress and mismatch rate; and business signals such as completed orders, payments and monetary totals.

Is rollback always possible?

No. After new data formats, external actions or destructive changes occur, deploying the old code may be unsafe. Define rollback and roll-forward options for each phase and identify the point after which roll-forward is the approved response.

When is a migration finished?

It is finished when the new owner handles the intended traffic, data has been reconciled, old consumers have moved, the confidence period has passed and temporary code, access, infrastructure and old schemas have been safely removed.

Final Summary

Reliable microservice migration is a compatibility and data-correctness problem, not merely a deployment task.

The most important practices are:

Discover real dependencies before changing ownership.
Define one authoritative owner at every stage.
Migrate incrementally instead of using a big-bang rewrite.
Use expand-and-contract changes for databases, APIs and events.
Keep large backfills separate, batched and resumable.
Use transactional outbox or CDC to avoid unsafe dual writes.
Make commands, consumers and migration jobs idempotent.
Reconcile data using business invariants, not only row counts.
Shift traffic gradually with measurable gates.
Design recovery before the first production change.
Remove temporary migration paths after a confidence period.

The ideal migration may look slower on a project plan because it uses several releases. In production, those smaller reversible steps usually reduce risk, reveal problems earlier and allow normal product development to continue while the architecture evolves.

Best Practices for Migrations in Microservices: A Zero-Downtime Guide