Skip to main content

What are you looking for?

Explore our services and discover how we can help you achieve your goals

Client Workspace

Change data capture for operational databases: Debezium, native CDC or polling

Use the lightest change data capture method your consumers can live with: a nightly snapshot or timestamp polling covers most reporting, while native database CDC or log-based CDC with Debezium earns its cost only when consumers need deletes, every change in order and fresh data. Netbase scores each source table before choosing.

Book a solution review See the related solution

By Netbase Commerce and Platform Engineering Desk · Reviewed by David (CEO) · Updated · 10 min read

star

This guide is for data and platform engineering leads, CTOs and IT managers who need rows from an operational database (PostgreSQL, MySQL or SQL Server behind an ERP, a store or a SaaS product) to reach a warehouse, a search index or another service. It sits beside Netbase's data engineering platforms service and covers one decision inside a pipeline project: how changes leave the source database.

In this guide

What problem does change data capture solve, and what does it cost?

Change data capture (CDC) turns the changes in a database into a stream that another system can apply. The alternative is to copy whole tables or to ask the tables for recently changed rows. CDC solves three problems that copying cannot: it sees deletes, it sees every intermediate change rather than only the latest state, and it keeps the order in which changes were committed.

The cost is operational. A log-based reader becomes a client of the database's transaction log, so somebody has to watch it, size the log retention behind it and plan for schema changes. The PostgreSQL documentation on logical decoding warns that a replication slot keeps the write-ahead log it still needs, so an abandoned or lagging consumer makes the primary database keep log files it would otherwise remove. That is the trade: better data for the consumer, a new moving part for the team that runs the source.

When is a nightly snapshot enough?

A nightly snapshot is enough when the business reads the data the next morning, when the tables fit inside the batch window and when nobody needs to know what happened between two snapshots. Finance reporting, monthly dashboards and most first warehouse projects fit that description, and a snapshot can be rerun from scratch when something breaks.

Keep the snapshot and skip CDC when all of these hold:

  • The consumer accepts data that is up to a day old, and no process acts on a change within the day.
  • Deletes either do not happen (soft-delete flags) or can be found by comparing two full copies.
  • The source can be read at night without hurting users, ideally from a read replica.
  • Nobody needs the history of a row between snapshots, for audit or for event-driven services.

When one of these fails for one table, change the method for that table, not for the whole database.

Three capture methods compared

The matrix below scores the three families on the criteria that decide a choice. It describes the methods, not a vendor ranking; native CDC and Debezium are both log-based and are compared in the next section.

Criterion Nightly snapshot Timestamp polling Log-based CDC
Load on the source A full read in the batch window A filtered read on every poll, needs an index on the change column A log reader, plus an initial snapshot
Deletes Found only by comparing copies Missed unless rows are soft-deleted Captured
Intermediate changes Lost Lost between polls Kept
Ordering Not relevant By timestamp, unreliable for equal or late-committed values By log position per table and key
Schema drift Picked up on the next full copy Picked up if queries select named columns Tool-specific; some sources do not report DDL
Operating burden Lowest Low Medium to high: log retention, offsets, alarms
Choose it when Next-day data is enough Few tables, no hard deletes, minutes are fine Consumers need deletes, order and low lag

Native CDC or Debezium: which log reader fits?

  • Sources

    Native database CDC
    One engine, its own feature or a managed service
    Debezium
    Many engines through one connector model
  • Consumers

    Native database CDC
    Tools in the same estate read change tables
    Debezium
    Several services share one change stream
  • Operating burden

    Native database CDC
    Database jobs and retention
    Debezium
    Connectors, offsets and a message platform
  • Choose it when

    Native database CDC
    One database, a team that already runs it
    Debezium
    Several sources or consumers, and a platform team

Native CDC differs by engine. SQL Server's built-in feature, described in Microsoft's change data capture overview, reads the transaction log into change tables ordered by log sequence number and keeps a fixed column shape for each capture instance when the source table changes. Managed services read the same logs: the AWS Database Migration Service guide to ongoing replication explains that it uses row-based binary logs for MySQL and logical replication slots for PostgreSQL, and states that its CDC is not real-time replication.

Debezium runs as a connector per source. Its PostgreSQL connector documentation describes a consistent snapshot on first connection, then streaming through logical decoding and a replication slot, and notes that logical decoding does not report DDL changes, so schema drift has to be handled outside the stream.

How do ordering and schema drift break a CDC pipeline?

Ordering breaks when the consumer applies changes for the same row out of sequence. The fix is to key every change by primary key, carry the source log position with it and make the target merge idempotent, so a replayed or late event cannot overwrite a newer one.

Schema drift breaks a pipeline when the source team renames or retypes a column and the consumer still expects the old shape. Log-based tools differ: some keep the old shape, some report nothing, some pass the new column through. Treat a schema change on a captured table as a release with its own checklist, not as something the pipeline discovers.

How to choose and roll out a capture method

  1. List the consumers per table

    Write down who reads each table, how fresh the data must be and whether deletes and history matter to them. Most tables usually land on snapshot or polling.

  2. Check the source

    Confirm the database engine and version, whether a read replica exists, whether the log format and retention support CDC, and who on the source team approves a log reader.

  3. Pick the method per table

    Use the matrix above; mix methods when tables differ instead of forcing one choice on the whole database.

  4. Design the merge

    Define the key, the ordering field, the delete handling and an idempotent upsert on the target before the first event flows.

  5. Plan the initial load

    Decide how the first full copy and the start of the stream line up, so no change falls in the gap between them.

  6. Set alarms on lag and retention

    Alert on consumer lag, on log growth caused by a stalled slot or capture job, and on reconciliation counts between source and target.

  7. Write the schema-change rule

    Agree with the source team how a captured table changes: notice period, test in staging, and who updates the consumer.

  8. Rehearse a restart

    Stop the connector or job, let changes pile up, restart it and check that the target catches up without duplicates or gaps.

Where AI fits in a change data capture project

AI helps with the reading and checking around the pipeline, not with moving the changes. An assistant can draft the table-by-table consumer inventory from schema exports and query logs, summarise reconciliation reports and flag a schema change in a pull request that touches a captured table. Netbase works with the major commercial and open-source AI tools and models, chosen per project.

Engineers still choose the capture method, write the merge logic, set the alarms and approve every schema change on a captured table; no model changes a connector, a replication slot or a target table on its own.

What delivery record exists, and what does not

Netbase's ERP and e-commerce work includes integration with ERP through API sync of product catalogues, inventory, orders and customer data, and cloud ERP platform development, such as the multi-tenant cloud ERP record. Those projects move operational data between systems, which is the problem this guide is about, but they are API integrations rather than database log capture.

Delivery runs in a six-step lifecycle from discovery and architecture planning to rollout and ongoing support, and most Netbase projects are delivered on fixed-price contracts. No published Netbase record describes a Debezium or native database CDC deployment, and no latency, load or result figure is claimed here.

Alternatives: other ways to move operational data

Alternative Fits when Selection criterion
Application events (outbox table) The team owns the application code Business events matter more than row changes, and the code can write them in the same transaction
API sync between systems Systems expose stable APIs The consumer needs business objects such as orders, not raw rows; see ERP, CRM and e-commerce integration architecture
Read replica queries Reporting can query a copy directly A replica exists and the reports tolerate its lag
One-off delta sync The goal is a migration, not a running feed Changes stop after the switch-over; see the cloud migration cutover plan

When many systems already exchange data through APIs and files, the wider choice between point-to-point links and a hub is covered in the enterprise systems integration guide, and Netbase's integration hub adds monitoring, retries and replay for those flows.

What are the limits of this guide?

  • It compares methods and gives decision rules; it does not benchmark tools or promise a latency.
  • Connector and engine behaviour changes between versions; check the current documentation of your database and tool before you design the merge.
  • It covers relational operational databases; document stores and SaaS application APIs need their own review.

Plan the next step with a Netbase consultant

Frequently asked questions

Usually not at the start. A nightly snapshot or timestamp polling covers most reporting. Add CDC for the tables where consumers need deletes, the history of each change or data fresher than the batch window allows.

Neither is better in general. Native CDC suits one engine run by a team that already operates it, while Debezium suits several sources or consumers that share one change stream, at the cost of running connectors and a message platform.

It reads the log instead of querying tables, which is often lighter than polling, but it is not free. A stalled consumer can make the database keep log files, so monitor lag and log retention from the first day.

Treat it as a release. Agree a notice period with the source team, test the change in staging with the connector running, update the consumer's mapping and only then apply the change in production.

Next step

Netbase can score your source tables against the four methods and design the pipeline behind them as part of a data engineering platforms engagement or through systems and API integration. Book a data pipeline review.

Data engineering platforms for reporting and AI Data engineering platforms for reporting and AI

Netbase builds data pipelines and analytics foundations for companies whose data is spread across a store, an ERP, a CRM and spreadsheets. We collect it, clean it, model it and check its quality, so reports agree with each other and AI projects start from data people trust. Data engineering is a capability we are growing alongside the platforms we deliver.

Learn More
line
API integration services that keep your systems in agreement and ready for AI API integration services that keep your systems in agreement and ready for AI

Netbase provides API integration services for commerce and operations teams that need e-commerce, ERP, CRM, shipping and payment systems to agree, so data moves once and AI services can act on it, with retries, monitoring and reconciliation on every API and webhook. According to its published client record, Netbase has worked with 500+ businesses across 80 countries since 2012.

Learn More
line
Enterprise integration hub ready for AI agents: one monitored layer for commerce, ERP and CRM data Enterprise integration hub ready for AI agents: one monitored layer for commerce, ERP and CRM data

An enterprise integration hub is a central layer that moves orders, customers, stock and invoices between your commerce, ERP and CRM systems, with monitoring, retries and replay in one place, and scoped APIs that AI agents can call safely. Netbase builds it as a custom layer on the stack you already run, starting from the flows that break most often.

Learn More
line

Work with Netbase remotely

Every Netbase engagement is delivered remotely, so clients in any country and any industry can work with us; Netbase has worked with clients in 80 countries. Contracts and invoices are issued by Netbase JSC, and payment is made by bank transfer to Netbase JSC's company account or PayPal Business. Talk to the CEO, Huy Nguyen (David), on WhatsApp or by email.

Contact Netbase

Discuss a project

Netbase JSC helps organizations design, build, modernize, and operate digital products and AI-enabled business systems.
Project enquiries

project@netbasejsc.com

WhatsApp

+84 937 869 689

Office address

91 Nguyen Chi Thanh, Dong Da, Hanoi, Vietnam

Get in touch

Tell us what you want to build, modernize, or operate.

Tell us what you want to build, modernize, or operate.

Contact Netbase