https://www.josiete.com/en/tags/cdc/Recent content in CDC on josiete.comHugoen-usWed, 07 Oct 2026 00:00:00 +0000https://www.josiete.com/en/posts/cdc-change-data-capture/Wed, 07 Oct 2026 00:00:00 +0000https://www.josiete.com/en/posts/cdc-change-data-capture/<p><img src="https://www.josiete.com/images/cdc-change-data-capture.webp" alt="A transactional database sends small blocks of changes to an analytical system"></p> <p>Imagine a shop with ten million orders in its database. The analytics team needs to query today&rsquo;s sales, so it copies those orders into an analytical warehouse every night. If only ten thousand changed today, transferring all ten million again is slow and expensive. <strong>Change Data Capture (CDC)</strong> identifies the changes so the pipeline can send only what is needed.</p> <p>The idea is simple: after creating an initial copy, the pipeline observes which records are inserted, updated, or deleted at the source. Each change becomes an event that the destination can apply. This updates the affected part of the data without rebuilding the entire dataset on every run.</p> <h2 id="from-the-transactional-system-to-analytics">From the transactional system to analytics</h2> <p>The shop uses an <strong>OLTP</strong> database to record orders and payments. It is optimized for many short transactions: creating an order, changing its status, or cancelling a purchase. The business team uses an <strong>OLAP</strong> system — a data warehouse, for example — to aggregate sales by day, product, and region. These are different workloads, and analytical queries should not compete with the shop&rsquo;s operations.</p> <p>Without CDC, a simple solution would periodically export the entire <code>orders</code> table. With CDC, the flow changes:</p> <p><img src="https://www.josiete.com/images/cdc-flow-en.svg" alt="Diagram: an initial copy moves data from OLTP to OLAP; afterwards the transaction log feeds a CDC capture process that sends only inserts, updates, and deletes to the analytical system"></p> <ol> <li><strong>Initial copy:</strong> existing orders are loaded into OLAP. This step is still needed when the destination starts empty.</li> <li><strong>Capture:</strong> the CDC process reads committed changes from the database&rsquo;s transaction log, when the engine and connector support it. PostgreSQL uses its WAL, for example, while MySQL uses its binlog.</li> <li><strong>Apply:</strong> the pipeline transforms events and updates analytical tables. It must preserve the order key and handle deletions correctly.</li> </ol> <p>Suppose the ten million orders have already been copied. Over the next hour, 8,000 are created, 1,500 are updated, and 500 are cancelled. The next step processes <strong>10,000 changes</strong>, rather than another ten million rows. If order <code>P-42</code> moves from “pending” to “paid,” the destination updates that order. If a record is deleted, it receives a deletion event instead of waiting to discover the missing row in a new full export.</p> <h2 id="what-are-the-benefits">What are the benefits?</h2> <ul> <li><strong>Less data transferred:</strong> when only a small part of a table changes, there is no need to move the whole dataset on every cycle.</li> <li><strong>Less repeated work:</strong> full reads at the source and reloads at the destination are reduced, although capture still uses resources.</li> <li><strong>Fresher data:</strong> changes can flow continuously or in frequent batches, depending on the design and capacity of the systems. CDC does not guarantee zero latency.</li> <li><strong>Inserts, updates, and deletes are visible:</strong> a cursor such as <code>updated_at</code> usually finds new and modified rows, but needs another mechanism to detect physical deletes; the transaction log can include them.</li> <li><strong>More possible destinations:</strong> the same events can feed analytics, search, or caches, provided each consumer applies its own logic.</li> </ul> <h2 id="what-needs-attention">What needs attention</h2> <p>CDC is not a “set and forget” feature. The initial load and log reading must be coordinated so changes made during the copy are not lost. The process must also store its <strong>log position</strong> to resume after an interruption, and the source must retain logs long enough for it to catch up.</p> <p>At the destination, applying events by key should be <strong>idempotent</strong>: retrying a delivery must not duplicate an order. Teams also need to monitor source-to-destination lag, schema changes, and the meaning of a deletion in analytical tables. If the saved log position is lost or the log expires before it is read, a new load or reconciliation may be necessary.</p> <h2 id="common-tools">Common tools</h2> <p><a href="https://debezium.io/documentation/reference/stable/architecture.html">Debezium</a> provides connectors that capture database changes and publish them as events. <a href="https://docs.airbyte.com/platform/understanding-airbyte/cdc">Airbyte</a> offers CDC synchronization in supported connectors to move data between a source and a destination. <a href="https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Task.CDC.html">AWS Database Migration Service (DMS)</a> can combine an initial load with ongoing replication, or capture only changes when the destination is already populated. Exact support and setup depend on the database and destination.</p> <p>CDC does not remove the first copy or replace analytical data modeling. Its value lies in what happens afterwards: replacing millions of repeated row copies with a manageable stream of changes, so analytics can keep up with the business without starting over every time.</p>