Every system eventually needs its database changes somewhere else — a search index, a cache, an analytics warehouse, another service. The tempting shortcut is the dual write: update Postgres, then publish to Kafka from the same code path. Change data capture exists because that shortcut reliably corrupts data. The two writes are not atomic, and under crashes, retries, and partial failures the database and the stream drift apart in ways nobody notices until reconciliation week.
CDC flips the dependency. The database's write-ahead log is already a totally ordered, durable record of every committed change; Debezium reads it through logical replication and publishes each change to Kafka. Consumers get exactly what the database committed, in commit order, with no application cooperation required.
Debezium Mechanics Worth Understanding
Debezium's Postgres connector creates a logical replication slot and decodes WAL through the pgoutput plugin. Three operational details matter more than any tutorial admits.
- ▸Slots retain WAL: if the connector stops, the slot holds WAL on the primary indefinitely. Monitor slot lag in bytes and alert early — a connector paused on Friday becomes a full disk on Sunday. Set max_slot_wal_keep_size as a circuit breaker, and accept that tripping it means re-snapshotting.
- ▸Snapshots are expensive: the initial snapshot reads entire tables before streaming begins. For large tables, use incremental snapshots — signal-driven and chunked — so you can capture history without blocking the live stream or locking the table.
- ▸REPLICA IDENTITY shapes the payload: by default, update events carry the new row and the primary key. If consumers need previous values, or you have TOAST-ed columns arriving as unchanged placeholders, you need REPLICA IDENTITY FULL and the write amplification it brings. That is a real trade — decide it per table.
Schema drift is the quiet killer. Pair Debezium with a schema registry using Avro or Protobuf rather than loose JSON, so a column rename becomes an explicit compatibility decision instead of a consumer crash at 02:00.
The Outbox Pattern for Domain Events
Raw table changes are an implementation detail; broadcasting them couples every consumer to your schema. The outbox pattern keeps CDC's atomicity while publishing intentional events: the service writes its business rows and an event row into an outbox table in the same transaction, Debezium captures the outbox inserts, and its event router turns them into clean, versioned domain events on named topics.
You get exactly the guarantee dual writes promised and never delivered — an event exists if and only if the transaction committed — plus a contract you can evolve deliberately instead of leaking column names across team boundaries. Prune the outbox with periodic deletes or partition drops; Debezium has already shipped the rows, so retention there is purely operational.
At-Least-Once Is the Honest Contract
Kafka plus Debezium delivers at-least-once by default: after a crash, the connector re-emits events from its last committed offset, and consumers will see duplicates. Exactly-once processing exists inside Kafka transactions, but the moment a consumer writes to an external system — Postgres, Elasticsearch, an HTTP API — you are back to needing idempotency at the sink.
Design for it explicitly. Key events by aggregate ID so ordering holds per entity. Make sinks upsert by natural key, or track processed event IDs in the same transaction as the write. Use topic compaction for latest-state topics and plain retention for event history — they answer different questions. Consumers that are idempotent by construction turn redelivery from a bug class into a non-event, which is the entire point.
A Rollout That Does Not Hurt
1. Start with one table and one rebuildable consumer — a search index is ideal, because you can always reindex from scratch. 2. Size WAL retention and wire slot-lag alerts before the connector ever touches production. 3. Put the schema registry in place on day one; retrofitting compatibility rules onto live topics is miserable. 4. Prove correctness by diffing the incrementally maintained index against one rebuilt from a fresh snapshot. 5. Only then adopt the outbox pattern for cross-service domain events, one bounded context at a time.
The payoff is architectural leverage. Once changes flow through CDC, adding a consumer — a cache invalidator, a warehouse feed, a fraud-scoring stream — is configuration, not a code change in every service that writes the table. Teams stop scheduling nightly exports and reconciliation jobs, and downstream systems stop being hours stale. For the business, that is the difference between reporting on yesterday and reacting to right now — and it comes from respecting a log the database was already keeping.