A sovereign real-time CDC pipeline streams row-level changes out of your operational databases into an analytics lakehouse while every byte, encryption key, and control-plane call stays on infrastructure you can audit, relocate, and leave. This is the reference build we deploy for EU clients: Debezium, Apache Kafka, Apache Flink, and Apache Iceberg, all open source and all self-hostable.
What does a sovereign CDC pipeline look like end to end?
Debezium reads the database write-ahead log and publishes change events to Kafka running on EU infrastructure via the Strimzi operator. Flink consumes, transforms, and upserts into Apache Iceberg tables on S3-compatible EU object storage, tracked by a self-hosted REST catalog. Every component is open source, so residency is verifiable rather than contractual.
The concrete bill of materials:
- ▸Capture: Debezium 3.x reading PostgreSQL logical replication (pgoutput) or the MySQL binlog
- ▸Transport: Apache Kafka 4.x in KRaft mode, deployed with the Strimzi operator on Kubernetes
- ▸Processing: Apache Flink 2.x under the Flink Kubernetes Operator, or plain Kafka Connect for simple landing
- ▸Table layer: Apache Iceberg format v2 behind a self-hosted REST catalog such as Lakekeeper or Apache Polaris (incubating)
- ▸Storage: S3-compatible object storage from an EU provider (OVHcloud, Scaleway, Exoscale) or self-managed MinIO or Ceph
- ▸Schemas: Apicurio Registry or Karapace instead of a proprietary schema registry
Downstream, anything that reads Iceberg works: Trino, Spark, DuckDB, ClickHouse, or a warehouse engine pointed at the same catalog.
Why does sovereignty change the architecture, not just the hosting?
Because a CDC stream is the most sensitive dataset you own: a running, replayable copy of your databases, including before-images of updated and deleted rows. Piping it through a managed service operated from outside the EU places personal data under a foreign control plane. Sovereignty therefore dictates component choices, not just region selection.
Managed streaming offers such as MSK, Confluent Cloud, and Event Hubs trade operational convenience for a control-plane dependency that EU legal teams increasingly question after the Schrems II line of cases and the ongoing US CLOUD Act debate; check the current legal position with counsel rather than blog posts. The architectural consequence is a rule of thumb: open protocol, self-hostable implementation, EU-controlled keys at every stage. The strategic groundwork on provider selection, exit tests, and key custody is covered in Sovereign Cloud and EU Data Residency in 2026: An Engineering Playbook and, for the no-hyperscaler variant, Sovereign-by-Design: Architecting EU and North Africa Data Residency Without a Hyperscaler.
How do you capture changes without hurting the source database?
Use log-based capture only: Debezium reading PostgreSQL logical replication or the MySQL binlog. Source overhead is typically modest, WAL retention plus one replication slot, and no polling queries touch your tables. The real operational risks are replication-slot bloat during consumer outages and the initial snapshot of very large tables.
What we configure on every engagement:
- ▸Monitor replication-slot lag and set max_slot_wal_keep_size so a stalled pipeline cannot fill the primary disk
- ▸Use Debezium incremental snapshots (signal table) so large-table backfills interleave with live changes instead of blocking them
- ▸Set a heartbeat interval on low-traffic databases so slots keep advancing
- ▸Decide REPLICA IDENTITY FULL per table: complete before-images are useful downstream but add write amplification
Replication slots and failover interact badly if unrehearsed; PostgreSQL 17 failover slots help, but you should still drill the scenario as described in PostgreSQL High Availability: The Failover You Rehearse. For the capture patterns themselves, the outbox pattern, snapshot mechanics, and event contracts, see Change Data Capture in Practice: Debezium, Kafka, and the Outbox Pattern; this post builds the sovereign runtime around them.
How should you run Kafka itself on EU infrastructure?
Run Kafka 4.x in KRaft mode with the Strimzi operator on Kubernetes: three dedicated controllers, three or more brokers with rack awareness across availability zones, TLS everywhere, SCRAM or OAuth authentication, and tiered storage offloading cold segments to object storage. Strimzi automates rolling upgrades, certificate rotation, and partition rebalancing.
Details that matter for CDC specifically:
- ▸ZooKeeper is gone as of Kafka 4.0, so any migration plan starts with KRaft
- ▸Partition change topics by primary key so per-key ordering holds end to end
- ▸Keep retention long enough to replay a full reprocessing window; tiered storage makes weeks of retention affordable
- ▸Apply quotas so a misbehaving backfill consumer cannot starve production traffic
- ▸Prefer Apicurio Registry (Apache-2.0 licensed) for Avro or Protobuf schemas; it self-hosts cleanly
If operating Kafka is genuinely out of reach, an EU-operated managed service is a defensible middle ground, but evaluate who runs the control plane and where support engineers sit, not just where the brokers live.
Where does Flink fit, and when is Kafka Connect enough?
If you only need change streams landed one-to-one into Iceberg tables, skip Flink: the Kafka Connect Iceberg sink connector is dramatically simpler to operate. Reach for Flink when you need stateful processing such as streaming joins, deduplication, enrichment against reference data, or windowed aggregates, or sub-minute freshness with exactly-once guarantees.
When Flink does earn its place:
- ▸Flink SQL upsert semantics on the primary key model CDC changelogs naturally
- ▸Checkpoint RocksDB state to your EU object storage; state is data too and inherits residency requirements
- ▸Run one job per domain under the Flink Kubernetes Operator rather than a monolith that fails together
- ▸Flink CDC 3.x can even read databases directly and skip Kafka, but keeping Kafka buys replay, fan-out, and buffering, which is usually worth it once a second consumer exists
How do you land the stream in Apache Iceberg correctly?
Write Iceberg format v2 tables using merge-on-read: streaming upserts produce data files plus delete files, committed atomically at each checkpoint through your REST catalog. Small files are inevitable at streaming cadence, so schedule compaction and snapshot expiry from day one, or query latency and storage cost degrade within weeks.
The operational core:
- ▸Run compaction (rewrite data files) and snapshot expiry as recurring maintenance jobs, sized to your commit cadence
- ▸Use hidden partitioning on event time so analysts never hand-manage partition columns
- ▸Understand GDPR erasure: a delete masks rows logically; the bytes persist until compaction rewrites the files and expiry removes old snapshots, so define your erasure SLA on that full chain
- ▸Choose the catalog deliberately: Lakekeeper is a lean Rust implementation, Apache Polaris (incubating) adds richer RBAC and credential vending; both back onto PostgreSQL you already run
- ▸Enable bucket versioning or object lock for audit-relevant tables
How do you keep the pipeline correct through failures and reprocessing?
Correctness comes in layers: exactly-once delivery from Flink checkpoints plus atomic Iceberg commits; idempotent upserts keyed on primary key and log position as the safety net beneath it; schema evolution constrained by registry compatibility rules; and backfills you have rehearsed, replaying from Kafka or re-snapshotting from the source.
Rules we hold every build to:
- ▸Ordering is guaranteed per key, never globally, so design consumers accordingly
- ▸Route poison messages to a dead-letter topic with alerting instead of stalling the stream
- ▸Let additive schema changes flow automatically; handle renames and type changes as new column, backfill, retire
- ▸Chaos-test before go-live: kill a broker and a Flink TaskManager mid-checkpoint and prove there are no duplicates or gaps
The design discipline is the same one that makes batch reruns safe, covered in Idempotent Data Pipelines: Batch Jobs That Survive Reruns.
What does the stack cost, and where do teams overspend?
On EU providers, a mid-size estate of tens of tables and a few hundred gigabytes of daily change volume typically runs on six to ten Kubernetes nodes plus object storage, commonly landing in the hundreds of euros per month range on providers like Hetzner, Scaleway, or OVHcloud. The honest extra line item is roughly half an engineer to own it.
Where the money actually leaks:
- ▸Over-partitioned topics that force oversized broker clusters for tiny throughput
- ▸No tiered storage, so weeks of retention sit on expensive block volumes
- ▸Peak-sized Flink clusters running around the clock instead of scaling with the adaptive scheduler
- ▸Small-file explosions that quietly inflate storage, slow every query, and grow the compaction bill
Your numbers will vary with volume and freshness SLOs; treat the checkpoint interval and the retention window as cost dials and revisit them quarterly.
How TuniCyberLabs helps
We design, build, and operate exactly this stack for EU and North African clients: Debezium and Kafka on Strimzi, Flink jobs with tested exactly-once semantics, Iceberg lakehouses on EU object storage, and the residency and erasure evidence your DPO and auditors will ask for. Engagements range from a one-week architecture review to full build-and-operate. If a sovereign streaming lakehouse is on your roadmap, start a conversation through our services.