Apache Kafka's cloud bill is mostly a replication bill. KIP-1150, known as Diskless Topics, proposes letting Kafka persist topic data straight to object storage such as Amazon S3 instead of local broker disks. Here is what actually changes, what it costs in latency, and how to decide whether and when to adopt it.
What are diskless topics in KIP-1150?
Diskless topics are a proposed Apache Kafka topic type that writes records directly to object storage (Amazon S3, Google Cloud Storage, Azure Blob) instead of replicating them across broker disks. Brokers become leaderless for these topics, cross-availability-zone replication traffic disappears, and durability is delegated to the object store. Existing Kafka clients keep working unchanged.
KIP-1150 was proposed by engineers at Aiven in April 2025 and went through one of the longest design debates in Kafka's history before being accepted, with the implementation split across sub-KIPs covering the storage engine and the batch coordinator. Two design decisions matter most:
- ▸Per-topic opt-in. Diskless is a topic-level configuration, not a cluster mode. Classic low-latency topics and diskless high-throughput topics coexist in the same cluster.
- ▸Protocol compatibility. Producers and consumers use the standard Kafka protocol. No client forks, no new SDK.
Timelines shift, so verify current status and the target release on the Apache Kafka KIP wiki and dev mailing list before planning around it. As of early 2026, early access in a 4.x release was the working expectation.
Why does classic Kafka cost so much in the cloud?
Because replication multiplies network and storage. A three-replica cluster spread across three availability zones pays inter-AZ transfer fees on most produced bytes twice, first producer to leader, then leader to followers, plus three copies on provisioned block storage. For high-throughput clusters, that inter-AZ traffic frequently dominates the Kafka bill.
The unit economics explain the pressure:
- ▸Inter-AZ transfer is typically on the order of 0.01 USD per GB in each direction on AWS, charged on both sides of the link. At tens of MB per second sustained, it compounds into a large monthly line item.
- ▸Block storage is provisioned, not consumed. You size EBS or its equivalents for peak retention across three replicas and pay for the headroom.
- ▸Elasticity is poor. Scaling brokers in or out moves partition data over the network, so teams over-provision to avoid rebalancing during peaks.
Object storage inverts all three: roughly 0.023 USD per GB-month on S3 Standard, durability engineered to eleven nines with replication included, and capacity billed on actual use. If you have not measured which line dominates your own bill, start there. The attribution approach in Kubernetes FinOps: From Cluster Bill to Unit Economics applies directly to streaming infrastructure.
How do diskless topics actually work?
Any broker can accept produce requests for a diskless topic. The broker buffers batches from many partitions in memory, flushes them as combined objects to the object store, and only then acknowledges the producer. A batch coordinator assigns offsets afterwards and records which object contains which batch, so ordering is preserved without a partition leader.
The mechanics worth understanding:
- ▸Leaderless writes. With no partition leader for diskless topics, a producer in zone A talks to a broker in zone A. The cross-zone hop, and its fee, disappears.
- ▸Shared write-ahead objects. Batches from different partitions and topics are packed into one object per flush to amortize PUT request costs; background compaction later reorganizes data into larger per-partition segments for efficient reads.
- ▸The batch coordinator is the new critical path. It sequences batches into offsets and stores batch-to-object mappings on top of Kafka's KRaft consensus layer (Kafka 4.0 removed ZooKeeper entirely). Coordinator scalability and metadata growth were the most debated points of the design.
- ▸Consumers still fetch through brokers, which read objects and serve them from cache, so consumption looks familiar even though the storage path is new.
The architecture will look familiar to anyone who has studied WarpStream, AutoMQ or Bufstream. The significance of KIP-1150 is bringing that pattern into upstream Apache Kafka.
What happens to latency when Kafka writes to S3?
Produce latency rises from single-digit or low-tens-of-milliseconds to hundreds of milliseconds. Object storage PUTs typically complete in roughly 50-200 ms and brokers buffer before flushing, so design discussions target p99 produce acknowledgements in the 200-500 ms range and end-to-end latency under about one second.
Treat those numbers as community estimates until stable releases ship public benchmarks. The latency floor is structural:
- ▸Flush interval is a cost dial. Flushing more often lowers latency but multiplies PUT requests, which are billed per call. Most deployments will tune toward fewer, larger objects.
- ▸S3 Express One Zone offers much faster PUTs and can pull latency down significantly, but it is single-zone and pricier per GB, which reintroduces some of the trade-offs diskless topics were meant to remove.
- ▸End-to-end lag matters more than produce latency for most pipelines; budget for roughly a second before data is readable by consumers.
That rules out fraud scoring, trading and interactive request-response patterns. It is a non-issue for log and telemetry ingestion, clickstreams, and change-data-capture feeds like those described in Change Data Capture in Practice: Debezium, Kafka, and the Outbox Pattern.
Are the 80 percent cost-cut claims real?
Treat them as community estimates, not guarantees. The widely cited figure of up to 80 percent total-cost reduction comes from the KIP's authors and from vendors of similar architectures, and it models high-throughput, multi-AZ, cloud-hosted clusters where inter-AZ replication dominates. Actual savings depend on throughput, retention, request patterns and current over-provisioning.
Where the savings are real:
- ▸Inter-AZ replication traffic goes to zero for diskless topics, usually the largest line item.
- ▸Three provisioned block-storage copies become one consumed object-storage copy.
- ▸Brokers become nearly stateless for diskless topics, so autoscaling no longer moves data and clusters can run closer to actual load.
Where the estimate shrinks:
- ▸Single-AZ or small clusters already avoid most transfer fees.
- ▸PUT and GET request charges grow if you tune for low latency with small, frequent flushes.
- ▸Latency-critical topics stay classic, so mixed clusters keep part of the old cost base.
- ▸On-premises Kafka gains little on cost; the arithmetic that matters there is closer to Cloud Repatriation: When Bare Metal Wins the Math.
Model it with your own numbers: sustained MB per second produced, replication factor, retention days, AZ layout, and your object storage request pricing.
How does KIP-1150 compare to tiered storage and WarpStream-style systems?
Tiered storage (KIP-405, production-ready since Kafka 3.9) offloads only closed historical segments; fresh writes still replicate across broker disks, so it cuts storage cost but not inter-AZ traffic. WarpStream, AutoMQ and Bufstream rebuilt Kafka-compatible systems around object storage first but are single-vendor products. KIP-1150 brings that architecture into Apache Kafka itself.
Practical implications:
- ▸Tiered storage remains complementary. It ships today and helps long retention on classic topics; diskless targets the write path itself.
- ▸Proprietary object-storage-first systems are more mature. WarpStream (acquired by Confluent in 2024), AutoMQ and Bufstream have production miles that upstream diskless topics will need time to accumulate.
- ▸Upstream matters for lock-in. A KIP eventually lands in every distribution and managed service, keeps one wire protocol, and is maintained by the community rather than one vendor's roadmap.
If your motivation is cost today, tiered storage plus honest capacity planning is available now; diskless is the structural fix arriving behind it.
When should you adopt diskless topics, and when should you wait?
Adopt where latency budgets exceed roughly one second and throughput is high: log and telemetry ingestion, CDC into a lakehouse, clickstream collection, ML feature pipelines. Keep classic topics for latency-sensitive paths. Wait for at least one stable release cycle and public benchmarks before moving regulated or business-critical pipelines.
A sensible sequence:
- ▸Classify every topic by latency SLO. Most estates discover that the majority of their bytes tolerate seconds of delay.
- ▸Migrate per topic, not per cluster. The per-topic design means the telemetry firehose can move first while payment events stay classic.
- ▸Re-verify consumer assumptions. Higher and more variable lag exposes fragile consumers; the practices in Idempotent Data Pipelines: Batch Jobs That Survive Reruns become mandatory rather than nice-to-have.
- ▸Rework the DR story. Durability now rides on bucket replication policies and object-store availability, and EU teams should pin buckets to compliant regions before anything else moves.
How do you prepare a Kafka estate for diskless topics today?
Measure first: attribute inter-AZ traffic, broker storage and over-provisioning to Kafka specifically, and classify topics by latency tolerance. Upgrade to Kafka 4.x on KRaft, adopt tiered storage where it already pays for itself, and keep clients on standard APIs so a later switch is configuration, not a rewrite.
- ▸Baseline the bill with per-workload attribution, following the method in How to Cut Cloud Costs by 40% Without Downtime: A FinOps Playbook.
- ▸Stay current on 4.x and KRaft so the upgrade path to an early-access release is short.
- ▸Avoid client-side forks and vendor-specific protocol extensions that would complicate moving between classic topics, diskless topics and managed alternatives.
- ▸Track KIP-1150 and its sub-KIPs on the Kafka dev list rather than relying on secondhand summaries; scope has shifted during the debate and may shift again.
How TuniCyberLabs helps
TuniCyberLabs designs and operates streaming and data platforms for teams across the EU and North Africa: Kafka capacity and cost audits, CDC pipelines, lakehouse ingestion, and migration planning with EU data residency built in. If your Kafka bill is growing faster than your traffic, we will baseline it and give you a defensible adoption plan. Talk to an engineer.