The defining property of a production-grade batch pipeline is not speed or cleverness — it is that you can run any task twice and get the same result. Idempotency is what turns a 02:00 page into a morning coffee decision: if reruns are safe, almost every pipeline failure becomes a retry instead of a forensic investigation into duplicated rows and double-counted revenue.
Most pipeline pain — silent duplicates, drifted aggregates, backfills that eat a weekend — traces back to tasks that mutate state in ways that depend on when and how often they run. The fix is architectural, not heroic, and it rests on three habits: partition everything, write with replace semantics, and let the orchestrator own state.
Partitioning Is the Contract
Every batch task should process one explicit, bounded slice of data — usually a time window — passed in as a parameter, never derived from the wall clock inside the task. The moment a job computes its own date range from now, reruns become unreproducible.
- ▸Explicit intervals: Airflow's data interval boundaries, or Dagster's partition keys, define exactly which slice a run owns. The task reads the parameter; it never calls a now() function for scoping.
- ▸Aligned storage: land output in a structure that mirrors the partitions — daily Postgres partitions, Parquet files keyed by date in object storage, an Iceberg partition spec. One run, one partition, one owner.
- ▸A late-data policy: decide upfront how long a partition stays open for stragglers, then reprocess whole partitions after that window rather than patching individual rows.
Write with Replace Semantics
Blind INSERT is how duplicates are born. Every write path needs an answer to the question: what happens if this task already ran for this slice?
The cleanest pattern is delete-then-insert in one transaction — remove every row for the target partition, insert the fresh set, commit. Postgres makes this cheap when storage partitions align with pipeline partitions: truncate or drop-and-recreate the partition, then reload. For row-level work, INSERT with ON CONFLICT DO UPDATE holds up well at moderate volume, and MERGE (Postgres 15 and later) expresses richer logic; both require a real uniqueness key, which forces a healthy discussion about what actually identifies a record. In lakehouse tables, overwrite the partition atomically — Iceberg and Delta both support replace-where writes.
Non-idempotent side effects deserve special suspicion. Sending an email, calling a payment API, or appending to a Kafka topic inside a batch task will happen again on retry. Isolate side effects behind an outbox or a deduplication key, or move them out of the batch layer entirely.
Let the Orchestrator Own State
A pipeline where tasks consult their own bookkeeping tables to decide what to do is a pipeline nobody can reason about. Declare dependencies and freshness in the orchestrator and keep tasks simple.
Dagster's asset model is the strongest version of this idea: you declare that a table is a function of upstream assets over a partition, and the framework computes what is stale and what needs rebuilding. Airflow gets close with datasets and dynamically mapped tasks. Either way, retries with exponential backoff belong in orchestrator configuration, task timeouts should be explicit, and a task that cannot be retried safely should be flagged loudly in code review — it is a design bug, not a fact of life.
Data contracts close the loop upstream: schema and quality checks — dbt tests, Great Expectations, or plain SQL assertions — run as gating tasks so a producer changing a column type fails fast at the boundary, not three joins later inside an executive dashboard.
The Backfill Checklist
Backfills are where idempotency pays rent. Before reprocessing a year of partitions:
1. Confirm every task in the chain is rerun-safe; one appending task poisons the whole lineage. 2. Pin the code version — backfill with the logic you intend, not whatever is on main mid-refactor. 3. Cap concurrency so the backfill does not starve daily runs or overwhelm the source database. 4. Disable external side effects — emails, webhooks, notifications — explicitly for backfill runs. 5. Validate one sample partition end to end before launching the other three hundred. 6. Warn downstream consumers: a rebuilt table can invalidate caches and BI extracts that assumed immutability.
Idempotency sounds like an engineering nicety, but its business value is direct. Finance can trust that a rerun did not double revenue, on-call engineers resolve failures in minutes without escalation, and historical corrections become routine backfills instead of quarterly crises. Teams that build this discipline early spend their hours shipping new data products; teams that skip it spend the same hours reconciling numbers that should never have diverged — and the gap between those two teams widens every quarter.