Every PostgreSQL cluster is highly available until its first real failover. High availability in Postgres is not a product you install; it is a chain of decisions about replication mode, consensus, connection routing, and rehearsal — and each link has failure modes that only appear under pressure. The teams that survive a primary crash at 03:00 are the ones who made those choices deliberately and tested them on a quiet Tuesday afternoon.
This is the stack we run and recommend: physical streaming replication, Patroni coordinating leadership through etcd, and PgBouncer in front of everything. None of it is exotic. All of it fails in specific, predictable ways when the details are skipped, so this post focuses on the details.
Replication: Asynchronous by Default, Synchronous by Exception
Physical streaming replication ships WAL to standbys and is the backbone of every serious Postgres HA setup. The first decision is durability versus latency, and it deserves a real conversation with the business rather than a default.
- ▸Asynchronous replication: the primary acknowledges commits before standbys confirm receipt. Fast, but a crash can lose the last few hundred milliseconds of committed transactions. Acceptable for most workloads — as long as the loss window is documented, not discovered.
- ▸Synchronous commit: with a synchronous standby named, commits wait for the standby to confirm. Zero data loss on failover, but every write now pays a network round trip, and a dead standby stalls the primary unless you plan for it.
- ▸Quorum sync: ANY 1 (s1, s2) in synchronous_standby_names gives synchronous durability that tolerates one standby outage. This is usually the right compromise for money-moving tables.
Watch replication slots. A disconnected standby with an abandoned slot will retain WAL until the primary's disk fills. Set max_slot_wal_keep_size so a forgotten slot degrades one replica instead of taking down production.
Patroni and the Consensus Layer
Patroni does not detect failures by pinging Postgres; it manages a leader lease in a distributed configuration store — etcd, Consul, or ZooKeeper. The primary must renew its key before the TTL expires, or a standby wins the election and promotes. This design pushes the hard consensus problem onto a system built for it, which is exactly where it belongs.
Two operational consequences follow. First, your etcd cluster is now part of the database's availability math: run three nodes, monitor its latency, and know what Patroni does when the store is unreachable — it demotes the primary to prevent split-brain, which is surprising the first time you see it. Second, fencing matters: enable the Linux watchdog so a paused or partitioned primary cannot wake up and keep accepting writes on a stale timeline. After failover, reattach the old primary with pg_rewind rather than rebuilding from scratch; it repairs the diverged timeline in minutes instead of the hours a fresh base backup costs.
Connections: The Layer Everyone Forgets
A perfect thirty-second failover still looks like a ten-minute outage if clients cache dead connections. Route traffic through something that asks Patroni who the leader is: HAProxy health checks against the Patroni REST API — the primary endpoint returns 200 only on the leader — or a small VIP manager watching the DCS key directly.
PgBouncer in transaction pooling mode keeps connection counts sane, since Postgres suffers beyond a few hundred backends, but read the fine print: session state such as SET commands and advisory locks does not survive transaction pooling, and protocol-level prepared statements only work properly on PgBouncer 1.21 and later. On the application side, treat failover as a first-class event — retry idempotent statements on connection errors, and surface non-idempotent failures to a human decision rather than a blind retry loop.
The Drill: A Failover You Rehearse Quarterly
Runbooks decay. Rehearse the whole path end to end, in staging first and then in production during a low-traffic window:
1. Record the topology: current leader, standby lag, slot positions, archive status. 2. Kill the primary ungracefully — stop the VM or kill -9 the postmaster. A clean shutdown tests nothing. 3. Measure two numbers: time to leader election and time to first successful application write. 4. Verify WAL archiving and backups resumed from the new primary; pgBackRest or wal-g must follow the timeline switch. 5. Reattach the old node with pg_rewind and confirm it streams from the new leader. 6. Pull the application error rate for the window and check which requests retried cleanly. 7. Fix every surprise in the runbook the same week, while it still hurts.
The first drill always finds something: a hardcoded hostname, a health check wired to the old primary, an alert that never fired.
The business case for this work is straightforward. A rehearsed failover costs a few engineer-days per quarter; an unrehearsed one is measured in hours of downtime, lost transactions, and eroded customer trust. Treat Postgres high availability as an engineering practice rather than a checkbox, and database failures become routine events your team resolves before customers notice — and that reliability is precisely what lets you ship faster everywhere else in the stack.