Round robin over a handful of upstreams is where every load balancing story begins, and for plenty of internal services it is genuinely enough. But load balancing does more work in a modern stack than any other single component: it is your deploy mechanism, your failure domain boundary, your TLS front door, and often the only thing standing between a bad health check and a full outage. Treating it as a solved commodity is how teams end up learning about connection draining during an incident review.
The decisions that matter are not which vendor to pick but where in the stack to balance, how honestly your health checks reflect reality, and what happens in the thirty seconds after something fails.
L4 and L7 are different tools, not tiers of quality
- ▸Layer 4 (connection-level): IPVS, XDP-based balancers in the style of Katran, or cloud NLBs forward packets with minimal inspection. You get enormous throughput, low latency, and source-IP preservation with direct server return, but no retries, no per-route decisions, no visibility into HTTP semantics.
- ▸Layer 7 (request-level): HAProxy, Envoy, and nginx terminate the protocol and can route per path, retry idempotent requests, shed load selectively, and emit per-endpoint metrics. The price is CPU for TLS and parsing, plus a component that must scale with request complexity rather than raw bandwidth.
- ▸The durable pattern: an L4 tier spreading traffic across a horizontally scaled L7 tier. Google's Maglev paper described the shape a decade ago and it remains the reference, consistent hashing at L4 so connections survive balancer restarts, intelligence at L7 where the protocol lives.
- ▸Client IP preservation: once you stack tiers, the original address travels via the PROXY protocol or X-Forwarded-For. Decide at design time which hops append and which are trusted, or your rate limiter will happily throttle your own balancer.
Health checks that tell the truth
Most load balancer failures are health check failures wearing a disguise. An endpoint returning 200 from a process that cannot reach its database is lying; a deep check that exercises the database can fail the whole fleet simultaneously when the database blips, converting a partial degradation into a total outage.
The practice that holds up: shallow active checks for liveness (is the process accepting connections), passive outlier detection for quality (eject an upstream whose real traffic shows elevated errors or latency, as Envoy's outlier ejection does), and a maximum ejection percentage so the balancer can never empty the entire pool. Add slow-start for recovering instances, dumping a full connection share onto a cold JVM or an empty cache recreates the failure you just healed.
Deploys and drains: the seconds that decide your error rate
Every deploy is a load balancing event. Rolling restarts without connection draining show up as a small, steady error rate that teams learn to ignore, and that disappears the week someone configures graceful drain properly. The sequence matters: mark the instance unhealthy so new connections stop, wait out in-flight requests up to a deadline, close long-lived connections politely (GOAWAY frames for HTTP/2 and gRPC, close frames for WebSockets), then terminate. On Kubernetes this means preStop hooks aligned with the balancer's drain interval and terminationGracePeriodSeconds, misalign them and you drop requests on every rollout despite green dashboards.
Going global: anycast and honest failover
When one site is not enough, two mechanisms carry traffic between regions. GeoDNS is simple and works everywhere but is hostage to resolver behavior, some resolvers cache well past your 60-second TTL, so failover takes minutes, not seconds. Anycast, announcing the same prefix from multiple sites via BGP, converges in seconds and needs no client cooperation, at the cost of running your own address space, an ASN, and transit relationships. TCP sessions rerouting mid-connection across anycast paths are rare in practice but real; long-lived connections deserve a per-region unicast fallback. Most mature setups use both: anycast at the edge, GeoDNS as coarse steering, and failover rehearsed quarterly rather than discovered live.
A review checklist for your current setup
1. Map which tier terminates TLS, which sees real client IPs, and which components trust forwarded headers, in writing. 2. Verify passive outlier detection is enabled with a capped ejection percentage, and that shallow checks actually fail when the process dies. 3. Time a full drain during a normal deploy and confirm zero connection resets from the client side. 4. Confirm slow-start or gradual weight ramp for instances rejoining the pool. 5. Kill one balancer instance in staging and measure connection survival, consistent hashing either works or it does not. 6. Rehearse regional failover end to end and record the actual, not theoretical, recovery time.
A load balancer that drains cleanly, ejects honestly, and fails over in seconds converts infrastructure failures from customer-visible incidents into internal log lines. The difference compounds: teams deploy more often because deploys are silent, and postmortems get shorter because failure domains are real boundaries. The balancing tier is one of the few places where a week of focused engineering permanently upgrades every service behind it at once, few investments in the stack pay back faster.