Web Servers & Reverse Proxies

Load Balancing with Nginx

18 min Lesson 6 of 28

Load Balancing with Nginx

When a single application server can no longer keep up with traffic — or when you need zero-downtime deploys and fault tolerance — you add more servers and put a load balancer in front of them. Nginx does this elegantly in the same process that terminates TLS and serves your static assets. Understanding the upstream algorithms, health-check mechanics, and session-affinity options is the difference between a load balancer that works in staging and one that holds up under Black-Friday traffic.

The upstream Block

Everything starts with an upstream block. You name the pool, list the servers, choose an algorithm, and then proxy_pass to that pool name from any location block.

http {

    upstream app_pool {
        # Default algorithm: round-robin (no directive needed)
        server 10.0.1.10:8080;
        server 10.0.1.11:8080;
        server 10.0.1.12:8080;
    }

    server {
        listen 80;
        server_name example.com;

        location / {
            proxy_pass         http://app_pool;
            proxy_http_version 1.1;
            proxy_set_header   Host              $host;
            proxy_set_header   X-Real-IP         $remote_addr;
            proxy_set_header   X-Forwarded-For   $proxy_add_x_forwarded_for;
            proxy_set_header   X-Forwarded-Proto $scheme;
            proxy_set_header   Connection        "";   # Keep-alive to upstreams
        }
    }
}

Always set proxy_http_version 1.1 and clear Connection. HTTP/1.0 (the default) closes the TCP connection after every request. HTTP/1.1 with an empty Connection header tells Nginx to keep the connection to the upstream alive, reusing sockets for subsequent requests — a significant throughput gain at scale.

Upstream Balancing Algorithms

Nginx open-source ships with four algorithms. Nginx Plus adds more (least_time, random with two, zone-aware). Choose based on your workload profile:

round-robin (default) — Each new request goes to the next server in the list, cycling through equally. Works well when requests are homogeneous in cost and servers are identical. Add the weight parameter to skew distribution toward more powerful nodes.
least_conn — New request goes to the server with the fewest active connections. Correct choice for long-lived connections (WebSockets, streaming, slow database queries) where round-robin would pile up on one server while others sit idle.
ip_hash — Hashes the client IP (first three octets for IPv4) and always routes that client to the same upstream. A basic form of sticky sessions — no extra cookies. Breaks when clients are behind a shared NAT or a CDN (all traffic hashes to the same origin IP).
hash $variable [consistent] — Hash on any Nginx variable: URI, cookie value, request header. With consistent it uses a consistent-hash ring (Ketama), so adding or removing a server only remaps a fraction of keys — useful for proxying to upstream caches.

upstream api_pool {
    least_conn;

    server 10.0.1.10:3000 weight=3;   # 3x traffic share
    server 10.0.1.11:3000 weight=1;
    server 10.0.1.12:3000 backup;     # Only used when primaries are down
}

upstream cache_pool {
    hash $request_uri consistent;     # Same URI always hits same Varnish node

    server 10.0.2.10:6081;
    server 10.0.2.11:6081;
}

Passive Health Checks

Nginx open-source performs passive health checks: it watches live traffic and marks a server as unhealthy only after it fails real requests. The key parameters live in the upstream server directive:

max_fails=3 — how many consecutive failures before the server is considered down (default 1).
fail_timeout=30s — how long to stop sending requests once the threshold is reached, and also the window in which max_fails are counted (default 10s).

upstream app_pool {
    server 10.0.1.10:8080 max_fails=3 fail_timeout=30s;
    server 10.0.1.11:8080 max_fails=3 fail_timeout=30s;
    server 10.0.1.12:8080 max_fails=3 fail_timeout=30s;
}

# Complement passive checks by telling Nginx which errors count as failures:
proxy_next_upstream error timeout http_502 http_503 http_504;
proxy_next_upstream_tries 2;   # Retry at most once on a different upstream

Production default: proxy_next_upstream. Without it, a 502 from a crashing backend pod is returned directly to the user. With it, Nginx transparently retries on another server. Limit retries to non-mutating requests or be careful: retrying a POST that already committed to the database will duplicate the write.

Active Health Checks (Nginx Plus / OpenResty / Upstream Check Module)

Passive checks only detect failures on live traffic. Active checks probe upstreams on a background interval, so a server is removed from rotation before a user hits it. In open-source Nginx you achieve this with the ngx_http_upstream_check_module (compiled in) or by fronting with a tool like Consul + consul-template that rewrites the upstream block. Nginx Plus has it natively via the health_check directive:

# Nginx Plus syntax (reference — Nginx OSS needs a third-party module)
upstream app_pool {
    zone app_pool 64k;        # Shared memory zone required for active checks

    server 10.0.1.10:8080;
    server 10.0.1.11:8080;
    server 10.0.1.12:8080;
}

server {
    location / {
        proxy_pass http://app_pool;
        health_check interval=5s fails=2 passes=3 uri=/healthz;
    }
}

Sticky Sessions

Stateless applications — where any server can handle any request — are always preferred in cloud-native design. But legacy apps often store session data in process memory, making it mandatory that a given client always hits the same server. The diagram below shows both models:

Left: sticky sessions pin clients to specific servers — a dead server loses sessions. Right: stateless design stores sessions externally so any server can serve any client.

Nginx Plus provides a sticky cookie directive. In open-source Nginx you use ip_hash or a hash on a cookie value extracted with $cookie_sessionid:

upstream app_sticky {
    # Option 1: ip_hash — simple, no extra cookies, breaks behind CDN/NAT
    ip_hash;

    server 10.0.1.10:8080;
    server 10.0.1.11:8080;
    server 10.0.1.12:8080;
}

upstream app_cookie_sticky {
    # Option 2: hash on the app session cookie — more precise than ip_hash
    hash $cookie_PHPSESSID consistent;

    server 10.0.1.10:8080;
    server 10.0.1.11:8080;
    server 10.0.1.12:8080;
}

# Nginx Plus native sticky cookie (no OSS equivalent):
# sticky cookie srv_id expires=1h domain=.example.com path=/;

Sticky sessions hide a scalability time bomb. If a server goes down, every client pinned to it gets a session error regardless of the other healthy servers. At big-tech scale, the right fix is always to externalize session state to Redis or a database cluster. Sticky sessions are a migration tool — use them to unblock a launch, then plan to remove them.

Upstream Keepalive and Connection Pooling

For high-throughput services, TCP connection setup cost adds up. The keepalive directive in the upstream block tells Nginx to cache a pool of idle connections to each upstream, reusing them across requests. This is distinct from client-facing keepalive and dramatically reduces latency on services doing thousands of requests per second.

upstream app_pool {
    least_conn;

    server 10.0.1.10:8080;
    server 10.0.1.11:8080;
    server 10.0.1.12:8080;

    keepalive 32;            # Cache up to 32 idle connections per worker
    keepalive_timeout 60s;   # Close idle connections after 60s
    keepalive_requests 1000; # Recycle connection after 1000 requests
}

Observing Load Balancer Behavior

The Nginx stub_status module exposes a minimal status page. For richer upstream-level metrics — active connections per server, health state, requests routed — you need Nginx Plus or a third-party module like nginx-module-vts. In production, most teams ship Nginx logs to a metrics pipeline (Prometheus + nginx-prometheus-exporter, or Datadog) and alert on upstream 5xx rates and response time percentiles rather than polling a status page.

Add $upstream_addr to your access log. It records which backend server handled each request, making it trivial to confirm distribution is working and to correlate errors with a specific upstream during an incident. Add $upstream_response_time alongside it to spot latency outliers per server.