Networking · 15 min

Load balancing

One address in front of many servers. Follow one shopper’s request on sale day through DNS, the balancer’s choice of server, a crash, and back.

It is 10:00 on sale day. A shopper taps Add to cart on your shop, shop.example. Behind that one name are 3 app servers: app-1, app-2 and app-3.

The browser only knows the name. Something has to turn it into one of those 3 machines, avoid the one that is swamped, and stop picking the one that just crashed. Here is that one tap, there and back.

Step 1: the name becomes an address

The browser first asks DNS, the internet’s phone book, for the IP address of shop.example. An IP address is the number that finds one machine on the network, such as 198.51.100.20. The browser and the operating system then remember the answer for as long as DNS says, often a few minutes.

The address DNS gives back is not app-1’s. It belongs to the load balancer.

Why not put the 3 app servers’ addresses straight into DNS? Because of that remembering. If app-2 dies, browsers keep using its address until their saved answer runs out, which can take minutes. The balancer’s address never changes, so servers can come and go behind it and no shopper needs a new DNS answer.

The app servers themselves sit on private addresses, such as 10.0.0.12, that the internet cannot reach. Only the balancer can.

Step 2: what the balancer can see

The browser opens a TCP connection to the balancer: a two-way channel between two programs. Each end has an IP address and a port number. The address finds the machine; the port picks the program on it. HTTPS servers listen on port 443. What happens next depends on how much of the request the balancer reads.

A layer 4 (L4) balancer reads only the addresses: the shopper’s IP address and port, and its own. It picks a server when the connection opens, then copies bytes both ways. It never sees the URL or the cookies; with HTTPS they are encrypted anyway.

A layer 7 (L7) balancer reads the request itself. The certificate and private key are the files that let a server prove it is shop.example and unlock what browsers encrypt for it. Put them on the balancer instead of the app servers, and the browser’s encrypted connection ends at the balancer. It decrypts and sees the method, path, headers and cookies, such as POST /cart. Then it sends the request on to an app server over a connection of its own, often plain HTTP inside your private network.

	L4 balancer	L7 balancer
Sees	IP addresses and ports	Also the URL, headers and cookies
Picks a server	Once per connection	For every request
Can send /checkout to different servers	No	Yes
Work per request	Very little: it copies bytes	Decrypts and reads every request
Examples	AWS Network Load Balancer	AWS Application Load Balancer, nginx

4 and 7 are layer numbers from the OSI model, the standard 7-layer picture of networking. Layer 4 is TCP; layer 7 is the application’s own protocol, here HTTP.

“Once per connection” matters more than it looks. A browser keeps one connection open and sends many requests down it, so an L4 balancer sends all of them to the same server. The rest of this article, and the demo, use an L7 balancer that picks a server for every request.

Step 3: picking a server

Each app server runs 3 workers (threads or processes that each handle one request at a time), so it works on 3 requests at once. A product page takes 50 ms of work, so one worker does 1,000 ÷ 50 = 20 pages a second and one server does 3 × 20 = 60. A 4th request at the same moment waits in line for a free worker.

The plainest rule is round robin: app-1, then app-2, then app-3, then app-1 again. It needs one counter and knows nothing about the servers. If every request costs the same, that is perfect: each server gets a third.

They don’t cost the same. A checkout talks to the payment company and takes 500 ms, 10 times as long as a page.

Guess first: Shoppers send 30 requests a second, and every 3rd one is a checkout (500 ms); the rest are pages (50 ms). Round robin, 3 servers, 3 workers each. Where do the checkouts go?

All to the same server. Say request 1 goes to app-1. Requests 3, 6, 9 and so on are the checkouts, and round robin sends request 3, 6, 9 and so on to app-3 every time. That is 10 checkouts a second, but 3 workers at 500 ms each finish only 6 a second. So app-3’s line grows by 4 requests every second and its shoppers wait longer and longer. Meanwhile app-1 and app-2 each get 10 pages a second: 10 × 50 ms = half a second of work, shared by 3 workers, so each worker is busy 0.5 ÷ 3 = 1/6 of the time. Nearly idle.

Real traffic is not that regular, but the lesson holds: round robin counts requests, not work. A server stuck with a few slow requests keeps getting its full share of new ones.

Least connections uses that count. It sends each new request to the server with the fewest in flight. A server busy with checkouts has a longer line, so new requests go elsewhere until it catches up. The balancer never needs to know what a request costs.

Random picks any server that is up. Over many requests it evens out like round robin, and like round robin it ignores load. It is handy when several balancers share the same servers and cannot share a counter. nginx offers a middle way, random two: pick two servers at random and send to the one with fewer in flight. The busier of the pair never gets the request, so the busiest server of all never does, and no shared counter is needed.

Run sale day: pick a rule, then break a server

With Round robin selected, press Steady stream (30/s). In “Where each request went” (newest at the top), requests go to app-1, app-2, app-3 in turn, and p50 and p95 both sit at 50 ms.
Tick “Every 3rd request in the stream is a checkout”. One server’s worker boxes turn yellow (checkouts) and the red bar of waiting requests under them grows; within about 3 seconds p95 passes 1,000 ms. Press Least connections within a few seconds (after about 10 the line is full and new requests get a 503, counted as Failed). New requests now avoid that server, but p95 keeps climbing while the requests already stuck in its line finish late; the later you switch, the longer that takes. Then it drops to about 500 ms, the time of a checkout itself.
Untick the checkbox, press Round robin, then press Kill on app-2. Anything it was working on fails at once. Its border turns red, but it stays Up while its checks fail (✗), and every 3rd request fails. After 3 failed checks it shows Down and gets nothing. Press Revive: it needs 2 passing checks before requests come back.
Press Reset, then Steady stream, then Make slow on app-1, so every request there takes 4 times as long. Its checks keep passing, round robin still gives it a third of the requests, and p95 rises to 200 ms while p50 stays at 50. Press Reset again, press Least connections, and repeat: after 10 seconds app-1’s Served is about a third of each other server’s.

Step 4: the server it picked has crashed

Say the balancer picks app-2, and app-2 crashed a second ago. The connection is refused, or nothing answers at all. How does the balancer learn to stop picking it?

Why several and not one? One lost check can be a blip on the network. Taking a healthy server out pushes its third of the traffic onto the other two, so balancers wait for a run of results to be sure.

The demo checks every second: 3 failures mark a server down, 2 passes mark it up. Real defaults are slower. HAProxy checks every 2 seconds; AWS’s Application Load Balancer every 30.

Guess first: app-2 crashes just after a check passed. Checks run every second, 3 failures mark it down, and the shop gets 30 requests a second under round robin. About how many shoppers see an error?

About 30. It takes 3 seconds of failed checks to mark app-2 down. In those 3 seconds 90 requests arrive, and round robin sends a third of them, 30, to app-2. AWS’s balancer marks a server down after 2 failed checks, 30 seconds apart: up to a minute of every 3rd shopper getting an error.

That gap is why balancers also watch real requests. If requests to a server fail, stop sending to it for a while, without waiting for the timer. This is a passive check. nginx does it by default: 1 failed request takes a server out for 10 seconds.

Many balancers also retry a failed request on another server. nginx retries a failed GET but not a POST, because sending a payment twice could charge the shopper twice. The demo does neither, so you can see the gap.

A slow server passes every check

Now app-1 is not dead, just slow: its disk is failing, or another program on the same machine is hogging the CPU. It still answers GET /health in a millisecond, so the health checks say it is fine.

Round robin keeps sending it a third of the shoppers, and each of them waits 4 times as long. Least connections notices without being told: slow requests stay in flight longer, app-1’s count stays higher, so it gets fewer new ones. Most requests still go to healthy servers, so p50 barely moves. p95 shows it.

Sticky sessions: same shopper, same server

Suppose the shop keeps each cart in the app server’s memory. Then the shopper’s next request must go back to the same server, or the cart looks empty.

An L4 balancer can only do this with the shopper’s IP address, the one thing it sees. IP hash runs the address through a hash function, which turns any text into a big number and always gives the same number for the same text. It divides that by the number of servers that are up and uses the remainder, 0, 1 or 2, to pick one. The demo’s IP hash shows this sum for each request, writing the remainder as “mod”. You will meet it because it is the only sticky rule an L4 balancer has.

This has two problems. A whole office, or many phones on one mobile network, can share one public IP address, so thousands of shoppers stick to the same server. And when app-2 goes down, the sum divides by 2 instead of 3, so most shoppers land on a different server and lose their carts anyway. Take hashes 0 to 5. Dividing by 3 sends them to app-1, app-2, app-3, app-1, app-2, app-3. Dividing by 2, with app-2 gone, sends them to app-1, app-3, app-1, app-3, app-1, app-3. Only 0 and 5 keep their server: 4 shoppers in 6 move, though only 2 were on app-2. The Consistent hashing article on this site shows how to move far fewer of them.

An L7 balancer can stick by cookie instead: it sets a cookie naming the server on the first reply and follows it after. AWS’s Application Load Balancer calls this stickiness. Better still is not needing it: keep carts in a shared store such as Redis or the database, and any server can serve any shopper. Then losing a server loses no carts.

Step 5: the trip back

app-1 builds the page in 50 ms and hands it to the balancer, which sends it down the shopper’s connection. The balancer itself usually adds well under a millisecond. The shopper never learns which server answered.

One catch: every request now passes through one machine. If the balancer dies, the whole shop is down. So balancers run in pairs: a standby watches the active one. Both are set up to answer for the shop’s IP address, but only one does at a time. If the active one stops answering, the standby starts answering for that address and traffic flows to it instead. Managed balancers such as AWS’s spread themselves over several machines for you.

How many servers for sale day?

Put it together, and count work, not requests. Each server has 3 workers, so it has 3 seconds of work to give every second; three servers have 9. Last sale day peaked at 60 requests a second, 1 in 10 a checkout. That is 54 pages × 0.05 s + 6 checkouts × 0.5 s = 2.7 + 3 = 5.7 seconds of work a second. 5.7 ÷ 9 = 63% busy. Fine.

Now lose one, as you did in the demo. Two servers have 6 seconds of work to give for 5.7 arriving: 95% busy. Requests never arrive perfectly evenly; a second with a few extra checkouts needs more work than the servers have, and at 95% there is almost no slack afterwards to clear the line it leaves. And for the 3 seconds before the health checks notice, a third of the requests fail.

So plan for one server down: the ones left must carry the peak with room to spare. Four servers lose one and are back at 63%. Then use least connections so a slow or checkout-heavy server sheds load, health checks plus passive checks so a dead one leaves fast, and carts in a shared store so it doesn’t matter which server answers.

Remember

DNS points at the balancer; servers can come and go behind it without anyone noticing.
L4 sees addresses and ports and picks once per connection; L7 reads the HTTP request and picks per request.
Round robin counts requests, not work; least connections follows how busy each server really is.
Health checks take seconds to notice a crash and never notice a slow server; watch failures and p95 too.
Keep sessions in a shared store rather than sticky servers, and size so you survive one server down.

See it in a design

Flash sale: Limited drop that sells from an atomic Redis counter and writes orders through a queue.
Multi-region active-active: Two live regions behind geo DNS, each self-sufficient, replicating writes to each other asynchronously.
Rate limiter: Token buckets in Redis checked at the gateway, so no single client can overwhelm the API.