Networking · 10 min

TCP

Follow one page load from Enter to pixels and see how TCP turns a network that loses things into a connection that never does, and what that costs in round trips.

You type google.com and press Enter. About a quarter of a second later the page starts to draw. In between, about 15 messages cross the internet, and not one of them is guaranteed to arrive.

Yet your browser never shows you half a page or a page with its paragraphs shuffled. This article follows that one page load, message by message, to see who makes that promise and what it costs.

The internet only promises to try

Your request travels as packets, passed from router to router like a parcel between sorting offices. Each router just sends it one step closer. If a router is busy, it throws the packet away. Nobody tells you.

So packets can be lost, arrive twice, or arrive in the wrong order. That is the raw deal the network gives every program.

TCP (Transmission Control Protocol) is code in your operating system that sits on top of this and gives your program a better deal: every byte you send arrives once, complete and in order, or you are told the connection failed. Almost every HTTP request, database query and WebSocket message rides on it.

First, the address: 110 ms

Before TCP can do anything it needs a number to send to. Computers find each other by IP address (like 142.250.180.14), not by name.

So the browser first asks DNS, the internet’s phone book, “what is the IP of google.com?”. On a first visit that is 8 messages between servers, 110 ms in the demo. DNS is its own topic; for this journey, all that matters is that we now have an address and none of the page has moved yet.

Next, the handshake: agreeing how to count

TCP’s whole trick is to number every byte. If bytes are numbered, the receiver can spot a gap, throw away a copy, and put things back in order. But both sides must agree where the counting starts.

In the demo your browser starts at 1000 and the server at 5000. Agreeing on that takes three messages, the three-way handshake:

1. SYN (“synchronise”): browser → server. “I want to talk. My counting starts at 1000.”
2. SYN-ACK: server → browser. “Fine. Mine starts at 5000. The next byte I expect from you is 1001.”
3. ACK (“acknowledge”): browser → server. “Got it. The next byte I expect from you is 5001.”

Notice the ACK numbers: they always say “the next byte I expect”, not “the last one I got”. TCP counts the SYN as if it were one byte, so that the reply has something to acknowledge. It uses up number 1000, and real data starts at 1001.

Then lock it, then ask

The address is https://, so the connection must be encrypted before anything private is sent. That is TLS: the browser says hello and names the site, the server replies with a certificate: a document, vouched for by a company your browser already trusts, proving the server really is google.com. After that, everything is encrypted. In TLS 1.3 this is one more round trip, 40 ms. Older TLS 1.2 needs two.

Only now does the browser send what you actually wanted: GET /, about 80 bytes. Those bytes are numbered 1001 to 1080.

The server answers 200 OK with the HTML. Look at the Ack in that packet in the demo: 1081. That one number says “I have every byte up to 1080, send 1081 next”. The reply and the receipt travel together.

The packet that never arrives

That is the whole reliability recipe. Keep a copy until it is acknowledged. If the acknowledgement does not come in time, send it again.

How long is “in time”? The demo uses 200 ms. Real TCP measures how long acknowledgements usually take on this connection and waits a bit longer than that, never less than 200 ms on Linux. Before it has measured anything, for the SYN, it waits 1 second.

Now the other direction. Say the HTML comes back as three packets of 1,000 bytes: 5001–6000, 6001–7000 and 7001–8000. The middle one is lost.

1. Bytes 5001–6000 arrive. The browser acknowledges “next I expect 6001”.
2. Bytes 7001–8000 arrive. There is a gap, so the browser holds them back and still says “next I expect 6001”.
3. The server sees nothing new acknowledged, times out and resends 6001–7000.
4. Now the gap is filled. The browser hands all 3,000 bytes to the page in order and says “next I expect 8001”.

If the “lost” packet was only late and both copies turn up, the second has numbers already seen, so it is thrown away.

Step 2 has a cost: bytes 7001–8000 arrived fine but had to wait. This is called head-of-line blocking. HTTP/2 sends many files (HTML, CSS, images) over one TCP connection, so one lost packet of an image stalls the CSS too. That is the main reason HTTP/3 runs on QUIC instead of TCP: QUIC numbers each file’s bytes separately, so a gap in one file does not hold up the others.

Now move the server to the other side of the world

In the demo the server is 40 ms away, round trip. A user in Sydney loading a site hosted in Virginia is more like 200 ms away. Run the same journey, leaving DNS aside:

Two thirds of that 600 ms is set-up. Now the common fixes make sense:

• Keep-alive and connection pools reuse an open connection, so you pay the handshake once instead of per request. The last demo step keeps the connection open for exactly this reason.
• CDNs (content delivery networks) keep copies of your site in data centres in many cities, so the user talks to a nearby server and every round trip shrinks at once.
• A pool of database connections in your app server exists for the same reason: a fresh TCP (and often TLS) handshake per query would cost more than many queries do.

The price of keeping connections open is memory: each one holds memory for its unsent and unread bytes on the server and on any load balancer in between. That is the trade you are making every time you size a pool.