Distributed systems · 13 min

Replication

A copy of your database is not a live mirror. Copies run behind, so you can read old data, and a crash can lose a save you were told was safe.

Sakura opens her profile and changes her display name from “Sakura” to “Sakura Ito”. She presses Save. The app says “Saved” and reloads the page.

The page says “Sakura”. She refreshes again, and now it says “Sakura Ito”. Nothing crashed and no code has a bug. To see why, you first have to let go of a picture almost everyone carries.

The wrong picture: every copy is the same copy

This is the wrong picture: “A replica is an exact live copy, so reading from any of them is the same as reading from the original.”

It feels true because on your laptop, or on a quiet test system, the copies are a few milliseconds behind and you never catch them out.

The real picture: every change lands on one machine first, then travels to the others. Until it arrives, those copies still hold the old data, and anyone who reads from them gets it.

What happens when Sakura presses Save

Her profile lives on three database machines: tokyo-1 and tokyo-2, in the same Tokyo data centre, and london, on the other side of the world.

Each machine keeps its changes as a numbered list, its log. The leader’s log holds #1 name = Sakura. Here is the save, step by step:

1. The app sends the save to the leader, tokyo-1. It adds #2 name = Sakura Ito to its log.
2. tokyo-1 replies “saved”. The app shows “Saved” and reloads the page.
3. tokyo-1 sends #2 to each follower. tokyo-2 is in the same building, so it arrives in a few milliseconds. london is about 9,500 km away. Light in fibre covers about 200 km per millisecond, so even a straight cable would take about 48 ms; real cables wind around continents, and the trip takes about 110 ms, more when the network or london is busy.
4. Each follower adds #2 to its own log when it arrives. Until then it still answers “Sakura”.

The reload is a read. Most apps send reads to followers, to spread the load (more on that next). If Sakura’s reload reaches london before #2 does, she gets the old name.

A read that returns older data than the leader has is called a stale read. It is not an error. The database did exactly what it was set up to do.

So why keep copies at all?

• More reads. A profile is viewed far more often than it is edited. Every follower can answer reads, so adding a follower adds read capacity. Writes all still go to the one leader.
• Surviving a dead machine. Disks fail and data centres lose power. A follower already holds (almost) all the data, so it can take over instead of you restoring last night’s backup.
• Being close to readers. A follower in London answers London users in a few milliseconds instead of a round trip to Tokyo, which takes around 220 ms.

That is why PostgreSQL, MySQL and nearly every managed database offer followers. Copies will lag. The useful questions are by how much, and what you do about it.

Say “saved” now, or wait for the copies?

Look at step 2 again. tokyo-1 said “saved” before any follower had the change. That is asynchronous replication, or async: the leader acknowledges (replies “saved”) as soon as it has stored the change itself, and the copies follow later.

The other choice is synchronous replication, or sync: the leader waits until the followers confirm they have the change, and only then replies “saved”.

Making every save wait on every follower is rare, because one slow or dead follower would hold up every save. PostgreSQL and MySQL are async by default. MySQL’s semi-synchronous mode waits for just one follower to confirm. PostgreSQL lets you name which followers to wait for. A common setup is one sync follower nearby and the rest async.

How far behind is behind?

Lag depends on distance and on how busy the machines are. These are the numbers to plan with:

Now Sakura’s story adds up. After a save, the app reloads the page, and the browser asks for it again within tens of milliseconds of “Saved”. That is well inside london’s lag, and on a busy day it can be inside tokyo-2’s too.

Making sure Sakura sees her own change

Sakura doesn’t mind if a stranger sees her old name for half a second. She does mind seeing it herself, because it looks like her save failed. The guarantee she wants is called read-your-writes: after you save something, your own reads show it. Other people may still see the old value briefly.

There are two common ways to get it:

1. Read from the leader for a while. For a few seconds after a user saves, send that user’s reads to the leader, which always has the newest data. Everyone else still reads from followers. Cost: a little more work for the leader.
2. Wait for the follower. Remember the log number of the user’s last save (#2), for example in their session. When their read reaches a follower, it waits until that follower has reached #2. Cost: that one read can be slow.

In the demo, press Reset (back to async, london at 500 ms), set “Read your writes” to “Leader for 3 s”, save, and press “Read from london”. The read goes to the leader and comes back with the new name. With “Wait for my save”, the read stays with london but pauses until the copy lands, and tells you how long it waited.

When the leader dies

If the leader stops answering, a follower has to take its place. This is failover: noticing the leader is gone, promoting a follower to be the new leader, and pointing the app’s saves at it.

Agreeing which follower takes over, without two machines both deciding they are leader, is the problem Raft solves with a majority vote; see the Raft article on this site. It takes time: Amazon RDS says its failovers typically take 60 to 120 seconds.

The new leader should be the most up-to-date follower: the one whose log is longest (on a tie, the demo picks the first in its list). But with async, even that one can be missing the last few saves, the ones still on their way when the old leader died.

So the saves async can lose are the ones inside the promoted follower’s lag. With 10 ms of lag, you can lose any saves made in the last 10 ms before the crash; with 5 s of lag, any made in the last 5 s.

Press Reset, save in async, kill the leader, then press “Bring tokyo-1 back”. It still holds the save nobody else got, but the cluster has moved on without it, so it throws that save away and copies from the new leader. PostgreSQL ships a tool, pg_rewind, for exactly this clean-up.

Choosing for Sakura’s app

You can now make each choice on purpose instead of by accident:

• Profile names: async is fine. A stranger seeing an old name for half a second costs nothing. Add read-your-writes so Sakura always sees her own change.
• Payments and orders: losing a save after saying “saved” is not acceptable. Wait for at least one nearby follower (semi-sync) and accept slower saves.
• Readers in London: put a follower there for fast reads, and expect up to about a second of lag on them.
• Watch replication lag on a dashboard and alert when it grows. Every stale read and every lost save on failover happens inside the lag window.

The copies do end up the same. “End up” takes time, the lag, and everything in this article is about what happens inside it.