Infrastructure · Advanced

Multi-region active-active

Two live regions behind geo DNS, each self-sufficient, replicating writes to each other asynchronously.

Every part, and why it is there

Users worldwide

Users in America and Europe. Serving them from a nearby region cuts 100+ ms of round-trip time from every request.

Geo / latency DNS

Answers each user with the closest healthy region. If health checks fail for a region, DNS stops returning it and its users move to the other one within minutes (bounded by the record’s TTL).

US load balancer

Entry to the US region. Each region is a complete, independent copy of the stack.

EU load balancer

Entry to the EU region, mirroring the US one.

US app servers

Serve reads and writes locally, never calling the other region in the request path. Each region must have spare capacity to absorb the other’s traffic during a failover.

EU app servers

Same code, same configuration, different region. Keeping them identical is what makes failover boring.

US cache

Regional cache. Caches are not replicated across regions; each warms up from its local data.

EU cache

Regional cache for EU. After a failover it starts cold, so the database must cope with a burst of misses.

US data replica

A full copy of the data that accepts writes locally. Writes return as soon as the US copy has them; they reach the EU a moment later.

EU data replica

The EU copy. Because both regions accept writes, two users can change the same record at once and the system needs a rule to resolve it.

US change stream

Every write made in the US, in order. This is how changes leave the region without slowing the user’s request.

Replicator → EU

Applies US changes to the EU copy, resolving conflicts (often “last write wins” by timestamp). The lag between regions is usually well under a second.

EU change stream

Every write made in the EU, flowing the other way.

Replicator → US

Applies EU changes to the US copy. Each replicator must ignore changes that originally came from its own region, or writes would loop forever.

Concepts used here

  • Load balancing: One address in front of many servers. Follow one shopper’s request on sale day through DNS, the balancer’s choice of server, a crash, and back.
  • Replication: A copy of your database is not a live mirror. Copies run behind, so you can read old data, and a crash can lose a save you were told was safe.
  • CAP theorem: When the network between two copies of your data breaks, each side must either refuse some requests or risk a wrong answer. You choose which, ahead of time.
  • TCP: Follow one page load from Enter to pixels and see how TCP turns a network that loses things into a connection that never does, and what that costs in round trips.
  • Network layers: Build a real packet by hand: 42 bytes of HTTP inside three envelopes, and which boxes on the way open which one.
  • Raft consensus: Five servers hold copies of the same settings. Raft’s four rules stop them from ever disagreeing, even when the network splits them in two.

Learn more

All system design examples