Messaging & realtime · Intermediate

Chat app

Realtime messaging with WebSockets, presence in Redis and messages in a wide-column store.

Every part, and why it is there

Mobile app

Most messages come from phones. The app keeps one WebSocket open while in the foreground and falls back to push notifications when closed.

Web app

Same API as mobile. Browsers reconnect often (tab switches, sleep), so connect traffic is higher here.

Load balancer

Spreads connections and API calls across servers. It must support long-lived WebSocket connections, so idle timeouts are raised to minutes.

WebSocket gateway

Holds one open connection per device and pushes new messages down it instantly. It does no business logic, so one box can hold tens of thousands of connections.

Chat service

Validates and stores each message, then publishes an event. Clients get an ack once the message is durable, before anyone else receives it.

Presence (Redis)

Tracks who is online and which gateway holds their connection, with a short expiry refreshed by heartbeats. Losing it only costs a few seconds of wrong "online" dots.

Messages (Cassandra)

Messages are partitioned by conversation and sorted by time, so loading a chat is one sequential read. It handles very high write rates, which a single SQL primary would not.

Message events

Every stored message becomes an event. Delivery, push notifications and search indexing all consume it independently, so a slow consumer never slows sending.

Push notifications

Consumes message events and, for recipients who are offline, looks up their device tokens and sends a push.

APNs / FCM

Apple and Google deliver the notification and wake the phone. Your servers never talk to a sleeping phone directly.

Asked in interviews at

  • meta
  • anthropic
  • databricks
  • airbnb
  • uber
  • lyft
  • amazon
  • microsoft

Concepts used here

  • TCP: Follow one page load from Enter to pixels and see how TCP turns a network that loses things into a connection that never does, and what that costs in round trips.
  • Network layers: Build a real packet by hand: 42 bytes of HTTP inside three envelopes, and which boxes on the way open which one.
  • LSM tree: Follow one write through Cassandra- or RocksDB-style storage: append it to a log, sort it in memory, write whole files, and clean up old copies later.

Learn more

All system design examples