Fundamentals · Beginner

Rate limiter

Token buckets in Redis checked at the gateway, so no single client can overwhelm the API.

Every part, and why it is there

API clients

Callers identified by an API key or user id. Some are well behaved, some retry in tight loops; the limiter exists so one noisy caller cannot use up capacity meant for everyone.

API gateway

Every request passes through here, so it is the one place to enforce limits before any real work is done. Limiting inside each backend instead would mean every team re-implements it.

Rate limiter

Asks “may this key make one more request?” in a millisecond or two, forwards the request if yes and returns 429 if no. It is stateless itself, so you can add replicas freely; the state lives in Redis.

Token buckets (Redis)

Holds a small bucket per key: tokens left and when it was last refilled. A short Lua script reads, refills and decrements in one atomic step, so two limiter replicas cannot both spend the last token.

Limit rules

Which limit applies to which key or endpoint, e.g. 100 requests/second for free plans. Limiters cache these locally and refresh them, so changing a limit needs no deploy.

Throttle metrics

Counts allowed and rejected requests per key. Without it you cannot tell a misbehaving client from a limit set too low.

Backend API

The service being protected. It only sees requests that already passed the limiter, so its capacity planning can assume a known ceiling of traffic.

App database

The expensive resource the limits ultimately guard. Databases degrade badly when overloaded, which is why shedding load at the edge is cheaper than scaling the database for spikes.

Asked in interviews at

  • oracle
  • microsoft
  • pinterest
  • apple
  • openai
  • roku
  • amazon
  • jpmorgan

Concepts used here

  • Load balancing: One address in front of many servers. Follow one shopper’s request on sale day through DNS, the balancer’s choice of server, a crash, and back.
  • TCP: Follow one page load from Enter to pixels and see how TCP turns a network that loses things into a connection that never does, and what that costs in round trips.
  • Count-Min Sketch: Count 500 million hashtags a day in about half a megabyte. It may overcount a little, but it never undercounts.
  • Token bucket: A rate limit with two knobs: how fast permits come back, and how many can pile up. Set both and a real app’s burst gets through while a flood does not.

Learn more

All system design examples