Infrastructure · Advanced

Object storage

S3-like storage: stateless front ends, a sorted key index and erasure-coded data with constant repair.

Every part, and why it is there

SDKs & apps

PUT, GET and LIST objects by bucket and key over HTTP. Objects range from bytes to terabytes; large ones are uploaded in parts.

DNS

Resolves bucket.region.example.com to many front-end addresses, spreading clients across the fleet before any load balancer is involved.

Load balancers

Spread TCP connections over hundreds of front ends. Layer 4 is enough because the front ends do all request parsing.

API front ends

Stateless: authenticate, look up or write metadata, then stream bytes to or from storage nodes. Because they hold nothing, any one can fail without data loss.

Auth & policy

Checks the request signature and evaluates bucket and user policies. Runs on every request, so policies are cached close to the front ends.

Policies

Who may do what on which bucket. Changes are rare compared with reads, so they are replicated widely.

Metadata service

Maps bucket + key to where the bytes live and keeps size, checksum and version. Also decides where new data should be placed based on free space and failure domains.

Key index

A huge sorted, partitioned table of every object’s key. Sorted order is what makes LIST with a prefix efficient; partitions split as they grow hot.

Storage nodes (erasure-coded)

Data is cut into shards plus parity shards (e.g. 10 + 4) spread over different disks, racks and zones. Any 10 of the 14 rebuild the object, so durability is high with far less overhead than 3 full copies.

Object events

Emits “object created/deleted” so other systems can react, e.g. making thumbnails, without polling.

Event functions

Customer code triggered by events. Runs outside the storage system so a slow function never slows uploads.

Durability scanners

Constantly read and checksum data in the background and rebuild shards from failed disks. With millions of disks, some fail every day; repair speed decides real durability.

Asked in interviews at

  • coupang
  • pinterest
  • cohesity
  • databricks
  • oracle
  • google
  • amazon

Concepts used here

  • Merkle tree: Two copies of a million rows find the few that differ by comparing about 41 short fingerprints, not a million rows.

Learn more

All system design examples