Infrastructure · Advanced

Metrics & alerting

Buffered metric ingestion into a time-series DB, a cached query tier and deduplicated alerting.

Every part, and why it is there

Hosts & containers

Thousands of machines each reporting CPU, memory, request counts and latencies every 10–60 seconds. That is millions of samples per second in total.

Ingest load balancer

Spreads the constant stream of writes over the ingest tier.

Ingest gateway

Validates samples and rejects series with too many unique label values (e.g. a user id as a label), which would explode storage. Then hands data to the buffer and answers fast.

Sample buffer

Absorbs spikes and lets the database be restarted or upgraded without dropping data. Agents never wait on storage.

TSDB writers

Group samples by series and write them in compressed blocks. Consecutive values are similar, so delta encoding squeezes each sample to a couple of bytes.

Time-series DB

Recent, full-resolution data (e.g. 15 days) partitioned by time, so queries for “the last hour” touch only recent blocks.

Long-term blocks

Older data, downsampled to 5-minute or 1-hour averages, kept for a year or more at a fraction of the cost.

Dashboards

Engineers and auto-refreshing dashboards querying graphs. Many dashboards run the same queries every few seconds.

Query service

Splits a query by time range: recent data from the TSDB, older data from long-term storage, then merges the results.

Query result cache

Caches results for completed time ranges, which never change, so refreshing a dashboard only computes the newest minutes.

Alert rule evaluator

Every 30–60 seconds runs each alert query, such as “error rate above 2% for 5 minutes”. The “for 5 minutes” stops brief blips from paging anyone.

Firing alerts

Alerts that crossed their threshold. Buffered so a notification outage does not lose them.

Alert manager

Deduplicates, groups related alerts into one page, silences planned maintenance and routes to the right team. Without it one outage sends hundreds of pages.

Pager / chat

The on-call phone or chat channel. If the monitoring system shares fate with what it monitors, nobody gets paged, so it is often run separately.

Asked in interviews at

  • linkedin
  • stripe
  • google
  • meta
  • rippling
  • amazon
  • doordash
  • pinterest

Concepts used here

  • HyperLogLog: Count 10 million different visitors in 12 KB, to within about 1%, without storing a single ID.
  • Segment tree: Build a tree of totals over a ramen shop’s daily sales, then answer any “days l to r” question and absorb any change in about 20 steps for a million days.
  • Square-root decomposition: Keep one total per block of about √n numbers, so both “change one number” and “add up a stretch” stay cheap.
  • Fenwick tree: Running totals that stay fast while the numbers keep changing, in one plain array the same size as the data. The binary of each position does the work.

Learn more

All system design examples