Infrastructure · Intermediate

Centralised logging

Agents ship logs into Kafka; processors redact and enrich; a hot index and cheap archive serve queries.

Every part, and why it is there

Applications

Services writing log lines to stdout. Structured (JSON) logs with a request id are far easier to search than free text.

Node agents

One small agent per machine (e.g. Fluent Bit or the OpenTelemetry Collector) tails log files, adds host and service labels, and ships batches. Apps never block on logging.

Raw log topic

A durable buffer between thousands of agents and the processing tier. If processing falls behind or the index is down, logs wait here instead of being dropped.

Parse, enrich, redact

Turns text into fields, adds metadata such as team or region, and masks secrets and personal data before anything is stored. Doing this centrally means every team gets it.

Clean log topic

Processed logs, fanned out to every consumer that needs them: indexing, archiving and alerting each read independently.

Indexers

Write logs into the search index in bulk batches, which is much cheaper than one document at a time.

Hot index (7 days)

Fast full-text search over recent logs, which is what incident responders need. Indexing is expensive, so only a short window is kept here.

Archiver

Writes compressed, time-partitioned files (e.g. one per service per hour) to object storage.

Archive (1 year)

Cheap long-term storage for audits and rare investigations. Slower to query, but costs a small fraction of the index.

Log alerts

Watches the stream for patterns such as a spike of “payment failed” and raises an alert within seconds.

Pager / chat

Where log-based alerts land for the on-call engineer.

Engineers

Search logs during an incident, usually by request id, service and time range.

Log query API

Sends recent-time queries to the hot index and older ones to the archive, and enforces who may see which services’ logs.

Asked in interviews at

  • microsoft
  • apple
  • amazon
  • google

Concepts used here

  • LSM tree: Follow one write through Cassandra- or RocksDB-style storage: append it to a log, sort it in memory, write whole files, and clean up old copies later.

Learn more

All system design examples