Data & analytics · Intermediate

Batch + stream data platform

Lambda and kappa side by side: one retained event log feeding a speed layer, a batch layer and a merged query engine.

Every part, and why it is there

Product apps

Web and mobile apps emit events: page views, purchases, errors. Every team downstream wants them, in different shapes and at different speeds.

Analysts & dashboards

Ask questions like “sales in the last 5 minutes” and “sales by region last year”. One needs fresh data, the other needs complete data.

Load balancer

Spreads the event firehose and the query traffic across their respective services.

Event collector

Accepts event batches, checks them against a registered schema and appends them to the log. Rejecting malformed events here saves every downstream team from cleaning them.

Schema registry

The agreed shape of each event type, with versions. Producers and consumers evolve independently without breaking each other.

Event log (retained 30 days)

The single source of truth for raw events. Long retention means any job can re-read history, which is what makes the kappa approach possible.

Stream processor (speed layer)

Continuously aggregates events into minute-level views. Fast, but it only sees what has arrived so far.

Real-time views

Fresh aggregates for live dashboards, kept for days rather than years.

Batch layer (Spark)

Periodically reads the log into the lake and rebuilds complete, corrected tables in the warehouse. Slow, but it sees everything, including late events.

Data lake

Every event as cheap columnar files, forever. Reprocessing, ML training and audits read from here.

Warehouse

Clean, modelled tables for business questions over months and years.

Query engine

Lets one query read recent data from the real-time views and history from the warehouse, merging them, which is the serving layer of a lambda architecture.

Learn more

All system design examples