AI · Advanced

ML feature store & training

Shared features computed in batch and streaming, offline lake for training, online store for serving.

Every part, and why it is there

Customer app

Asks “how long until my food arrives?” and needs a prediction in tens of milliseconds.

API gateway

Routes the app’s request to the product service that needs a prediction.

ETA product service

Business logic that needs a model’s answer. It calls model serving with just an order id; it does not compute features itself.

Model serving

Loads the current model version, fetches its features by name from the online store and returns a prediction.

Online feature store

Latest feature values per entity (restaurant, courier) in a low-latency key-value store such as Cassandra. Serving reads it; it never scans history.

Model registry

Versioned model files with the exact feature list and training data they used, so you can roll back or explain a prediction.

Event stream

Orders, prep times and courier pings. The raw material for both real-time features and training data.

Streaming features

Computes fresh features such as “orders at this restaurant in the last 10 minutes” and writes them to the online store within seconds.

Batch features & lake loader

Lands events in the lake and computes slow-moving features such as “average prep time last 30 days”, writing them to both the lake and the online store.

Offline feature lake

Every feature value with its timestamp. Training reads values as they were at the moment of each past event, which avoids leaking the future into the model.

Training triggers

A new day of features landing (or a schedule) triggers a retrain.

Training pipeline

Builds a training set from the lake, trains and evaluates the model, and publishes it to the registry only if it beats the current one.

Learn more

All system design examples