AI · Advanced
LLM serving platform
Token-based limits, a per-model queue, continuously batched GPU workers and autoscaling on queue depth.
Every part, and why it is there
API customers
Developers send prompts and stream back tokens. A request might be 50 tokens or 100,000, so “one request” is a poor unit of load.
API gateway
Checks API keys and terminates long-lived streaming connections.
Request router
Picks the right model and region, checks quota, and enqueues the request with its priority. It keeps requests for the same long prompt on the same GPUs so cached prefixes can be reused.
Token rate limiter
Limits customers by tokens per minute, not requests per minute, because cost on the GPU scales with tokens.
Quota counters
Shared, fast counters for each customer’s token budget across all router instances.
Request queue (per model)
GPUs are expensive and cannot be added in seconds, so requests wait here briefly instead of overloading them. Queue depth is the clearest signal of how busy the fleet is.
GPU inference workers
Each worker runs an engine like vLLM that uses continuous batching: new requests join the batch between token steps instead of waiting for the slowest request to finish.
Model weights
Tens to hundreds of gigabytes per model. Loading weights is why a new GPU worker takes minutes to become useful, so scaling must start early.
Usage events
Input and output token counts for every request, published once and used for billing and analytics.
Billing aggregator
Sums tokens per customer per hour and writes invoices-in-progress.
Billing DB
Usage totals and invoices, the money record of the platform.
GPU & queue metrics
Queue wait time, GPU memory use and tokens per second per worker.
GPU autoscaler
Adds or removes GPU workers based on queue depth and wait time, and shifts capacity between models as demand moves.
Asked in interviews at
Learn more
- Fast LLM Serving with vLLM and PagedAttention (Anyscale)
- Mastering LLM Inference Optimization From Theory to Cost Effective Deployment: Mark Moyou (AI Engineer)
- vLLM: easy, fast, and cheap LLM serving with PagedAttention (vLLM)
- How continuous batching enables 23x throughput in LLM inference (Anyscale)
- Efficient memory management for large language model serving with PagedAttention (arXiv)
- Text Generation Inference (Hugging Face)