CI/CD build system

Every part, and why it is there

Developers

Push commits and open pull requests many times a day. Their main need is fast, trustworthy feedback: a flaky or slow CI gets ignored.

Git hosting

Stores the code and emits an event for every push and pull request. CI reacts to those events instead of polling the repository.

Repository storage

Git objects and pack files. Runners fetch shallow clones from a nearby mirror so thousands of checkouts do not overload it.

Push & PR events

Buffers webhook events so a burst of pushes before a release cannot overwhelm the scheduler, and nothing is lost if the scheduler restarts.

Merge queue & scheduler

Turns an event into a pipeline of jobs. The merge queue tests each change on top of the changes ahead of it, so main stays green even when many people merge at once.

Pipeline state

Which jobs ran, passed or failed, and their logs index. Needs real transactions because job states change concurrently.

Job queue

Jobs ready to run, by type of machine they need. Runners pull work, so adding capacity is just starting more runners.

Ephemeral runners

A fresh, clean machine or container per job, destroyed afterwards. That prevents one build’s leftovers from making another pass or fail by accident.

Remote build cache

Stores build outputs keyed by a hash of their inputs. If anyone already built the same inputs, the result is downloaded instead of rebuilt, turning many builds from minutes into seconds.

Artifact registry

The exact images and packages that passed tests. Deploying this same artifact, not rebuilding it, guarantees production runs what was tested.

Build results

Pass/fail events for chat notifications, dashboards and the deploy controller.

Deploy controller

Rolls a passing artifact out gradually: a small canary slice first, then the rest if error rates stay normal, and automatic rollback if they do not.

Production load balancer

Shifts a small percentage of traffic to the new version during a canary, so a bad release only affects a few users.

Production services

What all of this exists to change safely and often.

Asked in interviews at

• openai
• axon

Learn more

• CI/CD In 5 Minutes | Is It Worth The Hassle: Crash Course System Design #2 (ByteByteGo)
• Top 5 Most-Used Deployment Strategies (ByteByteGo)
• Your Build in a Datacenter Remote Caching and Execution in Bazel (FOSDEM)
• Continuous Integration (Martin Fowler)
• Remote caching (Bazel)
• Managing a merge queue (GitHub Docs)
• Canary release (Martin Fowler)