Fundamentals · Intermediate

Web crawler

A polite, deduplicating crawler with a per-host frontier, fetcher fleet and link-extracting parsers.

Every part, and why it is there

Crawl operators

People or jobs that submit seed URLs and crawl policies. Everything else the crawler finds on its own by following links.

API gateway

Entry point for seeds and crawl controls, with authentication so only trusted callers can steer the crawl.

Seed API

Validates and normalises seed URLs (lower-case host, strip fragments) before they enter the crawl. Normalising early is what makes deduplication work later.

Frontier scheduler

Decides what to fetch next: higher priority pages first, and never more than one request every few seconds to the same host. That politeness rule is why the frontier is organised by host.

URL frontier

Billions of URLs waiting to be crawled, partitioned by host with a “next allowed fetch time” per host. It is far too big for memory, so it lives in a partitioned store.

Fetch jobs

URLs that are ready to fetch right now. Decouples choosing work from doing it, so fetchers can be added or crash without losing work.

Fetchers

Download pages. Mostly waiting on the network, so many run in parallel. They obey robots.txt and time out slow sites instead of getting stuck.

DNS & robots.txt cache

DNS lookups and robots.txt fetches are repeated for every page on a host. Caching them removes a large hidden share of latency.

Raw pages

Every fetched page is stored as-is, so parsing or indexing can be re-run later without crawling the web again.

Fetched-page events

Announces each new page to the parsers. Fetching and parsing scale separately: one is network-bound, the other CPU-bound.

Parsers & link extractors

Extract text and outgoing links, drop links already seen, and add the rest to the frontier. A hash of the page content also catches mirror sites with different URLs.

Seen URLs (Bloom filter)

Answers “have we crawled this URL before?” using a Bloom filter, which uses a few bits per URL. It can rarely say “seen” for a new URL, which is an acceptable loss; it never says “new” for a seen one.

Content index

The point of the crawl: searchable text for a search engine or dataset.

Asked in interviews at

  • meta
  • atlassian
  • zoominfo
  • lyft
  • monzo
  • glean
  • google

Concepts used here

  • Bloom filter: Answer “have I seen this before?” for a hundred million names in about 10 bits each. It never wrongly says no.
  • Bit tricks and bitmasks: Store “which cities have I visited?” as one number, and the shortest round trip through 20 stops drops from years of work to under a second.

Learn more

All system design examples