Deep dive · Core

Design a web crawler

A pipeline and coordination problem. It tests queue design, deduplication at a scale where a hash set no longer fits in memory, and politeness — the constraint candidates most often forget, and the one that gets a real crawler banned.

What it trains: Queues, deduplication, politeness, horizontal scale

1. Requirements

Functional

Start from seed URLs, fetch pages, extract links, and continue
Store page content for downstream indexing
Respect robots.txt and per-domain crawl delays
Recrawl pages on a cadence based on how often they change

Non-functional

Crawl on the order of 1 billion pages per month
Never overwhelm a single host — politeness is a hard constraint, not a nice-to-have
Crawl each URL roughly once: deduplicate both URLs and content
Fault tolerant: workers crash constantly and the crawl must not lose or repeat work

2. Back-of-envelope numbers

1B pages/month ≈ 400 pages/sec sustained; at ~2 seconds per fetch, that's ~800 concurrent fetches
500 KB average page × 1B ≈ 500 TB/month raw — compress and store in blob storage, not a database
A URL is ~100 bytes; 10B seen URLs ≈ 1 TB of dedupe state, far too big for one machine's memory
A Bloom filter at 10 bits per URL holds 10B URLs in ~12 GB with ~1% false positives

3. Architecture

  ┌────────────┐      ┌──────────────────────────────┐
  │ Seed URLs  │─────▶│  URL frontier                │
  └────────────┘      │  ┌────────────────────────┐  │
                      │  │ priority queues (F)    │  │
  ┌────────────┐      │  ├────────────────────────┤  │
  │  Extracted │─────▶│  │ per-host queues (B)    │  │
  │  links     │      │  │  + next-fetch time     │  │
  └─────▲──────┘      │  └────────────────────────┘  │
        │             └───────────┬──────────────────┘
        │                         │ one host per worker
  ┌─────┴────────┐        ┌───────▼────────┐
  │ Link extract │◀───────│ Fetcher pool   │──▶ robots.txt cache
  │ + normalise  │        │ (async HTTP)   │
  └─────┬────────┘        └───────┬────────┘
        │                         │ raw HTML
  ┌─────▼─────────┐        ┌──────▼──────┐      ┌──────────────┐
  │ Bloom filter  │        │ Blob store  │─────▶│ Content hash │
  │ (URL seen?)   │        │  (S3)       │      │  dedupe      │
  └───────────────┘        └─────────────┘      └──────────────┘

The URL frontier is the heart of the design: front queues encode priority, back queues enforce one-worker-per-host politeness.

The pipeline

URL frontier → fetcher → content store → parser → link extractor → URL filter and dedupe → back to the frontier. Every stage is independently scalable and connected by queues, which is what makes worker crashes survivable: an unacknowledged message is simply redelivered.

The frontier is the design

A single FIFO queue fails on both axes: it gives you no way to prioritise important pages, and it will happily point fifty workers at the same domain. The standard structure is two layers — front queues by priority, back queues by host — with a selector that only hands a worker a URL from a host whose crawl delay has elapsed.

4. Deep dives

This is where levels are decided. Go deep on one or two of these rather than shallow on all of them.

Politeness

Map each hostname to exactly one back queue, and assign each back queue to at most one worker at a time. That single invariant guarantees no host is fetched concurrently. Track next_fetch_time per host from robots.txt Crawl-delay or a default (say 1 request/second), cache robots.txt per host with a TTL, and honour it before every fetch. Send a real User-Agent with contact information.

URL deduplication at scale

Normalise first — lowercase the host, strip fragments and tracking parameters, resolve relative paths, sort query parameters — or you will crawl the same page a hundred ways. Then check membership in a Bloom filter: constant memory, no false negatives, and ~1% false positives, meaning you occasionally skip a page you've never seen. That is an acceptable trade at this scale; back it with a sharded persistent set if you need exactness.

Content deduplication

Different URLs frequently serve identical or near-identical content — mirrors, print views, session IDs. Hash the normalised content (MD5/SHA) to catch exact duplicates, and use SimHash or MinHash to catch near-duplicates. This can cut your storage and indexing bill dramatically and is a detail that signals real experience.

Traps and hostile pages

The web actively breaks crawlers: infinite calendar links, dynamically generated URL spaces, enormous pages, redirect loops, and slow-loris responses. Defend with a max crawl depth, a per-domain page cap, URL length limits, response size limits, aggressive timeouts, and a redirect-hop limit. Mentioning crawler traps unprompted is a strong signal.

Recrawl scheduling

Not all pages deserve equal attention. Track observed change frequency per URL and schedule recrawls adaptively — a news homepage every few minutes, a static documentation page monthly. Combine with a priority signal (inbound links, domain authority) so limited capacity goes to pages that matter.

Fault tolerance and distribution

Shard the frontier by hostname hash so each crawler node owns a disjoint set of hosts — this preserves politeness without cross-node coordination. Checkpoint frontier state so a node restart resumes rather than restarts. Use queue acknowledgements with visibility timeouts: a worker that dies mid-fetch simply lets its message reappear.

5. Tradeoffs to argue

Bloom filter vs exact set for URL dedupe

Upside: Bloom holds 10B URLs in ~12 GB

Cost: ~1% of pages are silently skipped as false positives

BFS vs priority-driven crawl order

Upside: BFS is simple and gives broad coverage quickly

Cost: Wastes capacity on low-value pages; priority queues need a scoring signal

Aggressive vs polite crawl rate

Upside: Higher concurrency per host crawls faster

Cost: Gets you blocked, and can amount to a denial-of-service on small sites

6. Questions the interviewer will ask

Ten workers pull URLs and three of them are for the same host. What stops you hammering it?
Your dedupe set no longer fits in memory. What now, and what do you give up?
A site generates infinite unique URLs. How does your crawler notice and stop?
How do you decide when to recrawl a page you fetched last week?

7. Mistakes that sink this answer

A single global queue with no per-host isolation
Assuming an in-memory hash set can hold the seen-URL list
No URL normalisation, so the same page is crawled many times
Ignoring robots.txt or omitting politeness entirely

Practice this live

Run a mock system design interview and get scored against your target level.

Start mock interview