Design a web crawler

Question 1

Where does the list of URLs still to fetch live?

Accepted Answer

The frontier is the crawler. It has to be durable so a restart does not lose the work, shared so workers can be added, and partitioned by host so politeness is structural rather than a rule everyone has to remember.

Question 2

What actually makes the HTTP requests?

Accepted Answer

Fetching is IO-bound and embarrassingly parallel — many workers pulling from the frontier, which is why the frontier has to be shared.

Question 3

Two pages link to each other. What stops you crawling in circles forever?

Accepted Answer

Every extracted URL is checked against what has been seen before it is enqueued. At a billion pages that set does not fit in memory as strings — a hash set or Bloom filter is what makes it affordable, at the price of very occasionally skipping a page.

Question 4

Where do the fetched pages go?

Accepted Answer

100KB a page across a billion pages is 100TB a month. That is object storage, with the small structured parts — URLs, links, timestamps — kept separately where they can be queried.

Question 5

Who pulls the links out of a fetched page?

Accepted Answer

Parsing is CPU-bound where fetching is IO-bound, so they scale on different curves and are usually separated for that reason alone.

Question 6

How many pages a second do you have to sustain?

Accepted Answer

1,000,000,000 / 30 days / 86,400 seconds is about 390 pages a second. Modest — which tells you the difficulty here is politeness and de-duplication, not throughput.

Question 7

How do you avoid hammering a single small site?

Accepted Answer

Politeness works when it is structural: partition the frontier by host so each host has its own queue and its own pace. A global rate limit either throttles everything or protects nobody, and robots.txt has to be fetched and honoured per host.

Blueprint — system design