I had a look at the "design a crawler" part, since that's something I have experience with. It makes some pretty sketchy assumptions and weird design choices.

I had a look at the "design a crawler" part, since that's something I have experience with. It makes some pretty sketchy assumptions and weird design choices. First in the back-of-the-envelope calculation it overestimates the size of a document by about two orders of magnitude. It's closer to 7.5 KB than 500 KB on average, uncompressed. 500 KB is closer to the size of Homer's Odyssey, which I will state is much longer than most websites. I'm willing to give them the benefit of the doubt though, maybe I'm misunderstanding what these 500 KB is supposed to consist of. I also don't think the crawl frontier design would work at the sort of scales they imagine. This is a design that will get you to maybe 10 million documents. Apart from the profoundly questionable choice of using a nosql database for this, apart from the race condition in the crawling algorithm, Redis can only store 4 billion (2^32-1) items in a set, and let me tell you, crawl frontiers grow very quickly when you design them like this one has been designed. By the time you're indexing a billion documents, your crawl frontier will be of many orders of magnitude more. The biggest mistake is to think too link-oriented. You probably want to retrieve links one domain at a time as this permits connection pooling (otherwise your biggest crawling bottleneck will be waiting for the remote end to acknowledge the connection's closure), helps with deduplication and means you don't need to re-fetch robots.txt for every single document you retrieve. It also solves many problems limiting how much you fetch from a single domain, since you can just set a hard time and link limit. You don't have to re-fetch every document on that domain when you refresh it, just get the new ones and sample the existing ones. This also means you don't need a billion-entry priority queue, you can get away with a manageable queue of a few million instead.