Yep. Built a crawler, an indexer/queryprocessor, and an engine responsible for merging/compacting indexes. Crawling was tricky. Something like stackoverflow

Yep. Built a crawler, an indexer/queryprocessor, and an engine responsible for merging/compacting indexes. Crawling was tricky. Something like stackoverflow will stop returning pages when it detects that you're crawling, much sooner than you'd expect.