4 ms·
How We Built Our 60-Node (Almost) Distributed Web Crawler
- netvarun 14y agoOP here. This post is an expansion of my comment http://hackerne.ws/item?id=4369590 http://hackerne.ws/item?id=4369590 on the excellent post 'How to crawl a quarter billion webpages in 40 hours' ( http://hackerne.ws/item?id=4367933 http://hackerne.ws/item?id=4367933) Don't hesitate to ask me any questions.