3 ms·How to crawl a quarter billion webpages in 40 hours (2012)4 points by downbad_ 6mo agodownbad_ 6mo agohttps://news.ycombinator.com/item?id=10865568 https://news.ycombinator.com/item?id=10865568