Big spiders neither stop nor start. If each webpage gives you on average 200 new urls, you need to aggregate them, calculate ranks and schedule the top 0.5% for

Big spiders neither stop nor start. If each webpage gives you on average 200 new urls, you need to aggregate them, calculate ranks and schedule the top 0.5% for crawling. Crawling capacity is constant, the only question is the quality of your ranking formula.