> URLs get crawled basically in a first-come first-serve order I saw some references to round-robin in your code, I suppose you use it to hit a domain only eve

> URLs get crawled basically in a first-come first-serve order I saw some references to round-robin in your code, I suppose you use it to hit a domain only every x milliseconds during a crawl. Maybe additionally use Open Directory Project as start list: http://dmoz.org http://dmoz.org