4 ms·
DMOZ, Wikipedia, Yahoo Directory are the classic broad starting points. You could also begin with the top 100, 500, 1000, etc. sites from some ranking service (
by gojomo 18y ago
DMOZ, Wikipedia, Yahoo Directory are the classic broad starting points. You could also begin with the top 100, 500, 1000, etc. sites from some ranking service (like Alexa), or top N results from major search engines on queries of special interest.
Depending on how you order discovered URLs and sites for crawling, it may not make too much of a difference where you start a truly web-wide crawl: you'll quickly reach major hubs, and everything else, after a short period. Then it's a matter of where the crawler chooses to spend its attention: which paths, how deep.
If you keep crawling 'over and over' you may want to pick what you revisit based on your own followup analysis, not the seeds of your first crawl(s).