4 ms·
How did you go about seeding your web crawler with URLs to crawl?
by c0wb0yc0d3r 5y ago
How did you go about seeding your web crawler with URLs to crawl?
- dannyw 5y agoNot OP, but if I was to do this, I'd start by downloading Wikipedia and all its external links and references, and crawling from there. You should eventually reach most of the publicly visible internet.
- c0wb0yc0d3r 5y agoI feel a little embarrassed that I didn't think of something like that. When I did some crawler experimenting in my younger years, I thought I was pretty clever using sites that would let you perform a random Google searches. I would just crawl all the pages from the results returned. Your method would undoubtedly be more interesting I think. It would certainly lead to interesting performance problems quicker, I bet.
- marginalia_nu 5y agoI just started with my website and did a crawl. Subsequently I've been seeding it with the best results form my previous crawls. It's a directed search so it doesn't seem to need a particularly solid seed to get decent results.
- c0wb0yc0d3r 5y agoSo how long did it take to get to 17 million URLs?