3 ms·
Though you could end up with a load of URLs, titles, and a summary text, crawling google won't really get you to the important data in building a search engine.
by pedalpete 17y ago
Though you could end up with a load of URLs, titles, and a summary text, crawling google won't really get you to the important data in building a search engine.
No follow-on links, only a subset of the text, no referrals, etc.
Love the 'paradox' of the infinite spider loop, hadn't considered that before.
- Tichy 17y agoCrawling groups would be interesting. What other archives of usenet are there?
- jacquesm 17y agoalso, google had no problem crawling groups when it was still called dejanews.
- anarcticpuffin 17y agoMost spiders implement a maximum depth in their search though. So you'd really just get an inefficient crawl, not an infinite one.
- pedalpete 17y agoOf course, using Google as a seed list more than crawling only google. Not sure why I didn't think of that.