To build a smart spider, you can have a classifier that determines if you should crawl the outlinks of the page. This classifier can be trained using logistic

To build a smart spider, you can have a classifier that determines if you should crawl the outlinks of the page. This classifier can be trained using logistic regression. This is slightly tricky, but effective. I've been meaning to write in more depth about this topic (smart crawling). [edit: I'm stepping out so I can't write more right now. You can email me if you have any questions about this.]