One thing that would be very helpful to add to this is crawling strategies. Think a plugin architecture where crawler users can define metrics to decide whethe

One thing that would be very helpful to add to this is crawling strategies. Think a plugin architecture where crawler users can define metrics to decide whether a page found from a crawl is also worth crawling. Then provide to the plugin authors information like - Current Link depth - Current crawl chain for this page (so parent crawl site, grandparent, etc.) - Text from page - Etc. Then you can provide some default crawl strategies. Such as crawl to N depth. Crawl every other link. Crawl links matching some regular expression. Crawl links only on a specific domain, etc.