3 ms·
A distributed web crawler infrastructure implemented using Go
- ethanwillis 6y agoOne thing that would be very helpful to add to this is crawling strategies. Think a plugin architecture where crawler users can define metrics to decide whether a page found from a crawl is also worth crawling. Then provide to the plugin authors information like - Current Link depth - Current crawl chain for this page (so parent crawl site, grandparent, etc.) - Text from page - Etc. Then you can provide some default crawl strategies. Such as crawl to N depth. Crawl every other link. Crawl links matching some regular expression. Crawl links only on a specific domain, etc.
- eltonsv 6y agoHi @ethanwillis, thanks a lot for the feedback! I'm actually trying to look at expanding the crawler code to provision spiders and pass along a configuration when creating it. Probably through CLI arguments or a .conf file, still undecided yet, and move on from there. Thanks a lot for the view suggestions, will definitely look at this when having a stab at this aspect :)