For Ruby I recommend Medusa Crawler gem. [1] https://github.com/brutuscat/medusa-crawler https://github.com/brutuscat/medusa-crawler Which I maintain as a for

For Ruby I recommend Medusa Crawler gem. [1] https://github.com/brutuscat/medusa-crawler https://github.com/brutuscat/medusa-crawler Which I maintain as a fork of the unmaintained Anemone gem.