I think the good people at Scrapy [1] have done a great job in keeping pace with making open-source crawler components that handle javascript. It is called Spla

I think the good people at Scrapy [1] have done a great job in keeping pace with making open-source crawler components that handle javascript. It is called Splash [2], and I use it for rendering out screenshots. [1] - https://scrapy.org https://scrapy.org [2] - https://github.com/scrapy-plugins/scrapy-splash https://github.com/scrapy-plugins/scrapy-splash