I had a nice experience with https://simplescraper.io https://simplescraper.io for a similar use-case. Was able to scrape a few thousand URLs without too much f

I had a nice experience with https://simplescraper.io https://simplescraper.io for a similar use-case. Was able to scrape a few thousand URLs without too much fuss. The biggest complication with visual scrapers is all the edge cases. The selector algorithms usually become a mess on any complex website especially if there's uneven data. Then you have css selectors no longer working and so on. Very brittle.