I’ve done a good amount of it and it is always a pain. I’ve never setup broad scrapers that index tons of sites, but instead always a select few that I wrote sp

I’ve done a good amount of it and it is always a pain. I’ve never setup broad scrapers that index tons of sites, but instead always a select few that I wrote specific crawlers for. 1. The act of scraping public pages was declared legal (at least in the US). What you choose to do with that data may or may not be legal. Comparing prices shouldn’t be an issue though IANAL. 2. As someone else suggested I’d start with scrapy. It can easily take a sitemap.xml and start chugging along. You’ll need to build the specific scraper for each site. The browser dev tools will become your best friend. One thing to look into is the network calls. Rather than loading static content, they may be loading JSON from their servers and those are significantly easier to pull data from.