Interesting, I also have a big scraping project and the breakdown is probably like 70% HTML parsing, 25% JSON APIs, 5% weird APIs. I can of course imagine that

Interesting, I also have a big scraping project and the breakdown is probably like 70% HTML parsing, 25% JSON APIs, 5% weird APIs. I can of course imagine that this pie chart simply depends on the sites/genre/industry you're scraping. DDoS protection does throw a wrench into the mix, though I don't blame anyone for using it. DDoS protection might seem antithetical to the web, but... so is DDoS and abuse. Kind of like how being an asshole is antithetical to getting along as a society but you still have to address the reality that there will always be abusers and bad actors. I also think being able to do what you want with your service is a fundamental part of the web incl putting it behind a captcha. It's just part of the beautiful chaos.