5 ms·

I just wrote ~20 scrapers and maybe 3 ended up being able to grab data from a JSON API. Mostly what I ran into was (an HTTP API that returns) templated HTML, an

I just wrote ~20 scrapers and maybe 3 ended up being able to grab data from a JSON API. Mostly what I ran into was (an HTTP API that returns) templated HTML, and wacky Sharepoint stuff. For the Sharepoint stuff, I found I often had to grab tokens out of script tags. Sometimes they were in hidden inputs, but either way is kind of the same thing. I was ready to break out a JS parser, but fortunately I didn't need to. I did run into Cloudflare DDoS protection and Incapsula, which I will say is pretty irritating and IMO antithetical to the web. Incapsula is so bad I get captcha'd just browsing around in a Firefox private window. If I were polling every few seconds or something I'd get it, but denylisting all AWS IPs or looking for "headless" in the User Agent (or looking at navigator params, testing TLS fingerprints, etc.) is bonkers. It's the laziest kind of upselling from web developers where you're making the site harder to use, but not actually keeping real scrapers out, because they're doing even more JavaScript interventions ahead of the HTTP request and using residential IP proxies.

hombre_fatal 4y ago
Interesting, I also have a big scraping project and the breakdown is probably like 70% HTML parsing, 25% JSON APIs, 5% weird APIs. I can of course imagine that this pie chart simply depends on the sites/genre/industry you're scraping. DDoS protection does throw a wrench into the mix, though I don't blame anyone for using it. DDoS protection might seem antithetical to the web, but... so is DDoS and abuse. Kind of like how being an asshole is antithetical to getting along as a society but you still have to address the reality that there will always be abusers and bad actors. I also think being able to do what you want with your service is a fundamental part of the web incl putting it behind a captcha. It's just part of the beautiful chaos.
- camgunz 4y ago
  In the large I don't disagree: people have the right to protect themselves. But I think you've made an argument for caching, not for captchas. I'd even be fine with changing cache-control interpretations to "cache this, because if you come back before it's changed we won't serve it to you again". But this stuff is obvious web developer upsell. For what it's worth, it didn't even work. Headless Chrome and some editing of the JavaScript environment was all it took. So it's definitely a ripoff.

hombre_fatal 4y ago

Interesting, I also have a big scraping project and the breakdown is probably like 70% HTML parsing, 25% JSON APIs, 5% weird APIs. I can of course imagine that this pie chart simply depends on the sites/genre/industry you're scraping. DDoS protection does throw a wrench into the mix, though I don't blame anyone for using it. DDoS protection might seem antithetical to the web, but... so is DDoS and abuse. Kind of like how being an asshole is antithetical to getting along as a society but you still have to address the reality that there will always be abusers and bad actors. I also think being able to do what you want with your service is a fundamental part of the web incl putting it behind a captcha. It's just part of the beautiful chaos.

camgunz 4y ago
In the large I don't disagree: people have the right to protect themselves. But I think you've made an argument for caching, not for captchas. I'd even be fine with changing cache-control interpretations to "cache this, because if you come back before it's changed we won't serve it to you again". But this stuff is obvious web developer upsell. For what it's worth, it didn't even work. Headless Chrome and some editing of the JavaScript environment was all it took. So it's definitely a ripoff.