4 ms·

Cloudflare might end a golden era of scraping, when it was trivial to scrape data from any site. Now Cloudflare helps site owners to make sure than only humans

Cloudflare might end a golden era of scraping, when it was trivial to scrape data from any site. Now Cloudflare helps site owners to make sure than only humans can read their contents manually. As more site owners switch to similar services, web will become less and less machine readable. No automated data processing, no archiving.

nine_k 4y ago
But wait, AI models will help bots looks like real humans accessing a site! They'll try hard to will fool the AI models that check if a site is browsed by a human. Ha-ha, only serious.
- vdfs 4y ago
  No need for AI, browser can easily be automated and captcha can be solved using cheap services
  - 6c737133 4y ago
    > captcha can be solved using cheap services call it what it is - you're using slave labor in a 3rd world country to solve rudimentary puzzles for you
    - from 4y ago
      It's probably not slave labor. It may be really poorly paid labor but if you had slave labor you'd probably use it for something profitable like construction like they do in the Persian gulf countries instead of solving captchas that people pay $3 per 1000 for.
bjord 4y ago
this won't stop the overall trend, but it can help you get around cloudflare's effective scraping blocking (copying my comment from a previous thread): If you're scraping with Python, try cloudscraper—among other things(!), it supports JS rendering (basically the bare-minimum check cloudflare does), without needing to run a full browser in the background. It's built on requests, so integration was pretty easy. https://github.com/venomous/cloudscraper https://github.com/venomous/cloudscraper
- wefarrell 4y ago
  JS rendering is not enough. Cloudflare monitors UI interactions and browser footprints to assess whether it’s a human or a bot.
  - bjord 4y ago
    trust me, I'm aware—cloudscraper can also solve cloudflare challenges, including turnstile

nine_k 4y ago

But wait, AI models will help bots looks like real humans accessing a site! They'll try hard to will fool the AI models that check if a site is browsed by a human. Ha-ha, only serious.

vdfs 4y ago
No need for AI, browser can easily be automated and captcha can be solved using cheap services
- 6c737133 4y ago
  > captcha can be solved using cheap services call it what it is - you're using slave labor in a 3rd world country to solve rudimentary puzzles for you
  - from 4y ago
    It's probably not slave labor. It may be really poorly paid labor but if you had slave labor you'd probably use it for something profitable like construction like they do in the Persian gulf countries instead of solving captchas that people pay $3 per 1000 for.

bjord 4y ago

this won't stop the overall trend, but it can help you get around cloudflare's effective scraping blocking (copying my comment from a previous thread): If you're scraping with Python, try cloudscraper—among other things(!), it supports JS rendering (basically the bare-minimum check cloudflare does), without needing to run a full browser in the background. It's built on requests, so integration was pretty easy. https://github.com/venomous/cloudscraper https://github.com/venomous/cloudscraper

wefarrell 4y ago
JS rendering is not enough. Cloudflare monitors UI interactions and browser footprints to assess whether it’s a human or a bot.
- bjord 4y ago
  trust me, I'm aware—cloudscraper can also solve cloudflare challenges, including turnstile