3 ms·

> Neither Cloudflare, nor any other service, will ever be able to block all scrapers. They can make their operations more expensive, Cloudflare presents like s

by nabla9 1y ago

> Neither Cloudflare, nor any other service, will ever be able to block all scrapers. They can make their operations more expensive, Cloudflare presents like single platform for crawlers. The get the same amount of data as platforms to bock crawlers they don't want. Other big platforms can prevent scrapers effectively when they don't want them Google, Facebook. etc. Nifty new scraper might crawl few million url's before it's detected.

mmarian 1y ago
Hey! Sorry, didn't quite catch what you meant. Is it that Cloudlare can always spot crawlers because of the amount of data they collect? Or is it there's always a nifty new scraper that will get away with it?
- nabla9 1y ago
  It's that Cloudflare can always spot crawlers. Few million random urls crawled is nothing, and provides no value for AI companies, they want all. Comprehensive crawl of LinkedIn, FB, instagram, IMDB, Amazon, would be worth a lot.
  - mmarian 1y ago
    > Cloudflare can always spot crawlers I mention in the post a scraping service that Cloudflare isn't spotting: https://www.scrapingbee.com/blog/how-to-bypass-cloudflare-antibot-protection-at-scale/ https://www.scrapingbee.com/blog/how-to-bypass-cloudflare-an... Plenty of open-source ones as well that could bypass, eg maybe this one that came up in search https://github.com/VeNoMouS/cloudscraper https://github.com/VeNoMouS/cloudscraper Combine with residential proxies and you're just not going to find them. > Comprehensive crawl of LinkedIn, FB, instagram, IMDB, Amazon, would be worth a lot. Just from a quick Google search: - LinkedIn: https://brightdata.com/products/datasets/linkedin https://brightdata.com/products/datasets/linkedin - Amazon: https://www.junglescout.com/features/product-database/ https://www.junglescout.com/features/product-database/
    - nabla9 1y ago
      As I said, partial scrapes of small subsets over long time provide no real value for AI scrapers. Just an example: Brightdata linkedin database has 19 million entries. Linkedin has over 1 billion members. As I said, partial scrapes of small subsets over long time provide no real value for AI scrapers (repeating the main argument).

by nabla9 1y ago

mmarian 1y ago

Hey! Sorry, didn't quite catch what you meant. Is it that Cloudlare can always spot crawlers because of the amount of data they collect? Or is it there's always a nifty new scraper that will get away with it?

nabla9 1y ago
It's that Cloudflare can always spot crawlers. Few million random urls crawled is nothing, and provides no value for AI companies, they want all. Comprehensive crawl of LinkedIn, FB, instagram, IMDB, Amazon, would be worth a lot.
- mmarian 1y ago
  > Cloudflare can always spot crawlers I mention in the post a scraping service that Cloudflare isn't spotting: https://www.scrapingbee.com/blog/how-to-bypass-cloudflare-antibot-protection-at-scale/ https://www.scrapingbee.com/blog/how-to-bypass-cloudflare-an... Plenty of open-source ones as well that could bypass, eg maybe this one that came up in search https://github.com/VeNoMouS/cloudscraper https://github.com/VeNoMouS/cloudscraper Combine with residential proxies and you're just not going to find them. > Comprehensive crawl of LinkedIn, FB, instagram, IMDB, Amazon, would be worth a lot. Just from a quick Google search: - LinkedIn: https://brightdata.com/products/datasets/linkedin https://brightdata.com/products/datasets/linkedin - Amazon: https://www.junglescout.com/features/product-database/ https://www.junglescout.com/features/product-database/
  - nabla9 1y ago
    As I said, partial scrapes of small subsets over long time provide no real value for AI scrapers. Just an example: Brightdata linkedin database has 19 million entries. Linkedin has over 1 billion members. As I said, partial scrapes of small subsets over long time provide no real value for AI scrapers (repeating the main argument).