3 ms·
> Cloudflare can always spot crawlers I mention in the post a scraping service that Cloudflare isn't spotting: https://www.scrapingbee.com/blog/how-to-bypass-c
by mmarian 1y ago
> Cloudflare can always spot crawlers
I mention in the post a scraping service that Cloudflare isn't spotting: https://www.scrapingbee.com/blog/how-to-bypass-cloudflare-antibot-protection-at-scale/ https://www.scrapingbee.com/blog/how-to-bypass-cloudflare-an...
Plenty of open-source ones as well that could bypass, eg maybe this one that came up in search https://github.com/VeNoMouS/cloudscraper https://github.com/VeNoMouS/cloudscraper Combine with residential proxies and you're just not going to find them.
> Comprehensive crawl of LinkedIn, FB, instagram, IMDB, Amazon, would be worth a lot.
Just from a quick Google search:
- LinkedIn: https://brightdata.com/products/datasets/linkedin https://brightdata.com/products/datasets/linkedin
- Amazon: https://www.junglescout.com/features/product-database/ https://www.junglescout.com/features/product-database/
- nabla9 1y agoAs I said, partial scrapes of small subsets over long time provide no real value for AI scrapers. Just an example: Brightdata linkedin database has 19 million entries. Linkedin has over 1 billion members. As I said, partial scrapes of small subsets over long time provide no real value for AI scrapers (repeating the main argument).