3 ms·
One thing they mention in the article is monitoring the behavior as the irrelevant labyrinth of data is navigated in order to generate improved bot detection al
by hoherd 2y ago
One thing they mention in the article is monitoring the behavior as the irrelevant labyrinth of data is navigated in order to generate improved bot detection algorithms. Since the data they are serving is irrelevant to the expected page content, it's unlikely that a human would navigate very far into the labyrinth.
Also, it appears that it's not implemented through hidden links, but entirely different page content:
> rather than blocking the request, we will link to a series of AI-generated pages that are convincing enough to entice a crawler to traverse them
- pests 2y agoI think your last quote is incorrect, at least partially. I'm not sure they replace the entire target page: > This pre-generated content is seamlessly integrated as hidden links on existing pages via our custom HTML transformation process, without disrupting the original structure or content of the page. > We also ensured that these links remain invisible to human visitors through carefully implemented attributes and styling. To further minimize the impact to regular visitors, we ensured that these links are presented only to suspected AI scrapers, while allowing legitimate users and verified crawlers to browse normally.
- metadat 2y agoHow does a crawler become verified and blessed by Cloudflare?
- pests 2y agoI’m not involved with Cloudflare but IMO they probably calculate some score based on: honesty (not trying to hide user agent, etc), respecting robots.txt, proper crawl rate, etc.
- hoherd 2y agoGood catch, I missed that part completely. I'm heading into a long airport layover so I should have time to reread the full article.