3 ms·

what is the difference between a scraper or crawler and an "AI scraper/crawler"? And how does cloudflare differentiate between them?

• epanchin 2y ago
  The latter doesn’t request robots.txt
• twelve40 2y ago
  it's explained pretty well in their blog post in another thread. They have a whitelist of bots catergorized by type, and ban everything else. How easy it is to spoof (just the UA, or UA+IP whatever) is not mentioned.