5 ms·

Who gets to decide what is a good bot?

• tccole 3mo ago
  Me… obviously
• nullpoint420 3mo ago
  Cloudflare, apparently.
  • mark212 3mo ago
    no, their customers. Why do you assume that people who run websites are clueless and lack agency in this?
    • nullpoint420 3mo ago
      inb4 "we're updating our Terms of Service - all Cloudflare-based scrapers will now automatically be enrolled to scrape your cached websites" and they'll explain how it's great because it's only reading the cached versions, not hitting upstream.
• pythonaut_16 3mo ago
  Presumably the site owner with Cloudflare providing enforcement. Generally isn't a good bot one that respects robots.txt and is respectful of the site's resources by not being spammy?
• wnevets 3mo ago
  by checking which follow robots.txt?
• wmf 3mo ago
  According to their plan, the good bots pay for scraping.
• overfeed 3mo ago
  > Who gets to decide what is a good bot? Paying Cloudflare's tolls => good bot.