3 ms·

Robots.txt is a joke, use Cloudflare's Block AI Bots feature if you are using Cloudflare.

• graeme 2y ago
  On this point, if you turn on bot fight mode it also says it blocks verified bots. But, bot fight mode says "there is a newer version of this setting" however it does not link to it. Anyone have any insight on the blocked verified bots or the supposed new version?
• userbinator 2y ago
  More precisely described as "block non-sanctioned user agents". Using that feature will ensure I never visit your site again.
  • aetherspawn 2y ago
    It's a free wiki so not visiting is a right you have, sure .. but also OP shouldn't have to spend loads of $ to host a free wiki and at the same time support a very bespoke thing such as non standard user agents.
    • zaphodias 2y ago
      > is a right I believe they're saying that cloudflare will block them just for using a blacklisted client, even if they're legit users and not bots
      • userbinator 2y ago
        Not even that; it's a whitelist, not a blacklist, and the only clients whitelisted are essentially those of Big Browser.