4 ms·

Banning these crawlers is not the answer. Poisoning the well is.

• Aachen 2y ago
  There's one thing worse than a block page being shown to humans because an algorithm decided they're a bot: purposefully false information being shown to humans It's also just wasting more of the planet's resources as compared to blocking And more effort, with as only upside that it's not immediately obvious to the bot that it is being blocked so it'll suck in more of your pages I understand that people are exploring options but I wouldn't label this as a solution to anything as of today's state of the art
  • xeeeeeeeeeeenu 2y ago
    There are bot identification methods that may produce lots of false negatives, but zero false positives. Simple example: no legitimate user has "GPTBot" in their user agent string.
    • Aachen 2y ago
      Ah yeah, that's a good example It'll still trickle down to people using the system and waste people's time (from development to the people working to produce all the things used by these companies and mopping up the impact this has to eventually the users) but that would definitely resolve one of my concerns