3 ms·

even worse, detect aibot dissrespecting robot.txt, and redirect it to a poison page

• akira2501 3y ago
  You can also just do that for no particularly good reason. Language models aren't going to go anywhere particularly worthwhile if this is their primary mode of data gathering.
• hinkley 3y ago
  I needed to test whether an app could decompress inputs over 2GB properly. I was able to create a zip file that was about 45k to do it. I think the record is something like 5k, but 50k fits just fine in a repository. Set up direct Gzip sendfile, and that’ll keep them busy for a while. If I’m paying for servers by the hour and half of them are run just to keep the bots happy, I’m going to have opinions about who is allowed to visit my pages and for what reason. Mooches don’t have a moral leg to stand on.