4 ms·
It's worth noting that in this case "blocking" means "asking nicely for it not to index them" - so how effective this is depends on how well behaved the bots ar
by entuno 3y ago
It's worth noting that in this case "blocking" means "asking nicely for it not to index them" - so how effective this is depends on how well behaved the bots are.
There is a danger though if certain types of sites are more likely to block GPTBot than others, because that would end up skewing the data set that it trains off, which could have longer term impacts on all the content generated with it. If all the good quality sites block it and the sites full of AI generated junk don't, then that sounds like a downward spiral.
- rolph 3y agoeven worse, detect aibot dissrespecting robot.txt, and redirect it to a poison page
- akira2501 3y agoYou can also just do that for no particularly good reason. Language models aren't going to go anywhere particularly worthwhile if this is their primary mode of data gathering.
- hinkley 3y agoI needed to test whether an app could decompress inputs over 2GB properly. I was able to create a zip file that was about 45k to do it. I think the record is something like 5k, but 50k fits just fine in a repository. Set up direct Gzip sendfile, and that’ll keep them busy for a while. If I’m paying for servers by the hour and half of them are run just to keep the bots happy, I’m going to have opinions about who is allowed to visit my pages and for what reason. Mooches don’t have a moral leg to stand on.