> And robots.txt has been a thing since forever. Unfortunately, both archive.org and the affiliated Archive Team now intentionally ignore robots.txt files. Ar

> And robots.txt has been a thing since forever. Unfortunately, both archive.org and the affiliated Archive Team now intentionally ignore robots.txt files. Archive Team, in particular, wrote a very irritating and condescending screed called "ROBOTS.TXT IS A SUICIDE NOTE" [1]. Apparently, they don't understand that not everyone wants their stuff backed up forever and they treat anyone who disagrees with them as a moron. And that manifesto wasn't written by some random team member, it was from Jason Scott himself. I know that Archive Team isn't officially part of IA, but they are tightly integrated with their archiving operations. It's a bad look. [1] http://wiki.archiveteam.org/index.php?title=Robots.txt http://wiki.archiveteam.org/index.php?title=Robots.txt