"Archiveteam projects are different because we run closely-monitored, highly targeted crawls." Maybe, but does that make robots.txt useless? I don't think so.

"Archiveteam projects are different because we run closely-monitored, highly targeted crawls." Maybe, but does that make robots.txt useless? I don't think so. Your app may do the right thing but who's to say another app isn't? Why should someone ignore a tool because Archiveteam claims to do the right thing? The file has more uses than just blocking a particular URL. What about adding a crawler delay for a site on a shared server that gets slammed sometimes? Or what about blocking a particular user-agent that misbehaves? If someone shoots themselves in the foot with their robots.txt, it's their foot and their gun.