3 ms·
Archive projects usually listen to robots.txt
by homero 10y ago
Archive projects usually listen to robots.txt
- unimpressive 10y agoArchiveTeam on the other hand is a bit more aggressive: http://www.archiveteam.org/index.php?title=Main_Page http://www.archiveteam.org/index.php?title=Main_Page Though they generally do last minute emergency backups of closing sites, so this probably isn't a job for them.