3 ms·
archive.org obeys the robots.txt exclusion but archive.today doesn't. This means that many websites(like 4chan) cannot be archived with archive.org.
by deadalus 5y ago
archive.org obeys the robots.txt exclusion but archive.today doesn't. This means that many websites(like 4chan) cannot be archived with archive.org.
- oenetan 5y agoThat's true, so does webcitation.org, which doesn't obey robots.txt. 4chan has archives dedicated for that site anyway (Warosu, etc.).