4 ms·

Except archive.org doesn’t obey robots.txt files any more [1], and they also ignore requests to remove content. [1] https://blog.archive.org/2017/04/17/robots-

by wsh 8y ago

Except archive.org doesn’t obey robots.txt files any more [1], and they also ignore requests to remove content. [1] https://blog.archive.org/2017/04/17/robots-txt-meant-for-search-engines-dont-work-well-for-web-archives/ https://blog.archive.org/2017/04/17/robots-txt-meant-for-sea...

trevyn 8y ago
I doubt they ignore DMCA requests.
superkuh 8y ago
Archive.org is not the archive team.
0815test 8y ago
They don't obey robots.txt files posted after-the-fact by domain hoarders that have zilch to do with the original content. This is entirely proper on their part.