4 ms·
Not only that, if they are archived at one point and later decide to be excluded then all of the archives for that domain are deleted retroactively. This means
by chongli 7y ago
Not only that, if they are archived at one point and later decide to be excluded then all of the archives for that domain are deleted retroactively. This means that a valuable site can be archived for years but if the domain lapses and gets bought up by somebody who decides to exclude the Wayback Machine then all of that precious content is destroyed even though the destroyer never owned it!
I feel like we’re in the Internet equivalent of the early years of Hollywood. We’ve lost so many precious early movies because there were no backups of the film.
- schoen 7y agoI don't think your interpretation of the latter case is quite right. In my understanding, the Archive is willing to disable display of the site content retroactively due to a new robots.txt, but they won't delete that content. It will still be present in their files.
- mc32 7y agoDeleted or not made available?
- TJSomething 7y agoThis policy was actually changed in 2017 to completely ignore robots.txt. [0] https://blog.archive.org/2017/04/17/robots-txt-meant-for-search-engines-dont-work-well-for-web-archives/ https://blog.archive.org/2017/04/17/robots-txt-meant-for-sea...
- ryanlol 7y agoIf the site goes down and the robots.txt becomes unavailable, the content will become available on internet archive.