4 ms·
Wow, all the US Department of State files have just gone missing from archive.org. The servers hosting those files are conveniently down.
by neil_s 11y ago
Wow, all the US Department of State files have just gone missing from archive.org. The servers hosting those files are conveniently down.
- benyami 11y agoCheck out the Internet Archive FAQ on how to remove a document from their archives. https://archive.org/about/exclude.php https://archive.org/about/exclude.php It looks like they used robots.txt to do that.
- neil_s 11y agoHuh, so the wild-card user-agent will block not just searchbots, but also archivebots. Wonder how OP managed to get screenshots of archive.org having archives available for those documents.
- maxmcd 11y agoI have been able to view multiple pdfs and view the page screenshotted by the author.
- kjell 11y agoThey're there, at least the two I looked at. https://web.archive.org/web/20130413152316/http://www.state.gov/documents/organization/193775.pdf https://web.archive.org/web/20130413152316/http://www.state.... Each line is missing `/documents` in the snippet of the `robots.txt`