4 ms·
Check out the Internet Archive FAQ on how to remove a document from their archives. https://archive.org/about/exclude.php https://archive.org/about/exclude.php
by benyami 11y ago
Check out the Internet Archive FAQ on how to remove a document from their archives. https://archive.org/about/exclude.php https://archive.org/about/exclude.php
It looks like they used robots.txt to do that.
- neil_s 11y agoHuh, so the wild-card user-agent will block not just searchbots, but also archivebots. Wonder how OP managed to get screenshots of archive.org having archives available for those documents.