4 ms·
They seem to have had this problem in the past and decided to skirt around it by ignoring robots.txt a year ago[0]. Does anybody know what happened to revert th
by klez 9y ago
They seem to have had this problem in the past and decided to skirt around it by ignoring robots.txt a year ago[0]. Does anybody know what happened to revert this decision?
[0]https://blog.archive.org/2017/04/17/robots-txt-meant-for-search-engines-dont-work-well-for-web-archives/ https://blog.archive.org/2017/04/17/robots-txt-meant-for-sea...
- icebraining 9y agoAs per that post, they only ignored robots.txt for .gov and .mil sites.
- klez 9y agoYes, but it also says > We are now looking to do this more broadly. That's the part I'm asking about.
- icebraining 9y agoRight, but it doesn't mean they reverted it, they are probably still looking into it.
- aero-002 9y agoIA disallow in robots.txt will still block archive.org, the blog post was about ignoring parts that were meant for search engines.