3 ms·
As per that post, they only ignored robots.txt for .gov and .mil sites.
by icebraining 9y ago
As per that post, they only ignored robots.txt for .gov and .mil sites.
- klez 9y agoYes, but it also says > We are now looking to do this more broadly. That's the part I'm asking about.
- icebraining 9y agoRight, but it doesn't mean they reverted it, they are probably still looking into it.
- aero-002 9y agoIA disallow in robots.txt will still block archive.org, the blog post was about ignoring parts that were meant for search engines.