3 ms·
That's unfortunate. But understandable in a way. # robots.txt web.archive.org 2013-10-02 User-agent: * Disallow: / User-agent: ia_archiver
by emmelaich 9y ago
That's unfortunate. But understandable in a way.
# robots.txt web.archive.org 2013-10-02
User-agent: *
Disallow: /
User-agent: ia_archiver
Allow: /
- jxramos 9y agotouche, I don't suppose the old non commercial websites mentioned in the article suffer the same problem though right? Maybe an accidental robots.txt file was mistakenly left around?