3 ms·
https://web.archive.org/robots.txt https://web.archive.org/robots.txt
by paulcole 9y ago
https://web.archive.org/robots.txt https://web.archive.org/robots.txt
- emmelaich 9y agoThat's unfortunate. But understandable in a way. # robots.txt web.archive.org 2013-10-02 User-agent: * Disallow: / User-agent: ia_archiver Allow: /
- jxramos 9y agotouche, I don't suppose the old non commercial websites mentioned in the article suffer the same problem though right? Maybe an accidental robots.txt file was mistakenly left around?