3 ms·
you'd think it would at least come up in the internet archive if not anywhere else.
by jxramos 9y ago
you'd think it would at least come up in the internet archive if not anywhere else.
- paulcole 9y agohttps://web.archive.org/robots.txt https://web.archive.org/robots.txt
- emmelaich 9y agoThat's unfortunate. But understandable in a way. # robots.txt web.archive.org 2013-10-02 User-agent: * Disallow: / User-agent: ia_archiver Allow: /
- jxramos 9y agotouche, I don't suppose the old non commercial websites mentioned in the article suffer the same problem though right? Maybe an accidental robots.txt file was mistakenly left around?