2 ms·
Putting the article URL in robots.txt stops the Wayback machine from displaying it - that's why they do it, just in case the Internet Archive happened to have a
by jamesbrownuhh 12y ago
Putting the article URL in robots.txt stops the Wayback machine from displaying it - that's why they do it, just in case the Internet Archive happened to have a cached copy of the article.
- aw3c2 12y agoAnd that's an attempt to re-write history or at least remove something from it.