3 ms·
Wayback Machine does respect robots.txt, I think. I'm curious what happens if you lose control of your domain, though.
by mercer 9y ago
Wayback Machine does respect robots.txt, I think. I'm curious what happens if you lose control of your domain, though.
- kevingadd 9y agoIf you lose control the whole history of the site gets nuked by a new robots.txt. This has happened in a few notable cases fairly recently.