2 ms·
Wouldn't then the best logic be to finegrain this system a bit more? So blocking the archive blocks new content added after robots.txt is made available, and a
by CM30 10y ago
Wouldn't then the best logic be to finegrain this system a bit more?
So blocking the archive blocks new content added after robots.txt is made available, and adding a certain few lines in the robots.txt file explicitly hides older content on the domain in the archive.
That way, everyone wins. Sites that really want to remove content for some insane reason can do so, older sites usually aren't lost if the domain expires and domain holding page owners/cybersquatters don't accidentally cause older content to be hidden (since hey, they don't really want to block it, just stop their holding page from being archived).