4 ms·
Isn't this a problem with the archiving logic? Why on Earth would one apply robots.txt rules retroactively?
by Walf 10y ago
Isn't this a problem with the archiving logic? Why on Earth would one apply robots.txt rules retroactively?
- deleted 10y ago[deleted]
- detaro 10y agoThey need some way for people to easily hide archived content, since they have no clear right to publish their copy. You can send them an e-mail instead, but those take comparatively more resources to process and robots.txt at least sort-of indicates that you actually are in a position where you can demand the removal and can be processed automatically.
- CM30 10y agoWouldn't then the best logic be to finegrain this system a bit more? So blocking the archive blocks new content added after robots.txt is made available, and adding a certain few lines in the robots.txt file explicitly hides older content on the domain in the archive. That way, everyone wins. Sites that really want to remove content for some insane reason can do so, older sites usually aren't lost if the domain expires and domain holding page owners/cybersquatters don't accidentally cause older content to be hidden (since hey, they don't really want to block it, just stop their holding page from being archived).