3 ms·
Hmm, all they got to do is have a dynamic robots.txt that forbids wayback from the deleted articles, and they'll remove the workaround even. yes?
by jrochkind1 9y ago
Hmm, all they got to do is have a dynamic robots.txt that forbids wayback from the deleted articles, and they'll remove the workaround even. yes?
- sp332 9y agoSure, but they'd have to create a list of every deleted article which seems like it would be pretty long.
- jrochkind1 9y agoI assume their software is quite capable of automatically creating such a list to include in robots.txt, automatically generated.
- Exuma 9y agoOnce it's stored I imagine they don't need to even scrape the page again, so robots.txt wouldn't do anything.
- deleted 9y ago[deleted]
- dragonwriter 9y agoInternet archive does rescrape periodically, and it removes archived pages based on the current robots.txt. This behavior is documented behavior of the archive that goes beyond the normal conventions of robots.txt.
- LinuxBender 9y agoI would add, the content itself is not removed. They only stop displaying it whilst the robots.txt says not to. If they can not reach your robots.txt, the content comes back as I have experienced multiple times.