4 ms·
Somewhat Unrelated. I noticed that the wayback machine no longer lists historical sites if the latest/last revision of robots.txt denies access. Has anyone el
by neoCrimeLabs 9y ago
Somewhat Unrelated.
I noticed that the wayback machine no longer lists historical sites if the latest/last revision of robots.txt denies access. Has anyone else experienced this?
In the late-90's I helped build one of the first fortune-500 e-commerce web sites. The website was shutdown years ago, but it view viewable on the wayback machine as recently as a year ago. The company in question put a deny-all robots.txt on the domain, and now none of the history is viewable.
It's a shame -- used to use that website (and an easteregg with my name on it) as proof of experience.
- biofox 9y agoApplying denials retroactively makes no sense whatsoever -- it defeats the whole point of archiving. Do you know if archive.org still has a copy?