3 ms·
Maybe if they hadn't retroactively applied robot.txt, I'd consider it. Losing decades of a website because they updated their current terms is annoying and has
by justherefortart 9y ago
Maybe if they hadn't retroactively applied robot.txt, I'd consider it. Losing decades of a website because they updated their current terms is annoying and has made the site worthless to me.
- pfg 9y agoAs I understand it they're in the process of changing this behaviour[1]. [1]: https://blog.archive.org/2017/04/17/robots-txt-meant-for-search-engines-dont-work-well-for-web-archives/ https://blog.archive.org/2017/04/17/robots-txt-meant-for-sea...