4 ms·
How do you handle robots.txt? The previous incarnation of Always Online didn't care about robots.txt, while archive.org does.
by jorams 6y ago
How do you handle robots.txt? The previous incarnation of Always Online didn't care about robots.txt, while archive.org does.
- jgrahamc 6y agohttps://blog.cloudflare.com/cloudflares-always-online-and-the-internet-archive-team-up-to-fight-origin-errors/ https://blog.cloudflare.com/cloudflares-always-online-and-th... We tell archive.org about the URI, they crawl it. They handle robots.txt.
- AnonHP 6y agoarchive.org doesn't handle robots.txt in any meaningful way (see my comment above at https://news.ycombinator.com/item?id=24516875 https://news.ycombinator.com/item?id=24516875 ). If that's changed recently, I'd like to know more.
- AnonHP 6y agoNote that archive.org stopped respecting robots.txt since 2017. [1] In my experience, the site owner must email archive.org support to be excluded from its crawler and archiving. [1]: https://boingboing.net/2017/04/22/internet-archive-to-ignore-rob.html https://boingboing.net/2017/04/22/internet-archive-to-ignore...