2 ms·

https://blog.cloudflare.com/cloudflares-always-online-and-the-internet-archive-team-up-to-fight-origin-errors/ https://blog.cloudflare.com/cloudflares-always-on

https://blog.cloudflare.com/cloudflares-always-online-and-the-internet-archive-team-up-to-fight-origin-errors/ https://blog.cloudflare.com/cloudflares-always-online-and-th... We tell archive.org about the URI, they crawl it. They handle robots.txt.

AnonHP 6y ago
archive.org doesn't handle robots.txt in any meaningful way (see my comment above at https://news.ycombinator.com/item?id=24516875 https://news.ycombinator.com/item?id=24516875 ). If that's changed recently, I'd like to know more.