This is a huge problem. Sites like NASA's NTRS are retroactively blocked.[1] It's not clear which user agents one must allow in robots.txt. NTRS allows archive.

This is a huge problem. Sites like NASA's NTRS are retroactively blocked.[1] It's not clear which user agents one must allow in robots.txt. NTRS allows archive.org_bot, but apparently ia_archiver is also needed. At some point the allow directive in the NTRS robots.txt[2] no longer matched, nuking all historical data. 1. See http://web.archive.org/web/20121029225832/http://ntrs.nasa.gov/archive/nasa/casi.ntrs.nasa.gov/19770085619_1977085619.pdf http://web.archive.org/web/20121029225832/http://ntrs.nasa.g... for an example. 2. http://ntrs.nasa.gov/robots.txt http://ntrs.nasa.gov/robots.txt