3 ms·

Fortunately, they ignore robots.txt for .gov and .mil sites: https://twitter.com/internetarchive/status/824063784792657920 https://twitter.com/internetarchive/

by CM30 10y ago

Fortunately, they ignore robots.txt for .gov and .mil sites: https://twitter.com/internetarchive/status/824063784792657920 https://twitter.com/internetarchive/status/82406378479265792... So in this case, the Internet Archive should work fine.

astrodust 10y ago
That's a really sensible policy and an especially forward-thinking one.