3 ms·
They give the Internet Archive the most access, but oddly go out of their way to block their TOS and privacy policy from them and not anyone else. Sneaky. It's
by mapleoin 13y ago
They give the Internet Archive the most access, but oddly go out of their way to block their TOS and privacy policy from them and not anyone else. Sneaky.
It's actually the opposite. They allow IA access to their terms and policy, but block everything else:
User-agent: ia_archiver
Allow: /about/privacy
Allow: /full_data_use_policy
Allow: /legal/terms
Allow: /policy.php
Disallow: /
- jonknee 13y agoDuh, all the others were Disallow statements and I just missed that the IA ones were Allow. Thanks for the tip.