3 ms·
FWIW, here's Facebook's robots.txt file. While this was a firehose agreement and not subject to robots.txt, it is an interesting look: https://facebook.com/rob
by jonknee 13y ago
FWIW, here's Facebook's robots.txt file. While this was a firehose agreement and not subject to robots.txt, it is an interesting look:
https://facebook.com/robots.txt https://facebook.com/robots.txt
They give the Internet Archive the most access, but oddly go out of their way to block their TOS and privacy policy from them and not anyone else. Sneaky.
Update: scratch that, I missed that they were Allow statements for ia_archiver. The Internet Archive has by far the least access.
- mapleoin 13y agoThey give the Internet Archive the most access, but oddly go out of their way to block their TOS and privacy policy from them and not anyone else. Sneaky. It's actually the opposite. They allow IA access to their terms and policy, but block everything else: User-agent: ia_archiver Allow: /about/privacy Allow: /full_data_use_policy Allow: /legal/terms Allow: /policy.php Disallow: /
- jonknee 13y agoDuh, all the others were Disallow statements and I just missed that the IA ones were Allow. Thanks for the tip.
- csharp_gooru 13y agoUser-agent: ia_archiver Allow: /about/privacy Allow: /full_data_use_policy Allow: /legal/terms Allow: /policy.php Disallow: /