What about https://www.facebook.com/robots.txt https://www.facebook.com/robots.txt ..and medium-sized/small sites are even worse. The irony of Facebook being

What about https://www.facebook.com/robots.txt https://www.facebook.com/robots.txt ..and medium-sized/small sites are even worse. The irony of Facebook being a core part of all NSA surveillance programs and their terms of service including their "Automated Data Collection Terms" https://www.facebook.com/apps/site_scraping_tos_terms.php https://www.facebook.com/apps/site_scraping_tos_terms.php