4 ms·
I notice they have updated their robots.txt to only allow user agents they have approved. http://www.facebook.com/apps/site_scraping_tos.php http://www.faceboo
by xd 16y ago
I notice they have updated their robots.txt to only allow user agents they have approved.
http://www.facebook.com/apps/site_scraping_tos.php http://www.facebook.com/apps/site_scraping_tos.php
- historical 16y agoI noticed that a while back when it had the effect of removing thefacebook.com from the Wayback Machine. There used to be some fascinating reading in their old TOS and Privacy Policy. Wish I'd kept a copy.