4 ms·
https://www.linkedin.com/robots.txt https://www.linkedin.com/robots.txt https://yelp.com/robots.txt https://yelp.com/robots.txt There goes all Linkedin + Yelp
by AznHisoka 10y ago
https://www.linkedin.com/robots.txt https://www.linkedin.com/robots.txt
https://yelp.com/robots.txt https://yelp.com/robots.txt
There goes all Linkedin + Yelp content from your index.
- deleted 10y ago[deleted]
- betolink 10y agoWhat about https://www.facebook.com/robots.txt https://www.facebook.com/robots.txt ..and medium-sized/small sites are even worse. The irony of Facebook being a core part of all NSA surveillance programs and their terms of service including their "Automated Data Collection Terms" https://www.facebook.com/apps/site_scraping_tos_terms.php https://www.facebook.com/apps/site_scraping_tos_terms.php
- greglindahl 10y agoIf you surf LinkedIn logged out, you'll see that there isn't very much information available anyway. And there's no money in people search. Yelp was very responsive when blekko wrote them; as you can see ScoutJet has the same access as googlebot.