5 ms·
At blekko, we did not find this issue to be a significant one... almost everyone who banned our crawler was a crappy over-SEOed website.
by greglindahl 10y ago
At blekko, we did not find this issue to be a significant one... almost everyone who banned our crawler was a crappy over-SEOed website.
- AznHisoka 10y agohttps://www.linkedin.com/robots.txt https://www.linkedin.com/robots.txt https://yelp.com/robots.txt https://yelp.com/robots.txt There goes all Linkedin + Yelp content from your index.
- deleted 10y ago[deleted]
- betolink 10y agoWhat about https://www.facebook.com/robots.txt https://www.facebook.com/robots.txt ..and medium-sized/small sites are even worse. The irony of Facebook being a core part of all NSA surveillance programs and their terms of service including their "Automated Data Collection Terms" https://www.facebook.com/apps/site_scraping_tos_terms.php https://www.facebook.com/apps/site_scraping_tos_terms.php
- greglindahl 10y agoIf you surf LinkedIn logged out, you'll see that there isn't very much information available anyway. And there's no money in people search. Yelp was very responsive when blekko wrote them; as you can see ScoutJet has the same access as googlebot.