4 ms·
It would help if Yelp allowed new, emerging search engines (potential google competitors) to crawl their site. But their robots.txt only allows Google, Bing and
by AznHisoka 9y ago
It would help if Yelp allowed new, emerging search engines (potential google competitors) to crawl their site. But their robots.txt only allows Google, Bing and a few others to crawl them (https://yelp.com/robots.txt https://yelp.com/robots.txt)
allow all crawlers and I would have at least a little sympathy for Yelp.