3 ms·
Don't know if Yelp does allow any search engines to scrape, but if they'd allow duckduckgo to scrape and help build some marginal competition against Google, it
by ballenf 9y ago
Don't know if Yelp does allow any search engines to scrape, but if they'd allow duckduckgo to scrape and help build some marginal competition against Google, it might give them more leverage overall.
- cabalamat 9y agoLooking at their robots.txt, they allow Google but not DDG.
- Ajedi32 9y agoHuh... Looks like they have a whitelist that allows Bing, Google, and Yandex (among others), but not DuckDuckGo: https://yelp.com/robots.txt https://yelp.com/robots.txt Does DuckDuckGo respect `robots.txt` files with onerous implicit restrictions like that? If I were them, I'd probably just write it so that if Googlebot is allowed then DuckDuckBot is too.