4 ms·
Looking at their robots.txt, they allow Google but not DDG.
by cabalamat 9y ago
Looking at their robots.txt, they allow Google but not DDG.
- Ajedi32 9y agoHuh... Looks like they have a whitelist that allows Bing, Google, and Yandex (among others), but not DuckDuckGo: https://yelp.com/robots.txt https://yelp.com/robots.txt Does DuckDuckGo respect `robots.txt` files with onerous implicit restrictions like that? If I were them, I'd probably just write it so that if Googlebot is allowed then DuckDuckBot is too.