3 ms·
Huh... Looks like they have a whitelist that allows Bing, Google, and Yandex (among others), but not DuckDuckGo: https://yelp.com/robots.txt https://yelp.com/ro
by Ajedi32 9y ago
Huh... Looks like they have a whitelist that allows Bing, Google, and Yandex (among others), but not DuckDuckGo: https://yelp.com/robots.txt https://yelp.com/robots.txt
Does DuckDuckGo respect `robots.txt` files with onerous implicit restrictions like that? If I were them, I'd probably just write it so that if Googlebot is allowed then DuckDuckBot is too.