3 ms·
> Specifically: why do so many people ban web crawlers other than the "large" ones, and then wonder why the quality of other search engines are lacking? Not to
by Someone1234 12y ago
> Specifically: why do so many people ban web crawlers other than the "large" ones, and then wonder why the quality of other search engines are lacking?
Not to be difficult but is this a real problem? Do a lot of sites ban DDG's crawler and then complain about the low quality experience? Can you name a single example?
- maximumoverload 12y agoAs far as I know, DDG doesn't have a crawler and just uses other search APIs (Google, Bing). edit: OK, I was wrong - https://duckduckgo.com/duckduckbot https://duckduckgo.com/duckduckbot
- lxle 12y ago2nd wrong, DDG doesn't use Google for search results at all. Google essentially killed Scroogle that did exactly what startpage is doing now by banning their search proxy, which makes one wonder, what is the special relationship that startpage has with google that scroogle did not.
- maximumoverload 12y agoOh. In that case I got DDG and StartPage confused. Maybe StartPage still displays Google's ads? They display some ads.