3 ms·
Alright, so this is probably the place to ask this. Why is there a juxtaposition between what people want and between what people are willing to give? Specifi
by TheLoneWolfling 12y ago
Alright, so this is probably the place to ask this.
Why is there a juxtaposition between what people want and between what people are willing to give?
Specifically: why do so many people ban web crawlers other than the "large" ones, and then wonder why the quality of other search engines are lacking?
- Someone1234 12y ago> Specifically: why do so many people ban web crawlers other than the "large" ones, and then wonder why the quality of other search engines are lacking? Not to be difficult but is this a real problem? Do a lot of sites ban DDG's crawler and then complain about the low quality experience? Can you name a single example?
- maximumoverload 12y agoAs far as I know, DDG doesn't have a crawler and just uses other search APIs (Google, Bing). edit: OK, I was wrong - https://duckduckgo.com/duckduckbot https://duckduckgo.com/duckduckbot
- lxle 12y ago2nd wrong, DDG doesn't use Google for search results at all. Google essentially killed Scroogle that did exactly what startpage is doing now by banning their search proxy, which makes one wonder, what is the special relationship that startpage has with google that scroogle did not.
- maximumoverload 12y agoOh. In that case I got DDG and StartPage confused. Maybe StartPage still displays Google's ads? They display some ads.
- hyperpape 12y agoI wonder if there's really much overlap between the groups. Maybe the difference is that you're just describing different people who prioritize different things?
- korzun 12y ago> Specifically: why do so many people ban web crawlers other than the "large" ones, Never heard of this. Third party crawlers usually get banned because they are poorly made (get stuck) or disobey robots.txt.