3 ms·

Getting a bit tangential: I've been trying Bing again recently out of curiosity, and at least for my usage I find it now pretty reasonable. A few years ago it w

Getting a bit tangential: I've been trying Bing again recently out of curiosity, and at least for my usage I find it now pretty reasonable. A few years ago it was barely usable, but now I'm fairly happy with it. It also feels slightly snappier to me than Google on loading results. The main problem is that a small but significant number of sites have a robots.txt that excludes all non-Google crawlers, making it impossible for other search engines to display equally good results in cases where that site has something relevant (unless they blatantly ignore robots.txt).

giancarlostoro 11y ago
So wait, who actually blocks Bing but not Google? That is just completely odd sounding to me... But Bing is definitely one engine I'm trying to switch to, it just isn't reliable when I'm looking up programming related queries sometimes. Then again I always know all the key words to throw at Google to find whatever I'm looking for. Googling is a (query) programming language by itself.
caseysoftware 11y ago
Why would someone do that? If a site owner wants traffic on their site - and most do - they want to make their site as easily discoverable as possible. That means being in all the search engines. Blocking one engine (or all except one) flies in the face of most business models. The only exception I can think of is Google properties. By blocking out competitors, they can try to worsen the experience on other engines. Like imagine a video search on Bing that didn't include YouTube? But even then, it seems like someone finding a Google property via Bing still furthers their interests.
- _delirium 11y ago
  Few deliberately block Bing specifically, it's just a sloppy attempt to block unwanted crawlers, like the billions of crawlers running on AWS and DigitalOcean boxes. A quick-but-not-great hack is to have an entry blocking * by default, and then specifically whitelist GoogleBot to avoid getting delisted from Google. You could also whitelist other search engines, but that requires you to know and care about them, and for a long period Google was the only engine people really cared about, since it drove >95% of organic search traffic. I do feel we're past the peak of that period, however, and site owners are now getting more cognizant of non-Google search engines having at least some value in driving traffic. Related, "Please Don't Block Everything but Googlebot in robots.txt": http://danluu.com/googlebot-monopoly/ http://danluu.com/googlebot-monopoly/
  - caseysoftware 11y ago
    So it's more out of ignorance than maliciousness. That makes sense. Odds are the more troublesome scrapers are probably ignoring robots.txt anyway, so I wonder how useful it is.
brobinson 11y ago
Bing's image and video search have been superior to Google's for over a year now, in my opinion. I use Bing as my default search engine, but it falls short of Google's results for a lot of programming-related queries. I still end up having to use Google a few times per day when I'm working.

giancarlostoro 11y ago

So wait, who actually blocks Bing but not Google? That is just completely odd sounding to me... But Bing is definitely one engine I'm trying to switch to, it just isn't reliable when I'm looking up programming related queries sometimes. Then again I always know all the key words to throw at Google to find whatever I'm looking for. Googling is a (query) programming language by itself.

caseysoftware 11y ago

Why would someone do that? If a site owner wants traffic on their site - and most do - they want to make their site as easily discoverable as possible. That means being in all the search engines. Blocking one engine (or all except one) flies in the face of most business models. The only exception I can think of is Google properties. By blocking out competitors, they can try to worsen the experience on other engines. Like imagine a video search on Bing that didn't include YouTube? But even then, it seems like someone finding a Google property via Bing still furthers their interests.

_delirium 11y ago
Few deliberately block Bing specifically, it's just a sloppy attempt to block unwanted crawlers, like the billions of crawlers running on AWS and DigitalOcean boxes. A quick-but-not-great hack is to have an entry blocking * by default, and then specifically whitelist GoogleBot to avoid getting delisted from Google. You could also whitelist other search engines, but that requires you to know and care about them, and for a long period Google was the only engine people really cared about, since it drove >95% of organic search traffic. I do feel we're past the peak of that period, however, and site owners are now getting more cognizant of non-Google search engines having at least some value in driving traffic. Related, "Please Don't Block Everything but Googlebot in robots.txt": http://danluu.com/googlebot-monopoly/ http://danluu.com/googlebot-monopoly/
- caseysoftware 11y ago
  So it's more out of ignorance than maliciousness. That makes sense. Odds are the more troublesome scrapers are probably ignoring robots.txt anyway, so I wonder how useful it is.

brobinson 11y ago

Bing's image and video search have been superior to Google's for over a year now, in my opinion. I use Bing as my default search engine, but it falls short of Google's results for a lot of programming-related queries. I still end up having to use Google a few times per day when I'm working.