4 ms·

The two major issues in this article were: - Google can see and return links to pages without crawling them. I made a video and a blog post about this a while

The two major issues in this article were: - Google can see and return links to pages without crawling them. I made a video and a blog post about this a while ago: http://www.mattcutts.com/blog/robots-txt-remove-url/ http://www.mattcutts.com/blog/robots-txt-remove-url/ - URL paths are case sensitive. Bing blocks /search in its robots.txt but not /Search. That's how the /Search urls got crawled. In a later edit, the author suggests "It would be fairly trivial for bots to test if the server is IIS (if the server identifies itself as such of course) or to try to retrieve Robots.txt and robots.txt, if those come up as equal then the sever can be assumed to be case insensitive." The issue of case sensitivity in robots.txt is a long, very nuanced topic. Here's just one example to get you started: at least back in 2007 when we were talking about this amongst ourselves at Google, the web server for developer.apple.com was case-insensitive, but their robots.txt had lines like this: Disallow: /documentation/quicktime/ Disallow: /documentation/Quicktime/ Disallow: /documentation/QUICKTIME/ Disallow: /documentation/macosx/ Why would they do that? Apparently because Apple wanted the canonical link to be /documentation/QuickTime . Back then, at least 21M robots.txt files on the web had mixed-case paths. If Google started interpreting robots.txt files from servers that claimed to be IIS differently... well, I'll leave it as an exercise to the reader to come up with some of the unexpected bugs and behavior that could result. I know it's really tempting to write a headline like "Bing search results showing up in Google," but I wish the author had done more research instead of going for a gotcha. Any SEO worth his/her salt could have explained what was going on here.

mc32 16y ago
>"...but I wish the author had done more research instead of going for a gotcha." That's a bit of irony right there, isn't it? --wow, Matt, I guess it cuts both ways, eh?
- Matt_Cutts 16y ago
  Anything in particular you're talking about? I've said a lot of stuff online over the last 10 years. :)
  - mc32 16y ago
    It's no secret that a fair number of commenters throughout the controversy have sided with Google's or Bing's side. Some people have thought that the initial story came to a premature conclusion (where, incidentally, Bing cried foul). Now we have Jacques, it appears, arriving at premature conclusion as well and we have Google crying foul. To me, that's the essence of irony. I'm sure both Google Search and Bing have your reasons for the way the dirty laundry has been brought out in public.
  - Natsu 16y ago
    I don't know, but I would assume that they're one of the people who don't think that what Bing did is a big deal. If I understand the arguments properly, most either believe that the clickstream data belongs to the user (who gave it away), or they believe that you haven't actually proven that they're singling out Google in their clickstream data. Anyhow, I've already posted in this story how anyone who wants to could go do their own experiment (it appears that Bing's data isn't too hard to fake, that goes double for anyone who can reverse engineer the toolbar). Similarly, I'm not convinced that Bing doing this is going to ruin search any time soon, mostly because I can see spammers/blackhat SEO types renting botnets to feed Bing all the bogus clickstream data they want. It appears to be a simple http request with some time zones, link text, and an identifier or two that can be harvested from actual toolbars. If they don't bother to spam them, it's because they believe that Bing is irrelevant. And when I say "irrelevant" I mean "even less relevant than a low-traffic wiki for a free game that's currently under a massive assault from spambots." Which, incidentally, might be one good reason for your team to look more at keeping wiki-spam out of your index. Some spam results from that wiki (in spite of having rel=no-follow) were seen in Google's index and stayed there until the admins caught on and cleaned things up.

mc32 16y ago

>"...but I wish the author had done more research instead of going for a gotcha." That's a bit of irony right there, isn't it? --wow, Matt, I guess it cuts both ways, eh?

Matt_Cutts 16y ago
Anything in particular you're talking about? I've said a lot of stuff online over the last 10 years. :)
- mc32 16y ago
  It's no secret that a fair number of commenters throughout the controversy have sided with Google's or Bing's side. Some people have thought that the initial story came to a premature conclusion (where, incidentally, Bing cried foul). Now we have Jacques, it appears, arriving at premature conclusion as well and we have Google crying foul. To me, that's the essence of irony. I'm sure both Google Search and Bing have your reasons for the way the dirty laundry has been brought out in public.
- Natsu 16y ago
  I don't know, but I would assume that they're one of the people who don't think that what Bing did is a big deal. If I understand the arguments properly, most either believe that the clickstream data belongs to the user (who gave it away), or they believe that you haven't actually proven that they're singling out Google in their clickstream data. Anyhow, I've already posted in this story how anyone who wants to could go do their own experiment (it appears that Bing's data isn't too hard to fake, that goes double for anyone who can reverse engineer the toolbar). Similarly, I'm not convinced that Bing doing this is going to ruin search any time soon, mostly because I can see spammers/blackhat SEO types renting botnets to feed Bing all the bogus clickstream data they want. It appears to be a simple http request with some time zones, link text, and an identifier or two that can be harvested from actual toolbars. If they don't bother to spam them, it's because they believe that Bing is irrelevant. And when I say "irrelevant" I mean "even less relevant than a low-traffic wiki for a free game that's currently under a massive assault from spambots." Which, incidentally, might be one good reason for your team to look more at keeping wiki-spam out of your index. Some spam results from that wiki (in spite of having rel=no-follow) were seen in Google's index and stayed there until the admins caught on and cleaned things up.