3 ms·

Everyone else in this thread has already explained that microsoft is not properly using robots.txt by having case insensitive urls, hence why these urls were in

Everyone else in this thread has already explained that microsoft is not properly using robots.txt by having case insensitive urls, hence why these urls were indexed.

benologist 16y ago
The Bing context might suck for you guys, but this is your problem - case insensitivity is every Windows server, not just the Bing website. Why would anyone hosted on Windows have to specify every possible spelling variation to keep search engines out of a folder or file? Here's another example: http://www.ifma.org/robots.txt http://www.ifma.org/robots.txt These guys are disallowing /pv/ http://www.google.com/search?sourceid=chrome&ie=UTF-8&q=site:ifma.org/pv/ http://www.google.com/search?sourceid=chrome&ie=UTF-8&#3... You guys are indexing /PV/
- andrewcooke 16y ago
  Maybe I'm misunderstanding your argument, but I think you're confusing the Windows file system with the URLs that a web service provides.
  - benologist 16y ago
    The problem exists when that web service is on Windows - ASP.NET, Cold Fusion, static html sites, probably a negligible percent of PHP sites etc - /pv is /PV is /pV is /Pv
jeroen 16y ago
"Be strict in what you emit, liberal in what you accept." Bing failed the first part, Google the second. Both should fix that.