4 ms·

This looks like microsoft is assuming robots.txt is case insensitive?

• ajays 16y ago
  It is. Only domain names are case-insensitive. This document explains how Google handles robots.txt : http://code.google.com/web/controlcrawlindex/docs/robots_txt.html http://code.google.com/web/controlcrawlindex/docs/robots_txt...
  • deleted 16y ago
    [deleted]
  • Natsu 16y ago
    Are we reading the same file? Under where it describes matching paths (so replace /fish and /Fish with /search and /Search if you like): ==Example path matches== [path] /fish Matches: /fish /fish.html /fish/salmon.html /fishheads /fishheads/yummy.html /fish.php?id=anything Does not match: /Fish.asp /catfish /?id=fish Comments: Note the case-sensitive matching.