3 ms·
What is the value of processing robots.txt in a case-sensitive way? If urls are to have different status when considering case change, the the site structure is
by xilun0 16y ago
What is the value of processing robots.txt in a case-sensitive way? If urls are to have different status when considering case change, the the site structure is just broken. Plus considering robots.txt in case-sensitive way has already result in lots of errors, this one included, and will result in even more in the future. Plus HTTP is not mandated to use case-sensitive URL (though it's recommended).
I can't think of any argument of why robots.txt should be processed in a case-sensitive way (i mean for a good reason -- obviously search engine have a very "good" incentive to handle it that way: the possibility to cheat and index more than they should, with an excuse when they are caught), on the other hand i can think of many for case-insensitive processing...
- moultano 16y agoWebmaster's shoot themselves in the foot a lot. "My site isn't showing up in google" is a frequent complaint on webmaster help forums, and typically the problem is robots.txt or meta noindex. From that standpoint, since most sites do want to be indexed, it makes sense to follow the standard as strictly as possible. It should be hard to remove your site by accident, which case insensitivity would make somewhat easier. Google states explicitly in their robots.txt policies that it is handled in a case-sensitive way.