3 ms·
Surprisingly interesting post that goes into history of robots.txt and details how it is not, in fact, a W3 standard or legal requirement
by tomcam 9y ago
Surprisingly interesting post that goes into history of robots.txt and details how it is not, in fact, a W3 standard or legal requirement
- walshemj 9y agodoes seem obsessed with the non standard crawl delay interestingly it doesn't mention those robots.txt files that have a BOM which can stop them working - which is why best practice is to have a comment of a blank line at the top of your robots.txt
- greglindahl 9y agoThat's a best practice for everyone?
- walshemj 9y agoyep if your first Line is a disallow and you have a text file with a BOM it will be ignored.
- greglindahl 9y agoI was more thinking that most webmasters don't use editors and OSes where a BOM might be emitted.