3 ms·
Handling of robots.txt files with BOM is trivial but google does not do that so your robots.txt file my not work as expected.
by walshemj 9y ago
Handling of robots.txt files with BOM is trivial but google does not do that so your robots.txt file my not work as expected.
- brohee 9y agoFrom https://developers.google.com/search/reference/robots_txt?hl=en https://developers.google.com/search/reference/robots_txt?hl... "An optional Unicode BOM (byte order mark) at the beginning of the robots.txt file is ignored." At least they document it.