Google used to have (must check the code) have problems with robots.txt files that had a BOM - If the First line was Disallow The D got mangled and that disal

Google used to have (must check the code) have problems with robots.txt files that had a BOM - If the First line was Disallow The D got mangled and that disallow directive got ignored.