2 ms·Hmm so why cant their crawler parse a robots.txt file with a BOM ?by C1sc0cat 8y agoHmm so why cant their crawler parse a robots.txt file with a BOM ?