5 ms·
Why not utf-8-sig, though? It handles optional BOMs. Had to fix a script last week that choked on it.
by a-french-anon 2y ago
Why not utf-8-sig, though? It handles optional BOMs. Had to fix a script last week that choked on it.
- otteromkram 2y agoOnly one script? Out of how many?
- hermitdev 2y agoNot the OP, but I see this pop up quite frequently in ETL, usually handling csv files.
- orf 2y agoBecause changing Python to silently prefixing all IO with an invisible BOM isn’t a good idea.
- int_19h 2y agoThe expectation isn't for it to generate BOM in the output, but to handle BOM gracefully when it occurs in the input.
- deleted 2y ago[deleted]
- shpx 2y ago> On encoding the utf-8-sig codec will write 0xef, 0xbb, 0xbf as the first three bytes to the file https://docs.python.org/3/library/codecs.html https://docs.python.org/3/library/codecs.html The codec you're imagining would also make reading a file and writing it back change the file if it contains a BOM.
- int_19h 2y agoIndeed it would, but since codecs are only used for files that are semantically text, and in such files BOM is basically a legacy no-op marker, it's not actually a problem. Naive code using text I/O APIs would also have this issue with line endings, for example, so there's precedent for not providing the perfect roundtrip experience (that's what bytes I/O is for).
- shellac 2y agoAt this point nothing ought to be inserting BOMs in utf-8. It's not recommended, and I think choking on it is reasonable behaviour these days.
- Athas 2y agoWhy were BOMs ever allowed for UTF-8?
- plorkyeran 2y agoWhen UTF-8 was still very much not the default encoding for text files it was useful to have a way to signal that a file was UTF-8 and not the local system encoding.
- josefx 2y agoSome editors used them to help detect UTF-8 encoded files. Since they are also valid zero length space characters they also served as a nice easter egg for people who ended up editing their linux shell scripts with a windows text editor.
- da_chicken 2y agoSome algorithms can operate much easier if they can assume that multibyte or variable byte characters don't exist. The BOM means that you don't have to scan the entire document to know if you can do that.
- account42 2y agoNo, a missing BOM does not guarantee that the file doesn't contain multi-byte code points and never has.
- da_chicken 2y agoThat isn't what I said. I said that if a BOM is present, then you explicitly know that multi-byte characters are possibly present. Therefore, if it's present you know that assuming that the Nth byte is the Nth code point is unsafe. The opposite is irrelevant. There's never any way to safely determine a text file's encoding if there is no BOM present.