5 ms·
Without. UTF-8 is such a distinctive pattern that if text with high bits set matches UTF-8, it's almost certainly UTF-8. There's no need for a BOM to tell you i
by jcranmer 10y ago
Without. UTF-8 is such a distinctive pattern that if text with high bits set matches UTF-8, it's almost certainly UTF-8. There's no need for a BOM to tell you it's UTF-8 (looking at you, Windows), and it can easily confuse software instead.
- umanwizard 10y agoHuh? What would a BOM in UTF-8 even do? 1-byte objects can't have an internal byte ordering.
- the_mitsuhiko 10y agoIn UTF-8 a BOM can be placed to support round tripping the information with UTF-16.
- umanwizard 10y agoSo what order do you put the BOM in? Does it not even matter?
- Avernar 10y agoThe unicode BOM is code point U+FEFF. The process of encoding it determines the order. Encoded to UTF-8 it becomes EF BB BF. Encoding to UTF-16 big endian it will become FE FF. Encoding it to UTF-16 little endian it becomes FF FE. Converting it back from UTF-8 always gives you U+FEFF since UTF-8 doesn't care about endianess. Converting it back from UTF-16 using the correct endianess gives you U+FEFF. Converting it using the wrong endianess gives you U+FFFE which is defined by unicode as a "non character" that should never appear in text.
- umanwizard 10y agoMakes sense, thanks :)
- jcranmer 10y agoMS popularized the idea of adding the UTF-16 BOM into UTF-8 to distinguish between UTF-8 text files and Windows code page files, or what they called "Unicode" and "ANSI." There's (nearly?) unanimous agreement among everyone else that BOMs in UTF-8 text are really stupid. Note that the "BOM" in this case means storing the U+FEFF character in UTF-8 form (just as UTF-16 stores it in the appropriate endianness). This means that the result would be EF BB BF.
- Const-me 10y agoNot everything is a web request or response that have a “content-encoding” header transmitted somewhere out of band. The BOM allows to distinguish a byte stream between non-Unicode, UTF8, UTF16 and UTF32. Like it or not, but it's part of the standard: http://unicode.org/faq/utf_bom.html#BOM http://unicode.org/faq/utf_bom.html#BOM
- Avernar 10y agoFrom section 2.6 in the standard: "Use of a BOM is neither required nor recommended for UTF-8, but may be encountered in contexts where UTF-8 data is converted from other encoding forms that use a BOM or where the BOM is used as a UTF-8 signature." Yes, it can be used to distinguish a UTF-8 stream but it's not recommended. One issue is you can't tell if the BOM is not valid text in some other non-unicode encoding. I'm curious where you've encountered missing content-encoding headers or other OOB indicators where it wasn't because of programmer error or laziness.
- Const-me 10y ago> it can be used to distinguish a UTF-8 stream but it's not recommended If a specification says “something may be encountered”, for me, when I write my software, it means I must support that thing. Otherwise, the software won’t conform to the spec. > I'm curious where you've encountered missing content-encoding headers or other OOB indicators where it wasn't because of programmer error or laziness. Everywhere. Most filesystems don’t have encoding headers for their text files. Most databases don’t have headers for their blob columns. Only web that has encoding headers.
- Avernar 10y agoThere's a difference between what a program should accept as input and what it should generate as output. The standard just says to expect a BOM on input and suggests not generating on output. In other words "A UTF-8 BOM is a bad idea but some yutz out there stated doing it so we should ignore it on input". Someone else mentioned that the yutz was Microsoft. I misread what you wrote about where you saw no idication that it was UTF-8. You were talking about places other than the web. BOM for UTF-8 text files seems to be a Microsoft thing. Everyone else just defaults to UTF-8. But you can't be sure that it's a UTF-8 BOM or some other encoding. Most editors let the user overide what it is. Why would you store text in a blob column? If a database can't handle UTF-8 in it's text column it needs to be fixed (or taken out back an shot).