4 ms·

Is that "UTF-8 with BOM" or "UTF-8 without BOM"? - because I've had lots of trouble with things supporting only one of those two options.

foobarrio 9y ago
Wait there's a UTF-8 with BOM? Why do you need a byte order mark for utf-8? The stream "unit" is a byte. I thought only utf-16 and above had an optional BOM?
- grzm 9y ago
  They're likely referring to this behavior on Windows: > "Many Windows programs (including Windows Notepad) add the bytes 0xEF, 0xBB, 0xBF at the start of any document saved as UTF-8. This is the UTF-8 encoding of the Unicode byte order mark (BOM), and is commonly referred to as a UTF-8 BOM, even though it is not relevant to byte order. A BOM can also appear if another encoding with a BOM is translated to UTF-8 without stripping it. Software that is not aware of multibyte encodings will display the BOM as three garbage characters at the start of the document, e.g. "ï»¿" in software interpreting the document as ISO 8859-1 or Windows-1252 or "∩╗┐" if interpreted as code page 437, a default for certain older Windows console applications." > "The Unicode Standard neither requires nor recommends the use of the BOM for UTF-8, but warns that it may be encountered at the start of a file as a transcoding artifact. The presence of the UTF-8 BOM may cause problems with existing software that can handle UTF-8..." https://en.wikipedia.org/wiki/UTF-8#Byte_order_mark https://en.wikipedia.org/wiki/UTF-8#Byte_order_mark
  - foobarrio 9y ago
    I should have searched. This is actually quite useful to be aware of. Thanks.
    - merb 9y ago
      you should remember it when creating csv's especially if your consumer uses windows. saves a ton of "what is a charset" questions.
- majewsky 9y ago
  I could imagine a UTF-8 BOM to be useful in helping identify that a given file is UTF-8-encoded, when the encoding is not clear from context (such as an outer MIME type specification or explicit encoding specs like those in XML documents).
cryptonector 9y ago
Without. The BOM is for UTF-16 and UTF-32. Nor would would you want a BOM on every string in memory -- that would make it impossible to cheaply substring, for example. BOMs are for external documents -- for interop.
- codeulike 9y ago
  Ok, but for files, it does happen https://docs.microsoft.com/en-us/globalization/encoding/byte-order-mark https://docs.microsoft.com/en-us/globalization/encoding/byte...
  - cryptonector 9y ago
    I understand, but it's a mistake and completely unnecessary.

foobarrio 9y ago

Wait there's a UTF-8 with BOM? Why do you need a byte order mark for utf-8? The stream "unit" is a byte. I thought only utf-16 and above had an optional BOM?

grzm 9y ago
They're likely referring to this behavior on Windows: > "Many Windows programs (including Windows Notepad) add the bytes 0xEF, 0xBB, 0xBF at the start of any document saved as UTF-8. This is the UTF-8 encoding of the Unicode byte order mark (BOM), and is commonly referred to as a UTF-8 BOM, even though it is not relevant to byte order. A BOM can also appear if another encoding with a BOM is translated to UTF-8 without stripping it. Software that is not aware of multibyte encodings will display the BOM as three garbage characters at the start of the document, e.g. "ï»¿" in software interpreting the document as ISO 8859-1 or Windows-1252 or "∩╗┐" if interpreted as code page 437, a default for certain older Windows console applications." > "The Unicode Standard neither requires nor recommends the use of the BOM for UTF-8, but warns that it may be encountered at the start of a file as a transcoding artifact. The presence of the UTF-8 BOM may cause problems with existing software that can handle UTF-8..." https://en.wikipedia.org/wiki/UTF-8#Byte_order_mark https://en.wikipedia.org/wiki/UTF-8#Byte_order_mark
- foobarrio 9y ago
  I should have searched. This is actually quite useful to be aware of. Thanks.
  - merb 9y ago
    you should remember it when creating csv's especially if your consumer uses windows. saves a ton of "what is a charset" questions.
majewsky 9y ago
I could imagine a UTF-8 BOM to be useful in helping identify that a given file is UTF-8-encoded, when the encoding is not clear from context (such as an outer MIME type specification or explicit encoding specs like those in XML documents).

cryptonector 9y ago

Without. The BOM is for UTF-16 and UTF-32. Nor would would you want a BOM on every string in memory -- that would make it impossible to cheaply substring, for example. BOMs are for external documents -- for interop.

codeulike 9y ago
Ok, but for files, it does happen https://docs.microsoft.com/en-us/globalization/encoding/byte-order-mark https://docs.microsoft.com/en-us/globalization/encoding/byte...
- cryptonector 9y ago
  I understand, but it's a mistake and completely unnecessary.