3 ms·

From section 2.6 in the standard: "Use of a BOM is neither required nor recommended for UTF-8, but may be encountered in contexts where UTF-8 data is converted

by Avernar 10y ago

From section 2.6 in the standard: "Use of a BOM is neither required nor recommended for UTF-8, but may be encountered in contexts where UTF-8 data is converted from other encoding forms that use a BOM or where the BOM is used as a UTF-8 signature." Yes, it can be used to distinguish a UTF-8 stream but it's not recommended. One issue is you can't tell if the BOM is not valid text in some other non-unicode encoding. I'm curious where you've encountered missing content-encoding headers or other OOB indicators where it wasn't because of programmer error or laziness.

Const-me 10y ago
> it can be used to distinguish a UTF-8 stream but it's not recommended If a specification says “something may be encountered”, for me, when I write my software, it means I must support that thing. Otherwise, the software won’t conform to the spec. > I'm curious where you've encountered missing content-encoding headers or other OOB indicators where it wasn't because of programmer error or laziness. Everywhere. Most filesystems don’t have encoding headers for their text files. Most databases don’t have headers for their blob columns. Only web that has encoding headers.
- Avernar 10y ago
  There's a difference between what a program should accept as input and what it should generate as output. The standard just says to expect a BOM on input and suggests not generating on output. In other words "A UTF-8 BOM is a bad idea but some yutz out there stated doing it so we should ignore it on input". Someone else mentioned that the yutz was Microsoft. I misread what you wrote about where you saw no idication that it was UTF-8. You were talking about places other than the web. BOM for UTF-8 text files seems to be a Microsoft thing. Everyone else just defaults to UTF-8. But you can't be sure that it's a UTF-8 BOM or some other encoding. Most editors let the user overide what it is. Why would you store text in a blob column? If a database can't handle UTF-8 in it's text column it needs to be fixed (or taken out back an shot).
  - Const-me 10y ago
    > There's a difference between what a program should accept as input and what it should generate as output. I’m a Windows developer. In my world, a program should generate its output in whatever format user wants it to be. When I press “File/Save as” in visual studio and click on the down arrow icon, I see a choice of more than 100 different encodings (including all flavors of Unicode with and without the BOM), and independent choice of 3 line endings (Window, mac, Unix). > BOM for UTF-8 text files seems to be a Microsoft thing Practically — maybe, most Microsoft apps tend to understand those BOMs, and most *nix tools don’t, even on input. Officially — definitely no, we both saw the spec on unicode.org.
    - Avernar 10y ago
      > I’m a Windows developer. In my world, a program should generate its output in whatever format user wants it to be. When generating output for a user, letting them choose is a good idea. But for interop with other programs I leave it off unless the program needs it. > Officially — definitely no, we both saw the spec on unicode.org. The spec says the BOM is optional. Some Microsoft programs however require it.
      - Const-me 10y ago
        > for interop with other programs I leave it off unless the program needs it. Plain text isn’t exactly a machine-friendly format. If you want to interop with other programs, the good choice is e.g. XML. That has this encoding problem fixed as a part of the standard. > The spec says the BOM is optional. Some Microsoft programs however require it. Could you please name a Microsoft program that you think requires a BOM? I’m asking because I have completely different experience. For me, Microsoft programs open text files just fine, with or without the BOM. But most *nix and osx programs show me garbage instead of BOM.
        Avernar 10y ago
        > Plain text isn’t exactly a machine-friendly format. Works fine for unix. :D > Could you please name a Microsoft program that you think requires a BOM? Visual C++ off the top of my head. It mangles UTF-8 string literals without the BOM in the source code. > For me, Microsoft programs open text files just fine, with or without the BOM. But most *nix and osx programs show me garbage instead of BOM. That's what I was trying to say about the BOM being prevalent on the Windows side of the fence. Some programs require it, some always generate it so most program now accept it. On the unix/osx side everyone switched to UTF-8 so the BOM is redundant. Everything is UTF-8 so the silliness of this needs a BOM that doesn't need a BOM doesn't exist. Good example of what the "UTF-8 Everywhere" site is trying to promote. Personally I really wish Microsoft would eventually fix their UTF-8 codepage. Would be so nice not having to convert to/from UTF-16 at the Win32 API boundary.

by Avernar 10y ago

Const-me 10y ago

> it can be used to distinguish a UTF-8 stream but it's not recommended If a specification says “something may be encountered”, for me, when I write my software, it means I must support that thing. Otherwise, the software won’t conform to the spec. > I'm curious where you've encountered missing content-encoding headers or other OOB indicators where it wasn't because of programmer error or laziness. Everywhere. Most filesystems don’t have encoding headers for their text files. Most databases don’t have headers for their blob columns. Only web that has encoding headers.

Avernar 10y ago
There's a difference between what a program should accept as input and what it should generate as output. The standard just says to expect a BOM on input and suggests not generating on output. In other words "A UTF-8 BOM is a bad idea but some yutz out there stated doing it so we should ignore it on input". Someone else mentioned that the yutz was Microsoft. I misread what you wrote about where you saw no idication that it was UTF-8. You were talking about places other than the web. BOM for UTF-8 text files seems to be a Microsoft thing. Everyone else just defaults to UTF-8. But you can't be sure that it's a UTF-8 BOM or some other encoding. Most editors let the user overide what it is. Why would you store text in a blob column? If a database can't handle UTF-8 in it's text column it needs to be fixed (or taken out back an shot).
- Const-me 10y ago
  > There's a difference between what a program should accept as input and what it should generate as output. I’m a Windows developer. In my world, a program should generate its output in whatever format user wants it to be. When I press “File/Save as” in visual studio and click on the down arrow icon, I see a choice of more than 100 different encodings (including all flavors of Unicode with and without the BOM), and independent choice of 3 line endings (Window, mac, Unix). > BOM for UTF-8 text files seems to be a Microsoft thing Practically — maybe, most Microsoft apps tend to understand those BOMs, and most *nix tools don’t, even on input. Officially — definitely no, we both saw the spec on unicode.org.
  - Avernar 10y ago
    > I’m a Windows developer. In my world, a program should generate its output in whatever format user wants it to be. When generating output for a user, letting them choose is a good idea. But for interop with other programs I leave it off unless the program needs it. > Officially — definitely no, we both saw the spec on unicode.org. The spec says the BOM is optional. Some Microsoft programs however require it.
    - Const-me 10y ago
      > for interop with other programs I leave it off unless the program needs it. Plain text isn’t exactly a machine-friendly format. If you want to interop with other programs, the good choice is e.g. XML. That has this encoding problem fixed as a part of the standard. > The spec says the BOM is optional. Some Microsoft programs however require it. Could you please name a Microsoft program that you think requires a BOM? I’m asking because I have completely different experience. For me, Microsoft programs open text files just fine, with or without the BOM. But most *nix and osx programs show me garbage instead of BOM.
      - Avernar 10y ago
        > Plain text isn’t exactly a machine-friendly format. Works fine for unix. :D > Could you please name a Microsoft program that you think requires a BOM? Visual C++ off the top of my head. It mangles UTF-8 string literals without the BOM in the source code. > For me, Microsoft programs open text files just fine, with or without the BOM. But most *nix and osx programs show me garbage instead of BOM. That's what I was trying to say about the BOM being prevalent on the Windows side of the fence. Some programs require it, some always generate it so most program now accept it. On the unix/osx side everyone switched to UTF-8 so the BOM is redundant. Everything is UTF-8 so the silliness of this needs a BOM that doesn't need a BOM doesn't exist. Good example of what the "UTF-8 Everywhere" site is trying to promote. Personally I really wish Microsoft would eventually fix their UTF-8 codepage. Would be so nice not having to convert to/from UTF-16 at the Win32 API boundary.