4 ms·

No, UTF-8 cannot be processed in exactly the same way as ASCII! It is highly compatible with ASCII-assuming environments because it can accept the same data (e

No, UTF-8 cannot be processed in exactly the same way as ASCII! It is highly compatible with ASCII-assuming environments because it can accept the same data (e.g. it remains a stream of bytes that doesn't have to be converted to some "fat" integer) and any UTF-8 text that just happens to contain only ASCII characters will work just fine with an old ASCII program. But complex multi-byte UTF-8 inputs will not work without special treatment. Historically code pages mapped character values directly to individual bytes: typically redefining the upper half (129+) while leaving the lower numbers the same as ASCII. Programs could display a wide variety of text encodings correctly as long as they knew what the encoding was, and all they had to do was read bytes individually. These encodings could all be handled in a way largely similar to ASCII. UTF-8 however is a multi-byte encoding, which means you could have (say) 3 bytes that combine to form a symbol. Not only that, but in certain forms multiple symbols could imply display as a single glyph (e.g. an accent followed by a letter). A program that does everything the old ASCII way would choke on multi-byte UTF-8, despite being otherwise-compatible. Consider a program with a fixed-size read buffer; if the last byte is only partway through a multi-byte character, that character will be mishandled unless the program knows how to preserve those bytes and "complete" the character when more bytes arrive.

ajross 14y ago
You're thinking from the point of view of a display program that needs to split strings into glyphs. That's a fine application, but very rare. And yes, it's inherently encoding dependent and tends to like wide characters instead of multibyte ones. But introducing a BOM for the sake of that application is a disaster, because it hurts everything else. You can (literally) feed UTF-8 to parsers written 30 years ago and apply all your existing intuition about string handling in C without worry. Unless you deliberately break it by including a binary, non encoding garbage furball at the front of your "file" (and good luck figuring out what a "file" should mean in a OS metaphor designed around streams).
- makecheck 14y ago
  If a file really is pure ASCII, leave it that way. I am not suggesting to do otherwise. If a 30-year-old program only deals with ASCII then make sure your input looks like ASCII. But if your input could contain complex UTF-8 (e.g. it's multi-language or whatever), you're not doing any favors by hiding this fact. The BOM is a quick way to know exactly what the file is, and it shows you that your program won't work with that input. So you translate the input or you fix the program. At some point in the future the majority of programs will handle even complex UTF-8 properly, and then the BOM will be pointless because virtually all inputs will be UTF-8.

ajross 14y ago

You're thinking from the point of view of a display program that needs to split strings into glyphs. That's a fine application, but very rare. And yes, it's inherently encoding dependent and tends to like wide characters instead of multibyte ones. But introducing a BOM for the sake of that application is a disaster, because it hurts everything else. You can (literally) feed UTF-8 to parsers written 30 years ago and apply all your existing intuition about string handling in C without worry. Unless you deliberately break it by including a binary, non encoding garbage furball at the front of your "file" (and good luck figuring out what a "file" should mean in a OS metaphor designed around streams).

makecheck 14y ago
If a file really is pure ASCII, leave it that way. I am not suggesting to do otherwise. If a 30-year-old program only deals with ASCII then make sure your input looks like ASCII. But if your input could contain complex UTF-8 (e.g. it's multi-language or whatever), you're not doing any favors by hiding this fact. The BOM is a quick way to know exactly what the file is, and it shows you that your program won't work with that input. So you translate the input or you fix the program. At some point in the future the majority of programs will handle even complex UTF-8 properly, and then the BOM will be pointless because virtually all inputs will be UTF-8.