6 ms·

I have an old saw about UTF-16 not being an irredeemable format and UTF-8 eating the world being bad, and I'm happy to dig it out again. UTF-16 is great for lo

I have an old saw about UTF-16 not being an irredeemable format and UTF-8 eating the world being bad, and I'm happy to dig it out again. UTF-16 is great for lots of East Asian languages, which billions of people use. In UTF-8, most of those languages require 3 bytes to encode a 32-bit codepoint, in UTF-16 they only ever need 2. This ends up being a huge savings. The main benefit of UTF-8 if you're say, Chinese, is interop. Everything else is worse. You might think "but BOMs are super evil." Checking a BOM is extremely, extremely easy. Furthermore, you don't get to bail out of checking anything just by using UTF-8, you have to check to ensure you have _valid_ UTF-8. That's right, you gotta scan the whole bytestream anyway, so you may as well just check the 2-byte BOM at the beginning too. You might also think "what about ASCII compatibility?" ASCII compatibility is an anti-feature. You should never be indexing into UTF strings (you always have to iterate, or save the results of an iteration), upper/lowercasing isn't addition/subtraction, etc. etc. You also can't just forget about encodings as a result--you can store ASCII in something expecting UTF-8, but you definitely can't store UTF-8 in something expecting ASCII. So if you're sniffing/decoding/tagging a format anyway, you may as well be agnostic. You might also think "OK OK, you could be right, but what about HTML, which is mostly ASCII and would nearly double in size if it went from UTF-8 to UTF-16." Practically all HTML is gzipped, so the difference is pretty small, plus the majority of text isn't HTML (almost anything stored in a database, almost anything in a file on your computer, etc.) Different encodings are good at different things. There's no one superior encoding for all uses. What we need is text encoding agnosticism. --- In fairness, I will say I've heard that UTF-8 is pretty popular in countries with exactly the kind of languages I'm talking about, so the issue is mostly moot at this point. I just think UTF-16 gets a really bad rap, and I think we shouldn't just gloss over UTF-8 having won because it's good for European languages.

JoshTriplett 5y ago
If you care about text size, you should compress your text; that'll save much more space, since it can optimize for what's actually used in the document. > ASCII compatibility is an anti-feature. ASCII compatibility is extremely useful if you're working with, for instance, filenames or programming languages. You can lex UTF-8 and handle separators like `/` or quotes like `"` and `'`, because those bytes can never occur otherwise.
- camgunz 5y ago
  Sure it's useful, but it's not at all a guarantee. You can't assume the byte stream is UTF-8, and UTF-8 has a NUL in it. Reading about it, filename encoding looks like a mess, GLib apparently assumes UTF-8 unless you set it with an environment variable, which I think is wild. Qt uses the locale, which feels reasonable (ignoring that locales are themselves bananas). But assuming UTF-8 and relying on ASCII compatibility and no NULs feels risky.
  - Dylan16807 5y ago
    ASCII has NUL too. You can't put it in a filename either way.
    - camgunz 5y ago
      Yeah but there's no reason to put it in strings because it's not printable. When you get to the 2/3/4 byte-wide parts or UTF-8 they can have zeroes in there. This is a long-winded way of saying UTF-8 can't really be NULL-terminated because it can contain NULLs as a matter of course. That's where you get modified UTF-8, which is indistinguishable from regular UTF-8 other than it's NULL encoding, which if you don't know it's the modified derivative will sneak up on you, just like UCS-2 vs UTF-16. This stuff just has to be OOB configurable. ASCII-compatibility lets us think we can do stuff like this, but you're always just assuming and hoping it doesn't blow up.
      - Dylan16807 5y ago
        > This is a long-winded way of saying UTF-8 can't really be NULL-terminated because it can contain NULLs as a matter of course. UTF-8 "can contain NULLs as a matter of course" in exactly the same way that ASCII can. Any system that doesn't allow null bytes in ASCII can just as easily not allow them in UTF-8. A system that does allow them can just as easily allow them in both. There is no difference between the two here. - Edit: Oh, wait, maybe you misunderstand UTF-8. The 2+ byte encodings never use bytes below 0x80. They never cause a null byte to happen. The only way you get a null byte is if it's an ASCII-compatible null character, code point 00000.
poorlyknit 5y ago
(author here) You are absolutely right! If your use case demands storage of CJK strings, UTF-8 is probably not your best bet. But our data (mostly stringly typed, think product descriptions or links) is >99% Basic Latin characters and we are getting to a point where memory is actually becoming an issue. So it's neat that Haskell allows us to "upgrade" to UTF-8. With the data being in memory I think compression would not be very helpful either. Edit: I also kind of agree that UTF-16 gets to much hate :D
jcranmer 5y ago
The main evil of UTF-16 is that it's usually not UTF-16, it's UCS-2, which is to say, a fixed-width format which has been shoehorned into a varible-width format to support non-BMP code points. As a consequence, systems that are built on "UTF-16" (i.e., really UCS-2) use designs centered around the ability to random-access Unicode strings despite the fact that doing so reliably isn't possible. > ASCII compatibility is an anti-feature. Actually, it's not. One of the main reasons it's important is... > What we need is text encoding agnosticism ... that bit that you want. If you're supporting multiple encodings, you need a reliable way of signalling that encoding. And file metadata tends to be insufficiently reliable, so it ends up being in-band for text (e.g., HTML files). If all the encodings you need to support are ASCII-compatible, and the directive to set the charset is itself pure ASCII, that means it is perfectly safe to parse the file as ASCII to figure out how to parse the file. If the charset isn't ASCII-compatible... well, that historically has led to security vulnerabilities. Ergo, a non-ASCII-compatible charset has heightened security risks. That said, my own experience when dealing with the mess that is charsets is that supporting multiple charsets is itself a painful process that isn't worth it.
- josephg 5y ago
  > systems that are built on "UTF-16" (i.e., really UCS-2) use designs centered around the ability to random-access Unicode strings despite the fact that doing so reliably isn't possible. 100% this. In javascript you can accidentally cut characters in half. This lets you accidentally construct strings which have invalid encodings - and that can lead to all sorts of weird problems. In comparison, using rust's safe API, its impossible to construct a string which contains invalid unicode. For example, if I put an emoji like this U+1F970 in javascript, I can then split it in half and put one half into a JSON string - like this: "\"\\ud83e\"". Javascript will treat this as "one character". Rust will treat this as an encoding error, and will refuse to parse it back into a string. Another implication is that javascript's .length field looks useful, but its almost never what you actually want. How many unicode characters in this string: "\ud83e\udd70"? Trick question - even though javascript says the string's length is 2, there's only 1 unicode character in that string. How many bytes does this string (with JS length 1) "\u270C" take up over the wire? Javascript says the length is 1, but it takes 2 bytes to send in any encoding (utf8 or utf16). What a mess.
josephg 5y ago
> In UTF-8, most of those languages require 3 bytes to encode a 32-bit codepoint, in UTF-16 they only ever need 2. This ends up being a huge savings. Meanwhile in lots of other countries, UTF-8 lets you use 1 byte to store each character instead of 2; which is also a huge savings. The question is, is it better for the world to pick one encoding and have all programs use it, or pick several encodings and switch between them depending on the context / country? Picking one encoding means we can reuse our code more easily. Picking multiple encodings means we get slightly smaller file sizes. For my money, the best approach is to use UTF-8 everywhere. This lets us reuse code. Then if you're worried about file size on disk or over the wire, compress your text content with LZ4 or snappy or something. That'll halve the size of text for everyone and LZ4 is so fast its essentially free from a computational standpoint. As this blog post demonstrates, quite a lot of code needs to deeply understand text encoding to work (be that UTF-8 or UTF-16 or whatever). Its a big savings in programming hours if we only have to write this code once.
- camgunz 5y ago
  I mostly disagree. We already have lots of code that deals with tons of different encodings, UTF-16 is pretty easy and there's lots of libraries already, etc. But if we ignore all that, your argument is against UTF-8, which succeeded UTF-16.
rectang 5y ago
> The main benefit of UTF-8 if you're say, Chinese, is interop. Everything else is worse. Well, for HTML it turns out to be a wash. While Han characters are typically three bytes in UTF-8, markup is ASCII and so only one byte — whereas in UTF-17, both the character data and the markup are two bytes. When you add them together the average cost per character is more or less two bytes, whether your Chinese-content HTML is encoded as UTF-8 or UTF-17.
- rectang 5y ago
  (LOL. How did I type "UTF-17", twice?)
  - camgunz 5y ago
    Off-topic but yeah, 6 is so far away. I really notice it doing bitwise stuff with '^'.
- camgunz 5y ago
  This is probably true for HTML, but not true for lots of other text like whatever's stored in databases or on hard drives.
Dylan16807 5y ago
> UTF-16 is great for lots of East Asian languages, which billions of people use. In UTF-8, most of those languages require 3 bytes to encode a 32-bit codepoint, in UTF-16 they only ever need 2. This ends up being a huge savings. Not really. That's just an excuse to be contrarian. Any significant length of text will be compressed and/or have lots of latin characters. > you definitely can't store UTF-8 in something expecting ASCII That's not true.
- camgunz 5y ago
  > Any significant length of text will be compressed and/or have lots of latin characters. A reasonable counterargument is Asian books on Project Gutenberg [0]. I guess you can say they're compressed on the wire, but they're not compressed in my RAM or cache. >> you definitely can't store UTF-8 in something expecting ASCII > That's not true. Well alright, you can jam bytes pretty much into anything. And you get luckier with UTF-8 than you do with probably any other encoding. But lucky isn't robust, is all I'm saying. [0]: https://www.gutenberg.org/browse/languages/zh https://www.gutenberg.org/browse/languages/zh
  - Dylan16807 5y ago
    > I guess you can say they're compressed on the wire, but they're not compressed in my RAM or cache. You don't really need to have very much of a book decompressed at once, and in your average program handling that book there's going to be much more RAM dedicated to layout and display than the actual character bytes. > Well alright, you can jam bytes pretty much into anything. And you get luckier with UTF-8 than you do with probably any other encoding. But lucky isn't robust, is all I'm saying. It won't work in all systems but it'll work in most of them. And you can check beforehand.
    - camgunz 5y ago
      In truth I think these are mostly arguments for letting apps/users pick their encoding. If GBK works for your app and user base, go for it. If UTF-8 ends up being more efficient, have at it. We can sit around hypothesizing about a book reading app or a grammar analyzing app or blah blah blah, but I'm pretty sure UTF-8 won't be the best encoding for every circumstance. Python at least used to let you do this at compile time, for example.
netheril96 5y ago
I'm Chinese, and I prefer UTF8 to UTF16. > You might think "but BOMs are super evil." Checking a BOM is extremely, extremely easy. Furthermore, you don't get to bail out of checking anything just by using UTF-8, you have to check to ensure you have _valid_ UTF-8. That's right, you gotta scan the whole bytestream anyway, so you may as well just check the 2-byte BOM at the beginning too. BOM breaks substring. Everything you take a substring, you need to prepend a BOM if you want to serialize it to somewhere else. If you want to concat two strings with BOM, you need to remove one of them. All of these are unnecessary pains. > You might also think "OK OK, you could be right, but what about HTML, which is mostly ASCII and would nearly double in size if it went from UTF-8 to UTF-16." Practically all HTML is gzipped, so the difference is pretty small, plus the majority of text isn't HTML (almost anything stored in a database, almost anything in a file on your computer, etc.) This just contradicts your own reasoning that UTF-16 is better for East Asians due to size savings.
- camgunz 5y ago
  Substrings just get the BOM too. EZ. Re: HTML, people usually bring it up because it's a lot of ASCII that'll all double up under UTF-16. Mostly my argument is that most text isn't HTML, so size savings on other text files still matter.
  - netheril96 5y ago
    Which standard string library ever does that?
    - camgunz 5y ago
      Are there UTF-16 string libraries that let you split a UTF-16 string into a string with a BOM and a string without it? That would feel like a bug to me.
- bjoli 5y ago
  That is a minor nuisance when dealing with raw bytes. Any sane language representation would probably store it as a separate field egen not on the wire. I would probably do BOM, LUT, BYTES. Where LUT is a lookup table of indices for every 64 chars so that you could do o(1) random access for the vast majority of cases.
thaumasiotes 5y ago
> I have an old saw about UTF-16 not being an irredeemable format and UTF-8 eating the world being bad, and I'm happy to dig it out again. > UTF-16 is great for lots of East Asian languages, which billions of people use. In UTF-8, most of those languages require 3 bytes to encode a 32-bit codepoint, in UTF-16 they only ever need 2. This ends up being a huge savings. That doesn't make UTF-16 not a terrible format, though. East Asian countries use better formats instead. For example, in China they use GB18030. When your case for "UTF-16 is not an irredeemable format" is "it's better than UTF-8 at a task where nobody would use either format anyway", you're not making a strong case.
- camgunz 5y ago
  > When your case for "UTF-16 is not an irredeemable format" is "it's better than UTF-8 at a task where nobody would use either format anyway", you're not making a strong case. Haha this is pretty fair; I know I'm being pedantic. But I wouldn't say any GB* encoding is better than UTF-16. Another commenter pointed out that you really want the ability to--say--paste arbitrary text into an editor and for that editor to be using an encoding that can handle it. Or even for the web, if we actually took the Accept-Language header [0] seriously, that could also be a big savings. [0]: https://www.rfc-editor.org/rfc/rfc7231#section-5.3.5 https://www.rfc-editor.org/rfc/rfc7231#section-5.3.5
hnfong 5y ago
> UTF-16 is great for lots of East Asian languages, which billions of people use. In UTF-8, most of those languages require 3 bytes to encode a 32-bit codepoint, in UTF-16 they only ever need 2. This ends up being a huge savings. "Only ever" is an exaggeration. The vast majority of standard Chinese (aka. Putonghua) characters are in the BMP, but occasionally there are characters outside of it. In particular a lot of Hong Kong characters are outside of the BMP, and before emoji forced software to fix things, systems claiming to support UTF-16 (but were instead UCS-2) kept failing for say 1% of the characters. I've had much better luck with UTF-8 systems than purported UTF-16 ones. The problem with UTF-16 is not necessarily in its inherent technical issues, but rather, it's how often systems with UCS-2 support just claim they support UTF-16, and leave users/developers with a crappy experience when working with characters outside the BMP. There's also the encoding agnosticism thing -- with UTF-16 you run into issues about byte order marks (with or without), endian problems, and generally incompatibility with ASCII. As I said, the situation probably improved quite a bit after emoji became popular. But if I had the choice, I'd choose UTF-8 over UTF-16 any time. Also, although it's generally true that you "can't store UTF-8 in something expecting ASCII", in practice a lot of systems are somewhat more graceful than that. For example, if I grep a UTF-8 text file, grep doesn't need to know the text encoding if all you're trying to find is an ASCII string. Similarly it's possible to edit a UTF-8 text file as an ASCII file if the editor preserves the "binary" bits. In conclusion, "different encodings are good at different things" is only true when considered in isolation. The historical baggage of UCS-2, and separately, of ASCII-compatibility tilts the balance strongly in favor of UTF-8 IMHO. The alleged 33% saving in Chinese text doesn't really matter that much in the grand scheme of things -- which is perhaps why people use UTF-8 in spite of the "advantages" you mentioned.
- camgunz 5y ago
  Yeah I mean, I broadly agree with what you're saying. Text encoding is a mess, and pragmatic solutions are needed. And when jamming text in various places, jamming UTF-8 is typically more successful than jamming in UTF-16, because of NULL bytes and the shortsightedness of UCS-2. But I think that 99% of the things you have to do for UTF-8 (validate, encode/decode, no indexing) you have to do for UTF-16, and if we're arguing for systems that are UTF-aware, UTF-16 does great. If we're arguing for trying to jam encoded bytes into systems, well it's a crapshoot. UTF-8 being better at winning that crapshoot doesn't feel like a recipe for robustness to me, and certainly doesn't strike me as a perfect solution. OOB configuration does, though. > For example, if I grep a UTF-8 text file, grep doesn't need to know the text encoding if all you're trying to find is an ASCII string. Well, this is a weird example. grep uses locales, so yeah you're probably fine if you're grepping for ASCII using an ASCII-compatible locale. But if you're not, then what you feed to grep has to be encoded in your locale's encoding (I think). So I think this is actually an example in favor of "this should be OOB configurable", and an example of "ASCII-compatibility lulls into a false sense of security".
account42 5y ago
> You might think "but BOMs are super evil." Checking a BOM is extremely, extremely easy. Checking the BOM is not enough, you also have to handle it. Non-LE BOMs (like surrogates, at least before the emojipocalypse) are rare enough that many "UTF-16" based tools simply only support UTF-16LE. They are also stupid because you need to know that you are dealing with UTF-16LE or UTF-16BE in the first place - either through heuristics or because it is specidified and then you can also guess/specify the byte order. And like surrogates, the BOM is another wasted Unicode character that is only needed for the UTF-16 mess. > Furthermore, you don't get to bail out of checking anything just by using UTF-8, you have to check to ensure you have _valid_ UTF-8. That's right, you gotta scan the whole bytestream anyway, so you may as well just check the 2-byte BOM at the beginning too. You don't need validation for most UTF-8 tasks, GIGO is often more reasonable for things entered by humans. > you definitely can't store UTF-8 in something expecting ASCII. Unix tools would disagree. $ echo "Hello Wörld!" | tr '!' '?' Hello Wörld? Many tools only care about finding substrings. UTF-8 gurantees if the byte sequence making up the encoding of a Unicode character (or sequence) appears in valid UTF-8 encoded text then it also decodes to that Unicode sequence. > Practically all HTML is gzipped Not when it is being parsed it isn't.
- camgunz 5y ago
  > Non-LE BOMs (like surrogates, at least before the emojipocalypse) are rare enough that many "UTF-16" based tools simply only support UTF-16LE. Gymnastics like this always occur when you don't know the encoding. If you didn't know you were getting UTF-8, welcome to heuristics or whatever. > And like surrogates, the BOM is another wasted Unicode character that is only needed for the UTF-16 mess. This isn't a big deal at all. > You don't need validation for most UTF-8 tasks, GIGO is often more reasonable for things entered by humans. You can't build robust systems this way. You always have to validate, and in order to do that, you gotta know what encoding you're getting. > Unix tools would disagree. Most of those use locales, or they don't expect ASCII they expect bytes. Again you can get lucky with this, but you can't build a robust system with luck. > Many tools only care about finding substrings. You have to scan through the string or save offsets (after an initial scan) either way, UTF-8 or 16. > Not when it is being parsed it isn't. Fair point!

JoshTriplett 5y ago

If you care about text size, you should compress your text; that'll save much more space, since it can optimize for what's actually used in the document. > ASCII compatibility is an anti-feature. ASCII compatibility is extremely useful if you're working with, for instance, filenames or programming languages. You can lex UTF-8 and handle separators like `/` or quotes like `"` and `'`, because those bytes can never occur otherwise.

camgunz 5y ago
Sure it's useful, but it's not at all a guarantee. You can't assume the byte stream is UTF-8, and UTF-8 has a NUL in it. Reading about it, filename encoding looks like a mess, GLib apparently assumes UTF-8 unless you set it with an environment variable, which I think is wild. Qt uses the locale, which feels reasonable (ignoring that locales are themselves bananas). But assuming UTF-8 and relying on ASCII compatibility and no NULs feels risky.
- Dylan16807 5y ago
  ASCII has NUL too. You can't put it in a filename either way.
  - camgunz 5y ago
    Yeah but there's no reason to put it in strings because it's not printable. When you get to the 2/3/4 byte-wide parts or UTF-8 they can have zeroes in there. This is a long-winded way of saying UTF-8 can't really be NULL-terminated because it can contain NULLs as a matter of course. That's where you get modified UTF-8, which is indistinguishable from regular UTF-8 other than it's NULL encoding, which if you don't know it's the modified derivative will sneak up on you, just like UCS-2 vs UTF-16. This stuff just has to be OOB configurable. ASCII-compatibility lets us think we can do stuff like this, but you're always just assuming and hoping it doesn't blow up.
    - Dylan16807 5y ago
      > This is a long-winded way of saying UTF-8 can't really be NULL-terminated because it can contain NULLs as a matter of course. UTF-8 "can contain NULLs as a matter of course" in exactly the same way that ASCII can. Any system that doesn't allow null bytes in ASCII can just as easily not allow them in UTF-8. A system that does allow them can just as easily allow them in both. There is no difference between the two here. - Edit: Oh, wait, maybe you misunderstand UTF-8. The 2+ byte encodings never use bytes below 0x80. They never cause a null byte to happen. The only way you get a null byte is if it's an ASCII-compatible null character, code point 00000.

poorlyknit 5y ago

(author here) You are absolutely right! If your use case demands storage of CJK strings, UTF-8 is probably not your best bet. But our data (mostly stringly typed, think product descriptions or links) is >99% Basic Latin characters and we are getting to a point where memory is actually becoming an issue. So it's neat that Haskell allows us to "upgrade" to UTF-8. With the data being in memory I think compression would not be very helpful either. Edit: I also kind of agree that UTF-16 gets to much hate :D

jcranmer 5y ago

The main evil of UTF-16 is that it's usually not UTF-16, it's UCS-2, which is to say, a fixed-width format which has been shoehorned into a varible-width format to support non-BMP code points. As a consequence, systems that are built on "UTF-16" (i.e., really UCS-2) use designs centered around the ability to random-access Unicode strings despite the fact that doing so reliably isn't possible. > ASCII compatibility is an anti-feature. Actually, it's not. One of the main reasons it's important is... > What we need is text encoding agnosticism ... that bit that you want. If you're supporting multiple encodings, you need a reliable way of signalling that encoding. And file metadata tends to be insufficiently reliable, so it ends up being in-band for text (e.g., HTML files). If all the encodings you need to support are ASCII-compatible, and the directive to set the charset is itself pure ASCII, that means it is perfectly safe to parse the file as ASCII to figure out how to parse the file. If the charset isn't ASCII-compatible... well, that historically has led to security vulnerabilities. Ergo, a non-ASCII-compatible charset has heightened security risks. That said, my own experience when dealing with the mess that is charsets is that supporting multiple charsets is itself a painful process that isn't worth it.

josephg 5y ago
> systems that are built on "UTF-16" (i.e., really UCS-2) use designs centered around the ability to random-access Unicode strings despite the fact that doing so reliably isn't possible. 100% this. In javascript you can accidentally cut characters in half. This lets you accidentally construct strings which have invalid encodings - and that can lead to all sorts of weird problems. In comparison, using rust's safe API, its impossible to construct a string which contains invalid unicode. For example, if I put an emoji like this U+1F970 in javascript, I can then split it in half and put one half into a JSON string - like this: "\"\\ud83e\"". Javascript will treat this as "one character". Rust will treat this as an encoding error, and will refuse to parse it back into a string. Another implication is that javascript's .length field looks useful, but its almost never what you actually want. How many unicode characters in this string: "\ud83e\udd70"? Trick question - even though javascript says the string's length is 2, there's only 1 unicode character in that string. How many bytes does this string (with JS length 1) "\u270C" take up over the wire? Javascript says the length is 1, but it takes 2 bytes to send in any encoding (utf8 or utf16). What a mess.

josephg 5y ago

> In UTF-8, most of those languages require 3 bytes to encode a 32-bit codepoint, in UTF-16 they only ever need 2. This ends up being a huge savings. Meanwhile in lots of other countries, UTF-8 lets you use 1 byte to store each character instead of 2; which is also a huge savings. The question is, is it better for the world to pick one encoding and have all programs use it, or pick several encodings and switch between them depending on the context / country? Picking one encoding means we can reuse our code more easily. Picking multiple encodings means we get slightly smaller file sizes. For my money, the best approach is to use UTF-8 everywhere. This lets us reuse code. Then if you're worried about file size on disk or over the wire, compress your text content with LZ4 or snappy or something. That'll halve the size of text for everyone and LZ4 is so fast its essentially free from a computational standpoint. As this blog post demonstrates, quite a lot of code needs to deeply understand text encoding to work (be that UTF-8 or UTF-16 or whatever). Its a big savings in programming hours if we only have to write this code once.

camgunz 5y ago
I mostly disagree. We already have lots of code that deals with tons of different encodings, UTF-16 is pretty easy and there's lots of libraries already, etc. But if we ignore all that, your argument is against UTF-8, which succeeded UTF-16.

rectang 5y ago

> The main benefit of UTF-8 if you're say, Chinese, is interop. Everything else is worse. Well, for HTML it turns out to be a wash. While Han characters are typically three bytes in UTF-8, markup is ASCII and so only one byte — whereas in UTF-17, both the character data and the markup are two bytes. When you add them together the average cost per character is more or less two bytes, whether your Chinese-content HTML is encoded as UTF-8 or UTF-17.

rectang 5y ago
(LOL. How did I type "UTF-17", twice?)
- camgunz 5y ago
  Off-topic but yeah, 6 is so far away. I really notice it doing bitwise stuff with '^'.
camgunz 5y ago
This is probably true for HTML, but not true for lots of other text like whatever's stored in databases or on hard drives.

Dylan16807 5y ago

> UTF-16 is great for lots of East Asian languages, which billions of people use. In UTF-8, most of those languages require 3 bytes to encode a 32-bit codepoint, in UTF-16 they only ever need 2. This ends up being a huge savings. Not really. That's just an excuse to be contrarian. Any significant length of text will be compressed and/or have lots of latin characters. > you definitely can't store UTF-8 in something expecting ASCII That's not true.

camgunz 5y ago
> Any significant length of text will be compressed and/or have lots of latin characters. A reasonable counterargument is Asian books on Project Gutenberg [0]. I guess you can say they're compressed on the wire, but they're not compressed in my RAM or cache. >> you definitely can't store UTF-8 in something expecting ASCII > That's not true. Well alright, you can jam bytes pretty much into anything. And you get luckier with UTF-8 than you do with probably any other encoding. But lucky isn't robust, is all I'm saying. [0]: https://www.gutenberg.org/browse/languages/zh https://www.gutenberg.org/browse/languages/zh
- Dylan16807 5y ago
  > I guess you can say they're compressed on the wire, but they're not compressed in my RAM or cache. You don't really need to have very much of a book decompressed at once, and in your average program handling that book there's going to be much more RAM dedicated to layout and display than the actual character bytes. > Well alright, you can jam bytes pretty much into anything. And you get luckier with UTF-8 than you do with probably any other encoding. But lucky isn't robust, is all I'm saying. It won't work in all systems but it'll work in most of them. And you can check beforehand.
  - camgunz 5y ago
    In truth I think these are mostly arguments for letting apps/users pick their encoding. If GBK works for your app and user base, go for it. If UTF-8 ends up being more efficient, have at it. We can sit around hypothesizing about a book reading app or a grammar analyzing app or blah blah blah, but I'm pretty sure UTF-8 won't be the best encoding for every circumstance. Python at least used to let you do this at compile time, for example.

netheril96 5y ago

I'm Chinese, and I prefer UTF8 to UTF16. > You might think "but BOMs are super evil." Checking a BOM is extremely, extremely easy. Furthermore, you don't get to bail out of checking anything just by using UTF-8, you have to check to ensure you have _valid_ UTF-8. That's right, you gotta scan the whole bytestream anyway, so you may as well just check the 2-byte BOM at the beginning too. BOM breaks substring. Everything you take a substring, you need to prepend a BOM if you want to serialize it to somewhere else. If you want to concat two strings with BOM, you need to remove one of them. All of these are unnecessary pains. > You might also think "OK OK, you could be right, but what about HTML, which is mostly ASCII and would nearly double in size if it went from UTF-8 to UTF-16." Practically all HTML is gzipped, so the difference is pretty small, plus the majority of text isn't HTML (almost anything stored in a database, almost anything in a file on your computer, etc.) This just contradicts your own reasoning that UTF-16 is better for East Asians due to size savings.

camgunz 5y ago
Substrings just get the BOM too. EZ. Re: HTML, people usually bring it up because it's a lot of ASCII that'll all double up under UTF-16. Mostly my argument is that most text isn't HTML, so size savings on other text files still matter.
- netheril96 5y ago
  Which standard string library ever does that?
  - camgunz 5y ago
    Are there UTF-16 string libraries that let you split a UTF-16 string into a string with a BOM and a string without it? That would feel like a bug to me.
bjoli 5y ago
That is a minor nuisance when dealing with raw bytes. Any sane language representation would probably store it as a separate field egen not on the wire. I would probably do BOM, LUT, BYTES. Where LUT is a lookup table of indices for every 64 chars so that you could do o(1) random access for the vast majority of cases.

thaumasiotes 5y ago

> I have an old saw about UTF-16 not being an irredeemable format and UTF-8 eating the world being bad, and I'm happy to dig it out again. > UTF-16 is great for lots of East Asian languages, which billions of people use. In UTF-8, most of those languages require 3 bytes to encode a 32-bit codepoint, in UTF-16 they only ever need 2. This ends up being a huge savings. That doesn't make UTF-16 not a terrible format, though. East Asian countries use better formats instead. For example, in China they use GB18030. When your case for "UTF-16 is not an irredeemable format" is "it's better than UTF-8 at a task where nobody would use either format anyway", you're not making a strong case.

camgunz 5y ago
> When your case for "UTF-16 is not an irredeemable format" is "it's better than UTF-8 at a task where nobody would use either format anyway", you're not making a strong case. Haha this is pretty fair; I know I'm being pedantic. But I wouldn't say any GB* encoding is better than UTF-16. Another commenter pointed out that you really want the ability to--say--paste arbitrary text into an editor and for that editor to be using an encoding that can handle it. Or even for the web, if we actually took the Accept-Language header [0] seriously, that could also be a big savings. [0]: https://www.rfc-editor.org/rfc/rfc7231#section-5.3.5 https://www.rfc-editor.org/rfc/rfc7231#section-5.3.5

hnfong 5y ago

> UTF-16 is great for lots of East Asian languages, which billions of people use. In UTF-8, most of those languages require 3 bytes to encode a 32-bit codepoint, in UTF-16 they only ever need 2. This ends up being a huge savings. "Only ever" is an exaggeration. The vast majority of standard Chinese (aka. Putonghua) characters are in the BMP, but occasionally there are characters outside of it. In particular a lot of Hong Kong characters are outside of the BMP, and before emoji forced software to fix things, systems claiming to support UTF-16 (but were instead UCS-2) kept failing for say 1% of the characters. I've had much better luck with UTF-8 systems than purported UTF-16 ones. The problem with UTF-16 is not necessarily in its inherent technical issues, but rather, it's how often systems with UCS-2 support just claim they support UTF-16, and leave users/developers with a crappy experience when working with characters outside the BMP. There's also the encoding agnosticism thing -- with UTF-16 you run into issues about byte order marks (with or without), endian problems, and generally incompatibility with ASCII. As I said, the situation probably improved quite a bit after emoji became popular. But if I had the choice, I'd choose UTF-8 over UTF-16 any time. Also, although it's generally true that you "can't store UTF-8 in something expecting ASCII", in practice a lot of systems are somewhat more graceful than that. For example, if I grep a UTF-8 text file, grep doesn't need to know the text encoding if all you're trying to find is an ASCII string. Similarly it's possible to edit a UTF-8 text file as an ASCII file if the editor preserves the "binary" bits. In conclusion, "different encodings are good at different things" is only true when considered in isolation. The historical baggage of UCS-2, and separately, of ASCII-compatibility tilts the balance strongly in favor of UTF-8 IMHO. The alleged 33% saving in Chinese text doesn't really matter that much in the grand scheme of things -- which is perhaps why people use UTF-8 in spite of the "advantages" you mentioned.

camgunz 5y ago
Yeah I mean, I broadly agree with what you're saying. Text encoding is a mess, and pragmatic solutions are needed. And when jamming text in various places, jamming UTF-8 is typically more successful than jamming in UTF-16, because of NULL bytes and the shortsightedness of UCS-2. But I think that 99% of the things you have to do for UTF-8 (validate, encode/decode, no indexing) you have to do for UTF-16, and if we're arguing for systems that are UTF-aware, UTF-16 does great. If we're arguing for trying to jam encoded bytes into systems, well it's a crapshoot. UTF-8 being better at winning that crapshoot doesn't feel like a recipe for robustness to me, and certainly doesn't strike me as a perfect solution. OOB configuration does, though. > For example, if I grep a UTF-8 text file, grep doesn't need to know the text encoding if all you're trying to find is an ASCII string. Well, this is a weird example. grep uses locales, so yeah you're probably fine if you're grepping for ASCII using an ASCII-compatible locale. But if you're not, then what you feed to grep has to be encoded in your locale's encoding (I think). So I think this is actually an example in favor of "this should be OOB configurable", and an example of "ASCII-compatibility lulls into a false sense of security".

account42 5y ago

> You might think "but BOMs are super evil." Checking a BOM is extremely, extremely easy. Checking the BOM is not enough, you also have to handle it. Non-LE BOMs (like surrogates, at least before the emojipocalypse) are rare enough that many "UTF-16" based tools simply only support UTF-16LE. They are also stupid because you need to know that you are dealing with UTF-16LE or UTF-16BE in the first place - either through heuristics or because it is specidified and then you can also guess/specify the byte order. And like surrogates, the BOM is another wasted Unicode character that is only needed for the UTF-16 mess. > Furthermore, you don't get to bail out of checking anything just by using UTF-8, you have to check to ensure you have _valid_ UTF-8. That's right, you gotta scan the whole bytestream anyway, so you may as well just check the 2-byte BOM at the beginning too. You don't need validation for most UTF-8 tasks, GIGO is often more reasonable for things entered by humans. > you definitely can't store UTF-8 in something expecting ASCII. Unix tools would disagree. $ echo "Hello Wörld!" | tr '!' '?' Hello Wörld? Many tools only care about finding substrings. UTF-8 gurantees if the byte sequence making up the encoding of a Unicode character (or sequence) appears in valid UTF-8 encoded text then it also decodes to that Unicode sequence. > Practically all HTML is gzipped Not when it is being parsed it isn't.

camgunz 5y ago
> Non-LE BOMs (like surrogates, at least before the emojipocalypse) are rare enough that many "UTF-16" based tools simply only support UTF-16LE. Gymnastics like this always occur when you don't know the encoding. If you didn't know you were getting UTF-8, welcome to heuristics or whatever. > And like surrogates, the BOM is another wasted Unicode character that is only needed for the UTF-16 mess. This isn't a big deal at all. > You don't need validation for most UTF-8 tasks, GIGO is often more reasonable for things entered by humans. You can't build robust systems this way. You always have to validate, and in order to do that, you gotta know what encoding you're getting. > Unix tools would disagree. Most of those use locales, or they don't expect ASCII they expect bytes. Again you can get lucky with this, but you can't build a robust system with luck. > Many tools only care about finding substrings. You have to scan through the string or save offsets (after an initial scan) either way, UTF-8 or 16. > Not when it is being parsed it isn't. Fair point!