9 ms·
If only Windows, Java and JavaScript could also move away from internal usage of UTF-16, it's purely a legacy format and the worst of both worlds (UTF-32 and UT
If only Windows, Java and JavaScript could also move away from internal usage of UTF-16, it's purely a legacy format and the worst of both worlds (UTF-32 and UTF-8). Even worse is that unicode itself, which should in theory be a list of codes for glyphs, modifiers and other script related values, that's independent of encoding, had to have some codes reserved for "surrogates" for the UTF-16 encoding anyway. UTF-8 doesn't need such a thing...
Besides the surrogate characters there are also some other noncharacters: https://www.unicode.org/faq/private_use.html#noncharacters https://www.unicode.org/faq/private_use.html#noncharacters
Because of modifier characters, control characters like for bidi, stuff like soft-hyphens and ligatures, locale-dependent semantics (upper/lowercase, collation etc.), the general discordance between glyphs and characters, and so on and so forth, Unicode is so complex, and in general always requires careful processing of code point (or code unit) sequences, that honestly the surrogate encoding doesn’t make that much of a difference. It’s just an additional wrinkle in a sea of wrinkles.
I still find the surrogates different. Bidi, private use, ligatures, ... are script or locale related.
Unicode uses numeric values from 0 to 1112063. You can invent all kinds of methods to encode numbers from 0 to 1112063 (variable length, fixed length, decimal, hexadecimal, anything else). But most ways I can think of to encode these numbers, including variable length ones that would use 8 bit or 16 bit primitives, don't require me to actually reserve some of those to-be-encoded numbers themselves for a special meaning. Yet for UTF-16 they managed to do it. Imagine that all other encodings out there would also want to reserve some Unicode values for their own purpose!
As far as valid Unicode is concerned, you care about the distinction between code points and scalar values, and surrogates are the only difference.
Noncharacters can be represented in any Unicode encoding. Surrogates code points cannot, but can be found in unvalidated UTF-16 (which is most UTF-16).
Dealing with Unicode text semantically requires that you be aware of a great many factors such as those you name, but you don’t need to be aware of those for just storing and transferring Unicode text. But with surrogates, UTF-16 managed to break it for everyone: any part of the system that uses unvalidated UTF-16 can introduce errors that you must care about. Hence surrogates are a special kind of atrocity.
> unvalidated UTF-16 (which is most UTF-16).
I prefer the name WTF-16 [0]
[0] https://simonsapin.github.io/wtf-8/#ill-formed-utf-16 https://simonsapin.github.io/wtf-8/#ill-formed-utf-16
How is that different from unvalidated UTF-8? Both are variable-length encodings that require decoding and have failure cases.
To begin with, the difference is one of convention. UTF-16 is almost never checked for well-formedness: not a single one of the major users of UTF-16 that I am familiar with validates (Windows wide API functions, JavaScript, Qt); I don’t know of any system that exposes UTF-16 strings as some kind of main string type that does check well-formedness. UTF-8, on the other hand, is normally validated. Not always, to be sure, but normally. (Probably the most significant “largely UTF-8 but not necessarily” thing is *nix paths and other related syscalls—which is not dissimilar from the Windows situation, except that one uses 16-bit code units and the other 8-. This would seem to weaken my point, but my point is again strengthened by the number of languages that can’t equanimously interact with non-UTF-8 paths on Linux, which is not such a problem for non-UTF-16 paths on Windows, for all that the latter are very much rarer.)
But beyond that, the difference lies in Unicode having been compromised for the sake of one particular encoding, and now there are all kinds of places that have to deal with potentially ill-formed UTF-16, more that potentially ill-formed UTF-8, I would say. Look at what Python 3 did: its string type is not a Unicode string, but rather a sequence of Unicode code points. That is, it can encode surrogates. Blech.
Microsoft is making improvements in their UTF-8 support. Getting rid of the `W` APIs will take forever. Java and JavaScript are even more stuck with UTF-16.
I have an old saw about UTF-16 not being an irredeemable format and UTF-8 eating the world being bad, and I'm happy to dig it out again.
UTF-16 is great for lots of East Asian languages, which billions of people use. In UTF-8, most of those languages require 3 bytes to encode a 32-bit codepoint, in UTF-16 they only ever need 2. This ends up being a huge savings.
The main benefit of UTF-8 if you're say, Chinese, is interop. Everything else is worse.
You might think "but BOMs are super evil." Checking a BOM is extremely, extremely easy. Furthermore, you don't get to bail out of checking anything just by using UTF-8, you have to check to ensure you have _valid_ UTF-8. That's right, you gotta scan the whole bytestream anyway, so you may as well just check the 2-byte BOM at the beginning too.
You might also think "what about ASCII compatibility?" ASCII compatibility is an anti-feature. You should never be indexing into UTF strings (you always have to iterate, or save the results of an iteration), upper/lowercasing isn't addition/subtraction, etc. etc. You also can't just forget about encodings as a result--you can store ASCII in something expecting UTF-8, but you definitely can't store UTF-8 in something expecting ASCII. So if you're sniffing/decoding/tagging a format anyway, you may as well be agnostic.
You might also think "OK OK, you could be right, but what about HTML, which is mostly ASCII and would nearly double in size if it went from UTF-8 to UTF-16." Practically all HTML is gzipped, so the difference is pretty small, plus the majority of text isn't HTML (almost anything stored in a database, almost anything in a file on your computer, etc.)
Different encodings are good at different things. There's no one superior encoding for all uses. What we need is text encoding agnosticism.
---
In fairness, I will say I've heard that UTF-8 is pretty popular in countries with exactly the kind of languages I'm talking about, so the issue is mostly moot at this point. I just think UTF-16 gets a really bad rap, and I think we shouldn't just gloss over UTF-8 having won because it's good for European languages.
If you care about text size, you should compress your text; that'll save much more space, since it can optimize for what's actually used in the document.
> ASCII compatibility is an anti-feature.
ASCII compatibility is extremely useful if you're working with, for instance, filenames or programming languages. You can lex UTF-8 and handle separators like `/` or quotes like `"` and `'`, because those bytes can never occur otherwise.
Windows is slowly but surely moving to UTF-8.
https://docs.microsoft.com/en-us/windows/apps/design/globalizing/use-utf8-code-page https://docs.microsoft.com/en-us/windows/apps/design/globali...
> As of Windows Version 1903 (May 2019 Update), you can [...] use UTF-8 as the process code page.
> Until recently, Windows has emphasized "Unicode" -W variants over -A APIs. However, recent releases have used the ANSI code page and -A APIs as a means to introduce UTF-8 support to apps.
Of course if you use this, your app will only run on very recent Windows versions. But that's how it goes with OS features. We'll start reaping the benefits 10-20 years from now.
Is UTF-16 part of the JVM or the Java language, or both?
Sure it's useful, but it's not at all a guarantee. You can't assume the byte stream is UTF-8, and UTF-8 has a NUL in it.
Reading about it, filename encoding looks like a mess, GLib apparently assumes UTF-8 unless you set it with an environment variable, which I think is wild. Qt uses the locale, which feels reasonable (ignoring that locales are themselves bananas). But assuming UTF-8 and relying on ASCII compatibility and no NULs feels risky.
The main evil of UTF-16 is that it's usually not UTF-16, it's UCS-2, which is to say, a fixed-width format which has been shoehorned into a varible-width format to support non-BMP code points. As a consequence, systems that are built on "UTF-16" (i.e., really UCS-2) use designs centered around the ability to random-access Unicode strings despite the fact that doing so reliably isn't possible.
> ASCII compatibility is an anti-feature.
Actually, it's not. One of the main reasons it's important is...
> What we need is text encoding agnosticism
... that bit that you want. If you're supporting multiple encodings, you need a reliable way of signalling that encoding. And file metadata tends to be insufficiently reliable, so it ends up being in-band for text (e.g., HTML files). If all the encodings you need to support are ASCII-compatible, and the directive to set the charset is itself pure ASCII, that means it is perfectly safe to parse the file as ASCII to figure out how to parse the file. If the charset isn't ASCII-compatible... well, that historically has led to security vulnerabilities. Ergo, a non-ASCII-compatible charset has heightened security risks.
That said, my own experience when dealing with the mess that is charsets is that supporting multiple charsets is itself a painful process that isn't worth it.
> systems that are built on "UTF-16" (i.e., really UCS-2) use designs centered around the ability to random-access Unicode strings despite the fact that doing so reliably isn't possible.
100% this. In javascript you can accidentally cut characters in half. This lets you accidentally construct strings which have invalid encodings - and that can lead to all sorts of weird problems. In comparison, using rust's safe API, its impossible to construct a string which contains invalid unicode.
For example, if I put an emoji like this U+1F970 in javascript, I can then split it in half and put one half into a JSON string - like this: "\"\\ud83e\"". Javascript will treat this as "one character". Rust will treat this as an encoding error, and will refuse to parse it back into a string.
Another implication is that javascript's .length field looks useful, but its almost never what you actually want. How many unicode characters in this string: "\ud83e\udd70"? Trick question - even though javascript says the string's length is 2, there's only 1 unicode character in that string. How many bytes does this string (with JS length 1) "\u270C" take up over the wire? Javascript says the length is 1, but it takes 2 bytes to send in any encoding (utf8 or utf16).
What a mess.
Without some specific way of knowing what encoding you're dealing with, you can have this problem with any number of encodings you're working with (including ASCII suddenly having UTF-8 bytes in it). As soon as you're in a "assume an encoding and maybe be surprised", you're asking to be surprised.
Totally agree these discussions circle around to "you need some kind of OOB way to know what encoding you're dealing with", which puts us back to everything being encoding agnostic, with some way to configure it. Maybe that's locales, maybe that's a type 32-byte integer at the beginning, dunno.
> including ASCII suddenly having UTF-8 bytes
People have been dealing with 8-bit ASCII extensions for longer than Unicode has been around. It isn't overwhelmingly difficult to have 8-bit clean text handling without any awareness of the encoding.
No, the best way is not to have people specify what encoding they intend to use--it is instead to not give people a choice. When you give people a choice, given that one encoding is far dominant above all others, many people will start assuming that is the only viable choice regardless of any actual declarations to the contrary.
An example of the latter effect in action is the system charset of Linux and most other Unixen... in practice, virtually every system is using UTF-8 for things like pathnames and the like. However, the mechanism to indicate that this is the case is frequently enough unreliable that it's often less bug-prone to assume the system is UTF-8 anyways.
Of course, there's a few more advantages to always using UTF-8:
* You eliminate the code path of "I'm sorry, I can't save the text that you wrote," since every character is usable in UTF-8, which is a feature very rare outside of UTF-* charsets. And, honestly, that's a wonderful code path to eliminate.
* One of the features of UTF-8 is that virtually any text that isn't UTF-8 will fail to parse as UTF-8. This allows you to build systems that produce errors on bad input, which means that maybe, just maybe, the people who produced the broken input will discover their problem and fix it.
> In UTF-8, most of those languages require 3 bytes to encode a 32-bit codepoint, in UTF-16 they only ever need 2. This ends up being a huge savings.
Meanwhile in lots of other countries, UTF-8 lets you use 1 byte to store each character instead of 2; which is also a huge savings.
The question is, is it better for the world to pick one encoding and have all programs use it, or pick several encodings and switch between them depending on the context / country? Picking one encoding means we can reuse our code more easily. Picking multiple encodings means we get slightly smaller file sizes.
For my money, the best approach is to use UTF-8 everywhere. This lets us reuse code. Then if you're worried about file size on disk or over the wire, compress your text content with LZ4 or snappy or something. That'll halve the size of text for everyone and LZ4 is so fast its essentially free from a computational standpoint.
As this blog post demonstrates, quite a lot of code needs to deeply understand text encoding to work (be that UTF-8 or UTF-16 or whatever). Its a big savings in programming hours if we only have to write this code once.
I mostly disagree. We already have lots of code that deals with tons of different encodings, UTF-16 is pretty easy and there's lots of libraries already, etc. But if we ignore all that, your argument is against UTF-8, which succeeded UTF-16.
> The main benefit of UTF-8 if you're say, Chinese, is interop. Everything else is worse.
Well, for HTML it turns out to be a wash.
While Han characters are typically three bytes in UTF-8, markup is ASCII and so only one byte — whereas in UTF-17, both the character data and the markup are two bytes. When you add them together the average cost per character is more or less two bytes, whether your Chinese-content HTML is encoded as UTF-8 or UTF-17.
(LOL. How did I type "UTF-17", twice?)
This is probably true for HTML, but not true for lots of other text like whatever's stored in databases or on hard drives.
I'm Chinese, and I prefer UTF8 to UTF16.
> You might think "but BOMs are super evil." Checking a BOM is extremely, extremely easy. Furthermore, you don't get to bail out of checking anything just by using UTF-8, you have to check to ensure you have _valid_ UTF-8. That's right, you gotta scan the whole bytestream anyway, so you may as well just check the 2-byte BOM at the beginning too.
BOM breaks substring. Everything you take a substring, you need to prepend a BOM if you want to serialize it to somewhere else. If you want to concat two strings with BOM, you need to remove one of them. All of these are unnecessary pains.
> You might also think "OK OK, you could be right, but what about HTML, which is mostly ASCII and would nearly double in size if it went from UTF-8 to UTF-16." Practically all HTML is gzipped, so the difference is pretty small, plus the majority of text isn't HTML (almost anything stored in a database, almost anything in a file on your computer, etc.)
This just contradicts your own reasoning that UTF-16 is better for East Asians due to size savings.
Substrings just get the BOM too. EZ.
Re: HTML, people usually bring it up because it's a lot of ASCII that'll all double up under UTF-16. Mostly my argument is that most text isn't HTML, so size savings on other text files still matter.
That is a minor nuisance when dealing with raw bytes. Any sane language representation would probably store it as a separate field egen not on the wire.
I would probably do BOM, LUT, BYTES. Where LUT is a lookup table of indices for every 64 chars so that you could do o(1) random access for the vast majority of cases.
> I have an old saw about UTF-16 not being an irredeemable format and UTF-8 eating the world being bad, and I'm happy to dig it out again.
> UTF-16 is great for lots of East Asian languages, which billions of people use. In UTF-8, most of those languages require 3 bytes to encode a 32-bit codepoint, in UTF-16 they only ever need 2. This ends up being a huge savings.
That doesn't make UTF-16 not a terrible format, though. East Asian countries use better formats instead. For example, in China they use GB18030.
When your case for "UTF-16 is not an irredeemable format" is "it's better than UTF-8 at a task where nobody would use either format anyway", you're not making a strong case.
> When your case for "UTF-16 is not an irredeemable format" is "it's better than UTF-8 at a task where nobody would use either format anyway", you're not making a strong case.
Haha this is pretty fair; I know I'm being pedantic. But I wouldn't say any GB* encoding is better than UTF-16. Another commenter pointed out that you really want the ability to--say--paste arbitrary text into an editor and for that editor to be using an encoding that can handle it.
Or even for the web, if we actually took the Accept-Language header [0] seriously, that could also be a big savings.
[0]: https://www.rfc-editor.org/rfc/rfc7231#section-5.3.5 https://www.rfc-editor.org/rfc/rfc7231#section-5.3.5
> UTF-16 is great for lots of East Asian languages, which billions of people use. In UTF-8, most of those languages require 3 bytes to encode a 32-bit codepoint, in UTF-16 they only ever need 2. This ends up being a huge savings.
"Only ever" is an exaggeration. The vast majority of standard Chinese (aka. Putonghua) characters are in the BMP, but occasionally there are characters outside of it. In particular a lot of Hong Kong characters are outside of the BMP, and before emoji forced software to fix things, systems claiming to support UTF-16 (but were instead UCS-2) kept failing for say 1% of the characters.
I've had much better luck with UTF-8 systems than purported UTF-16 ones. The problem with UTF-16 is not necessarily in its inherent technical issues, but rather, it's how often systems with UCS-2 support just claim they support UTF-16, and leave users/developers with a crappy experience when working with characters outside the BMP. There's also the encoding agnosticism thing -- with UTF-16 you run into issues about byte order marks (with or without), endian problems, and generally incompatibility with ASCII.
As I said, the situation probably improved quite a bit after emoji became popular. But if I had the choice, I'd choose UTF-8 over UTF-16 any time.
Also, although it's generally true that you "can't store UTF-8 in something expecting ASCII", in practice a lot of systems are somewhat more graceful than that. For example, if I grep a UTF-8 text file, grep doesn't need to know the text encoding if all you're trying to find is an ASCII string. Similarly it's possible to edit a UTF-8 text file as an ASCII file if the editor preserves the "binary" bits.
In conclusion, "different encodings are good at different things" is only true when considered in isolation. The historical baggage of UCS-2, and separately, of ASCII-compatibility tilts the balance strongly in favor of UTF-8 IMHO. The alleged 33% saving in Chinese text doesn't really matter that much in the grand scheme of things -- which is perhaps why people use UTF-8 in spite of the "advantages" you mentioned.
Yeah I mean, I broadly agree with what you're saying. Text encoding is a mess, and pragmatic solutions are needed. And when jamming text in various places, jamming UTF-8 is typically more successful than jamming in UTF-16, because of NULL bytes and the shortsightedness of UCS-2.
But I think that 99% of the things you have to do for UTF-8 (validate, encode/decode, no indexing) you have to do for UTF-16, and if we're arguing for systems that are UTF-aware, UTF-16 does great. If we're arguing for trying to jam encoded bytes into systems, well it's a crapshoot. UTF-8 being better at winning that crapshoot doesn't feel like a recipe for robustness to me, and certainly doesn't strike me as a perfect solution. OOB configuration does, though.
> For example, if I grep a UTF-8 text file, grep doesn't need to know the text encoding if all you're trying to find is an ASCII string.
Well, this is a weird example. grep uses locales, so yeah you're probably fine if you're grepping for ASCII using an ASCII-compatible locale. But if you're not, then what you feed to grep has to be encoded in your locale's encoding (I think). So I think this is actually an example in favor of "this should be OOB configurable", and an example of "ASCII-compatibility lulls into a false sense of security".
> You might think "but BOMs are super evil." Checking a BOM is extremely, extremely easy.
Checking the BOM is not enough, you also have to handle it. Non-LE BOMs (like surrogates, at least before the emojipocalypse) are rare enough that many "UTF-16" based tools simply only support UTF-16LE. They are also stupid because you need to know that you are dealing with UTF-16LE or UTF-16BE in the first place - either through heuristics or because it is specidified and then you can also guess/specify the byte order.
And like surrogates, the BOM is another wasted Unicode character that is only needed for the UTF-16 mess.
> Furthermore, you don't get to bail out of checking anything just by using UTF-8, you have to check to ensure you have _valid_ UTF-8. That's right, you gotta scan the whole bytestream anyway, so you may as well just check the 2-byte BOM at the beginning too.
You don't need validation for most UTF-8 tasks, GIGO is often more reasonable for things entered by humans.
> you definitely can't store UTF-8 in something expecting ASCII.
Unix tools would disagree.
$ echo "Hello Wörld!" | tr '!' '?'
Hello Wörld?
Many tools only care about finding substrings. UTF-8 gurantees if the byte sequence making up the encoding of a Unicode character (or sequence) appears in valid UTF-8 encoded text then it also decodes to that Unicode sequence.
> Practically all HTML is gzipped
Not when it is being parsed it isn't.
> Non-LE BOMs (like surrogates, at least before the emojipocalypse) are rare enough that many "UTF-16" based tools simply only support UTF-16LE.
Gymnastics like this always occur when you don't know the encoding. If you didn't know you were getting UTF-8, welcome to heuristics or whatever.
> And like surrogates, the BOM is another wasted Unicode character that is only needed for the UTF-16 mess.
This isn't a big deal at all.
> You don't need validation for most UTF-8 tasks, GIGO is often more reasonable for things entered by humans.
You can't build robust systems this way. You always have to validate, and in order to do that, you gotta know what encoding you're getting.
> Unix tools would disagree.
Most of those use locales, or they don't expect ASCII they expect bytes. Again you can get lucky with this, but you can't build a robust system with luck.
> Many tools only care about finding substrings.
You have to scan through the string or save offsets (after an initial scan) either way, UTF-8 or 16.
> Not when it is being parsed it isn't.
Fair point!
> In other words, if Rust used UTF-16 internally it would behave exactly the same as you're describing.
This is a good point. I suppose the real problem here isn't UTF-16. Its that all these languages and systems assumed unicode would never grow past 65536 characters. They weren't designed to use UTF16. They were designed to use UCS2. It was only after unicode grew past UCS2's limit that they added awful hacks to their systems in order to (badly) support UTF16. The problems I'm complaining about aren't UTF16's fault. They're problems that arise as a result of the hacks these languages added in order to support surrogate pairs.
You could build a string type in rust today which used UTF16 strings internally. It would work fine, and be just as safe as the UTF8 strings in the standard library. I just really don't see the point. The verdict is in. UTF8 has won the war to be the world's interoperable format for unicode text. UTF8 is usually smaller than UTF16 (except apparently in some parts of Asia). And it doesn't need byte-order-marks or any of that guff. Its a relief that this point is basically settled.
I hope that a future generation of programmers can grow up without needing to understand UCS2, UTF-16, byte-order-marks, surrogate pairs or any of this junk.
> in practice, virtually every system is using UTF-8 for things like pathnames and the like.
I mean, *nix filenames are a pretty weird case where you have these requirements of 0x2F always being a path separator and 0x00 always meaning "end of path". In truth, we can't guarantee anything about filename encoding other than those two things. So, for example, you might be dealing with UTF-8 or EUC-JP. In practice (as a sibling comment pointed out) all you usually need to do is scan for 0x2F and stop at 0x00, so this works out OK, but if you ever need to do anything else, welcome to guessing the encoding based on locales, environment variables, or other (OOB) configurations.
> However, the mechanism to indicate that this is the case is frequently enough unreliable that it's often less bug-prone to assume the system is UTF-8 anyways.
My argument, which I guess looking at it I've not articulated at all in this thread, is that UTF-8's ASCII compatibility and *nix filename compatibility have allowed us to be slack in fixing this. It oughta be possible to configure an encoding when creating a filesystem, just like creating a database. There's nothing inherently buggy about configuring an encoding, we just haven't gotten our systems programming act together, and UTF-8's amenability to the existing situation is probably at fault here.
> since every character is usable in UTF-8
I guess you mean like, vs. EUC-JP or something? Sure but UTF-16 is fine here too.
> One of the features of UTF-8 is that virtually any text that isn't UTF-8 will fail to parse as UTF-8.
IDK the "brittleness" of an encoding feels hard to quantify. Besides, I think if you're doing the (always required) validation step, this goes for any input. Look at things like weirdo MySQL/Oracle/MSSQL database encodings for example.
---
I should say I see where you're coming from. But I think we'd be in just as good a place if we specified a filename/text encoding at filesystem creation (OOB configuration) and quit relying on magic 0x2F/0x00 bytes. I don't know why this makes people tear their hair out as some kind of impossible problem to solve.