3 ms·
While they could, in theory, hide the actual encoding, the APIs all refer to UTF-16 code units as "characters;" so while they could use UTF-8 as the internal en
by lambda 14y ago
While they could, in theory, hide the actual encoding, the APIs all refer to UTF-16 code units as "characters;" so while they could use UTF-8 as the internal encoding, you need to use and understand UTF-16 in order to interact with them properly. When you ask for the "length" of a string, you are told the number of UTF-16 code units. When you get a character at an index, you get the UTF-16 code unit. That's what I mean when I say the APIs use UTF-16; everything in the API that deals with individual "chracters" is actually referring to UTF-16 code units.
The same is true of JavaScript; while you could technically implement the strings however you want, the APIs are all oriented around UTF-16 code units. And the Windows API as well, is all built around UTF-16 code units.
The problem with all of these APIs is that they make the mistake of conflating characters and code units. They all make the assumption that a character consists of a single, fixed width integer, of some given size (16 bits in the case of UTF-16). It is better to distinguish between indexing in code units (such as bytes in UTF-8 or 16 bit integers in UTF-16) and indexing in code points, or glyphs, or whatever higher level concept you are talking about. Really, for anything higher than the code unit level, you should be dealing with variable-length strings, and not try to force that into fixed length units. With UTF-8, there's no temptation to treat a single code unit as being an independently meaningful entity, as that assumption breaks down as soon as you get past the ASCII range; while with UTF-16, it's easy to make that mistake, since it holds true for everything in the Basic Multilingual Plane, which contains most characters you're likely to encounter on a day to day basis.