3 ms·

> Microsoft must have thought to themselves in the early 90's: "Gee, we should support unicode... that means we need wider characters, right? Two bytes ought to

by ygra 11y ago

> Microsoft must have thought to themselves in the early 90's: "Gee, we should support unicode... that means we need wider characters, right? Two bytes ought to be enough. Ship it!" Except it was Unicode which was a 16-bit code back then which got changed into 21 bits with Unicode 2 which also morphed UCS-2 into UTF-16. This happened in 1996, when Windows NT already existed. Blaming Microsoft (and Sun, and Netscape, ...) to follow a standard is a bit out of place. Heck, while UTF-8 everywhere would be nice, things are still messy and various different encodings are rampant. At least UTF-16 is immediately recognisable, as opposed to all the legacy encodings.

ninkendo 11y ago
I suppose that's right. It's really the Unicode 1.0 standard itself that's to blame here, but the point still stands. It took them between 1.0 and 2.0 to actually include any implementation guidelines, but by then it was too late; the Microsofts and Suns of the world have already implemented a dead-end way of encoding the standard. But at least in Unicode 2.0 in 1996, the definitions they finally gave were UTF-7 and UTF-8. UTF-16 didn't become standard until 3.0 in 1999, when it became apparent that they couldn't take the first 5 years back, and too much software was already written assuming 16-bit characters. My point is that UTF-16 is a completely worst-of-both-worlds hack that nobody would implement in a vacuum, short of a need to maintain compatibility with a 2-byte character API. The only reason it exists is because people (the Consortium included) truly for a brief period thought a 2-byte character set would actually be enough, and then had to figure out a way to preserve compatibility later. Meanwhile the UNIX world thankfully seems to have gone straight from ASCII to UTF-8 without any awkward dead-end in the middle [1], because if you need to maintain backward compatibility with anything, it's a lot more useful (and space efficient) for that something to be ASCII. [1] Sure, there exists wide-character versions of all the posix standard APIs, but in practice all the other API's you're likely to use in UNIX land have converged around wrapping the 8-bit character versions.

by ygra 11y ago

ninkendo 11y ago

I suppose that's right. It's really the Unicode 1.0 standard itself that's to blame here, but the point still stands. It took them between 1.0 and 2.0 to actually include any implementation guidelines, but by then it was too late; the Microsofts and Suns of the world have already implemented a dead-end way of encoding the standard. But at least in Unicode 2.0 in 1996, the definitions they finally gave were UTF-7 and UTF-8. UTF-16 didn't become standard until 3.0 in 1999, when it became apparent that they couldn't take the first 5 years back, and too much software was already written assuming 16-bit characters. My point is that UTF-16 is a completely worst-of-both-worlds hack that nobody would implement in a vacuum, short of a need to maintain compatibility with a 2-byte character API. The only reason it exists is because people (the Consortium included) truly for a brief period thought a 2-byte character set would actually be enough, and then had to figure out a way to preserve compatibility later. Meanwhile the UNIX world thankfully seems to have gone straight from ASCII to UTF-8 without any awkward dead-end in the middle [1], because if you need to maintain backward compatibility with anything, it's a lot more useful (and space efficient) for that something to be ASCII. [1] Sure, there exists wide-character versions of all the posix standard APIs, but in practice all the other API's you're likely to use in UNIX land have converged around wrapping the 8-bit character versions.