4 ms·

I remember this hoopla. And windows wasn't the only victim of using "two bytes for a char" for unicode support. Java fell in this trap too. The team was split

I remember this hoopla. And windows wasn't the only victim of using "two bytes for a char" for unicode support. Java fell in this trap too. The team was split in deciding "whether to waste a single byte per character for string" -- that's in their words, and then decided to rather be ready for the future. That's in 1995. And then everyone later watched UTF-8 take over the world. The PHP team also wasted their version 6 and a few years of development effort trying to support multi byte chars by ditching all previous code. And then failed. There was never a version 6. They continued with the old code base with version 7. Why didn't they go the UTF-8 round? May be Unicode consortium was late to introduce it. Maybe something else.

Cloudef 4y ago
UTF-8 was plan9's invention, I don't think anybody saw it coming.
- gpderetta 4y ago
  For more details: Rob Pike & Ken Thompson invented UTF-8 when prompted by X/Open (Not sure if X/Open and POSIX were already merged at that point) and they first implemented the proof of concept on plan9 (https://www.cl.cam.ac.uk/~mgk25/ucs/utf-8-history.txt https://www.cl.cam.ac.uk/~mgk25/ucs/utf-8-history.txt).
  - deleted 4y ago
    [deleted]
mananaysiempre 4y ago
> Why didn't they go the UTF-8 round? The original Unicode marketing / position statement from 1988[1] may provide a clue: “In the Unicode system, a simple unambiguous fixed-length character encoding is integrated into a coherent overall architecture for text processing.” “Unicodes [sic] are the most straightforward multilingual generalization of ASCII codes: - Fixed length of character code (16 bits); [...]” “Are 16 bits [...] sufficient to encode all characters of all the world’s scripts? [...] Yes.” “[A] fixed length-encoding is flat-out simple, with all the blessings attendant upon that virtue.” Etc., etc. The hypothetical possibility of more than 2^16 characters was introduced in Unicode 2.0 (1996), while actual such characters didn’t appear until Unicode 3.0 (1999). Windows NT shipped in 1993, OpenStep in 1994, Java and JavaScript in 1995. UTF-8 was presented at USENIX in January 1993; a contemporary exposition[2] says that “the 4[!], 5 and 6 byte sequences are only there for political reasons” (presumably referring to the fact that Unicode committed to 2^16 code points while the new, Unicode-compatible draft of ISO 10646 stuck with 2^31). [1] https://unicode.org/history/unicode88.pdf https://unicode.org/history/unicode88.pdf [2] https://www.cl.cam.ac.uk/~mgk25/ucs/utf-8-history.txt https://www.cl.cam.ac.uk/~mgk25/ucs/utf-8-history.txt
- zasdffaa 4y ago
  Do you (or anyone) have some idea why anyone could possibly have thought 16 bits would be enough? Many decisions are bad in hindsight but surely no hindsight was needed for that.
  - mananaysiempre 4y ago
    Nope. But on reflection, I can’t really tell if it was really that dumb of an idea. If you look at the text following the “Yes” quote, you’ll find that “all characters” is carefully defined to mean ”all characters in current use from commercially non-negligible scripts”. Compared to the current definition of “all characters we have reasonable evidence have ever been used for natural-language interchange”, it doesn’t sound as noble, but would also exclude a number of large-repertoire sets (Tangut and pre-modern Han ideograms, Yi syllables, hieroglyphs, cuneiform). Remove the requirement for 1:1 code point mapping with legacy sets, and you could conceivably throw out precomposed Hangul as well. (Precomposed European scripts too, if you want, but that wouldn’t net you eleven thousand codepoints.) At that point the question seems to come down to Han characters: the union of all government-mandated education standards (unified) would come down well below ten thousand characters, but how well does that number correspond to the number of characters people actually need? One potential source of death is uncommon characters people really, really want (proper names), but overall, I don’t know, you’d probably need a CJKV expert to tell. To me, neither answer seems completely implausible. On the other hand, it’s also unclear that a constant-width encoding would really be all that valuable. Most of the time, you are either traversing all code points in sequence or working with larger units such as combining-character sequences or graphemes, so aside from buffer truncation issues constant width does not really help all that much. But that’s an observation that took more than a decade of Unicode implementations to crystallize. It is certainly annoying how large and sparse the lookup tables needed to implement a current version of Unicode are—enough that you need three levels in your radix tree and not two—but if you aren’t doing locales it’s still a question of at most several dozens of kilobytes, not really a deal breaker these days. Perhaps that’s not too much of a cost for not marginalizing users of obscure languages and keeping digitized historical text representable in the common format.
- 4y ago
deleted 4y ago
[deleted]
krackers 4y ago
Cocoa (NextStep) strings use utf-16 as well, right?

Cloudef 4y ago

UTF-8 was plan9's invention, I don't think anybody saw it coming.

gpderetta 4y ago
For more details: Rob Pike & Ken Thompson invented UTF-8 when prompted by X/Open (Not sure if X/Open and POSIX were already merged at that point) and they first implemented the proof of concept on plan9 (https://www.cl.cam.ac.uk/~mgk25/ucs/utf-8-history.txt https://www.cl.cam.ac.uk/~mgk25/ucs/utf-8-history.txt).
- deleted 4y ago
  [deleted]

mananaysiempre 4y ago

> Why didn't they go the UTF-8 round? The original Unicode marketing / position statement from 1988[1] may provide a clue: “In the Unicode system, a simple unambiguous fixed-length character encoding is integrated into a coherent overall architecture for text processing.” “Unicodes [sic] are the most straightforward multilingual generalization of ASCII codes: - Fixed length of character code (16 bits); [...]” “Are 16 bits [...] sufficient to encode all characters of all the world’s scripts? [...] Yes.” “[A] fixed length-encoding is flat-out simple, with all the blessings attendant upon that virtue.” Etc., etc. The hypothetical possibility of more than 2^16 characters was introduced in Unicode 2.0 (1996), while actual such characters didn’t appear until Unicode 3.0 (1999). Windows NT shipped in 1993, OpenStep in 1994, Java and JavaScript in 1995. UTF-8 was presented at USENIX in January 1993; a contemporary exposition[2] says that “the 4[!], 5 and 6 byte sequences are only there for political reasons” (presumably referring to the fact that Unicode committed to 2^16 code points while the new, Unicode-compatible draft of ISO 10646 stuck with 2^31). [1] https://unicode.org/history/unicode88.pdf https://unicode.org/history/unicode88.pdf [2] https://www.cl.cam.ac.uk/~mgk25/ucs/utf-8-history.txt https://www.cl.cam.ac.uk/~mgk25/ucs/utf-8-history.txt

zasdffaa 4y ago
Do you (or anyone) have some idea why anyone could possibly have thought 16 bits would be enough? Many decisions are bad in hindsight but surely no hindsight was needed for that.
- mananaysiempre 4y ago
  Nope. But on reflection, I can’t really tell if it was really that dumb of an idea. If you look at the text following the “Yes” quote, you’ll find that “all characters” is carefully defined to mean ”all characters in current use from commercially non-negligible scripts”. Compared to the current definition of “all characters we have reasonable evidence have ever been used for natural-language interchange”, it doesn’t sound as noble, but would also exclude a number of large-repertoire sets (Tangut and pre-modern Han ideograms, Yi syllables, hieroglyphs, cuneiform). Remove the requirement for 1:1 code point mapping with legacy sets, and you could conceivably throw out precomposed Hangul as well. (Precomposed European scripts too, if you want, but that wouldn’t net you eleven thousand codepoints.) At that point the question seems to come down to Han characters: the union of all government-mandated education standards (unified) would come down well below ten thousand characters, but how well does that number correspond to the number of characters people actually need? One potential source of death is uncommon characters people really, really want (proper names), but overall, I don’t know, you’d probably need a CJKV expert to tell. To me, neither answer seems completely implausible. On the other hand, it’s also unclear that a constant-width encoding would really be all that valuable. Most of the time, you are either traversing all code points in sequence or working with larger units such as combining-character sequences or graphemes, so aside from buffer truncation issues constant width does not really help all that much. But that’s an observation that took more than a decade of Unicode implementations to crystallize. It is certainly annoying how large and sparse the lookup tables needed to implement a current version of Unicode are—enough that you need three levels in your radix tree and not two—but if you aren’t doing locales it’s still a question of at most several dozens of kilobytes, not really a deal breaker these days. Perhaps that’s not too much of a cost for not marginalizing users of obscure languages and keeping digitized historical text representable in the common format.
4y ago

deleted 4y ago

[deleted]

krackers 4y ago

Cocoa (NextStep) strings use utf-16 as well, right?