8 ms·

Unicode 88 (1988) [pdf]

57 points by robin_reala 3y ago

krispyfi 3y ago
"2.1 Sufficiency of 16 bits" Someone should go back in time and tell them...
- netcruiser 3y ago
  "it is certainly possible, albeit uninteresting, to come up with an unreasonable definition of "character" such there are more than 65536 of them". They probably didn't foresee the "unreasonable" and "uninteresting" use of emojis.
  - krispyfi 3y ago
    I was thinking more of Han Unification, which we now realize was a bad idea, and to this day makes it difficult if not impossible to correctly render a block of text that contains characters in more than one CJK language.
- thristian 3y ago
  Their justification is: > Unicode aims in the first instance at the characters published in modern text (e.g. in the union of all newspapers and magazines printed in the world in 1988), whose number is far below 2¹⁴ = 16,384. ...and yeah, that's probably true. It wasn't until several years later that they joined forces with the ISO10646 project, which aimed to encode every character ever used by any human writing system, and 16 bits turned out not to be enough.
  - jandrese 3y ago
    Adding all of the dead languages bloated the system very quickly. Shame UCS2 turned out to be such a mistake. Beyond being not just big enough, trading off space for compute power turned out to be the wrong trade, CPUs increased much more rapidly than memory sizes and data especially transmission rates.
    - WalterBright 3y ago
      UCS2 was not a mistake. Unicode forgot its mission.
      - otabdeveloper4 3y ago
        The idea of fixed-length characters was the mistake. Unicode is fine.
        numpad0 3y ago
        24- or 32-bit fixed length would be probably fine… The issue is doing a unified codepoint right run into problems of accepting that zh-CN/zh-HK/zh-TW and ja-JP/kr-KR each uses slightly different definitions for every letters in common use, which is not just memory inefficient but also have to do with Taiwanese identity problem. Much easier to keep it to just zh-CN and zh-TW(recently renamed zh-Hans and zh-Hant) with zh-CN completely double defined with ja-JP, and kr-KR dealt as footnotes on ja-JP, which is what they did.
        lifthrasiir 3y ago
        > The issue is doing a unified codepoint right run into problems of accepting that zh-CN/zh-HK/zh-TW and ja-JP/kr-KR each uses slightly different definitions for every letters in common use, which is not just memory inefficient but also have to do with Taiwanese identity problem. I'm not sure what exactly you wanted to say, but all of six or seven variants of Han characters are substantially different from each other. Glyphwise there are three major clusters due to the different simplification history (of the lack thereof): PRC, Japan and everyone else. (Yes, Korean Hanja is much similar to traditional Chinese.) Even when Taiwan didn't exist at all there would have been three major clusters anyway. Also pedantry corner: ko-KR, not kr-KR. And zh-TW wasn't renamed to zh-Hant; it is valid to say zh-CN-Hant if PRC somehow wants to use traditional Chinese characters.
        numpad0 3y ago
  - WalterBright 3y ago
    It's a pity that Unicode forgot its purpose and added: 1. formatting 2. fonts 3. semantic meaning 4. multiple encodings for the same code point 5. you vote for my invented character and I'll vote for your invented character 6. multiple code points for the same glyph 7. an incomprehensible document describing all this nonsense 8. an impossibility to implement it all
    - lifthrasiir 3y ago
      The only fault of Unicode is 4 (and we have eventually solved it anyway). Everything else is a natural complication from human writing systems.
      - WalterBright 3y ago
        Fonts, for example. That is a style, and should be set by the rendering instructions (like CSS) not the code point. Semantics: there's a code point for the letter 'a' and another code point for the mathematical symbol 'a', although the glyphs are identical. Unicode points should not have semantic meaning - semantic meaning comes from the context in which the points are used. You know, like the text in a printed book. And so on, for the other bullet points.
        lifthrasiir 3y ago
        I have already explained why mathematical alphanumeric symbols exist in the first place [1]. They are an extremely small portion of Unicode; you need way more examples. By the way, if we indeed decide to encode characters according to glyphs, we would have to distinguish a double-storey `a` from a single-storey `a` (among others). They have clearly different glyphs, so why shouldn't we? Think about that. [1] https://news.ycombinator.com/item?id=28294032 https://news.ycombinator.com/item?id=28294032
        WalterBright 3y ago
        > I have already explained But somehow mathematical books have done just fine without any need for such encodings. Meaning in written language is inferred from the context, not the symbol. This is because meanings are infinite in variety, and symbols are finite. > Think about that I did. That's what fonts are for. Fonts come in endless variety, and so cannot be encoded into Unicode. They are an orthogonal property of rendered text. And so is formatting - it's an orthogonal property.
  - wolverine876 3y ago
    It's not true if you count current languages of East Asia, such as Chinese. Or does the document address that omission somehow? EDIT: Skimming it, I don't see how the issue is addressed, except in a diagram that says, * Modern-Use Ideographs: 54x256 = 13,824 * Added Ideographs: 64x256 = 16,384 Also it talks about deduplicating identical characters in Chinese, the Japanese alphabets, and Korean (Hangul). More interesting is the full explanation of modern use: Distinction of "moderm-use" characters: Unicode gives higher priority to ensuring utility for the future than to preserving past antiquities. Unicode aims in the first instance at the characters published in modern text (e.g. in the union of all newspapers and magazines printed in the world in 1988), whose number is undoubtedly far below 2^14 = 16,384. Beyond those modern-use characters, all others may be defined to be obsolete or rare; these are better candidates for private-use registration than for congesting the public list of generally-useful Unicodes. It's very interesting, and revealing about the author, to define use as newspaper and magazine publishing.
    - lifthrasiir 3y ago
      Unicode 88 (wrongly) assumed that Han characters in current use are small enough (see p. 3 for specifics). This turned out to be massively incorrect because they have an extremely long tail. Same goes for Hangul. EDIT: > Beyond those modern-use characters, all others may be defined to be obsolete or rare; these are better candidates for private-use registration than for congesting the public list of generally-useful Unicodes. I do believe that the initial estimation was not very off. IICore 2.2 [1] contains 9,810 Han characters across all relevant countries and even assuming that every character should be encoded in two ways (to account for simplified-traditional splits) and IICore had some significant omissions we could have just encoded ~40,000 characters. If everything went well according to their plan. The reality however was that PUA for rare characters was an old concept---pretty much every East Asian character set has one---and proven so ineffective. PUA is by definition not interchangable, but rare characters are still in use and they have to be interchanged anyway. So people avoided them like a plague and argued for bigger character sets instad, East Asian character sets had to be ballooned up, and at the point of Unicode they became so large that a naive vision of Unicode 88 no longer worked. [1] https://en.wikipedia.org/wiki/International_Ideographs_Core https://en.wikipedia.org/wiki/International_Ideographs_Core
wolverine876 3y ago
In 1978, the initial proposal for a set of "Universal Signs" was made by Bob Belleville at Xerox PARC. Many persons contributed ideas to the development of a new encoding design. Beginning in 1980, these efforts evolved into the Xerox Character Code Standard (XCCS) by the present author, a multilingual encoding which has been maintained by Xerox as an internal corporate standard since 1982, through the efforts of Ed Smura, Ron Pellar, and others. Unicode arose as the result of eight years of working experience with XCCS. Its fundamental diferences rom XCCS were proposed oy Peter Fenwick and Dave Opstad (pure 16-bit codes), and by Lee Collins (ideographic character unification), Unicode retains the many features of XCCS whose utility have been proved over the years in an international line of communicating multilingual system products.
rurban 3y ago
Maybe when I have some time we'll have to go back to a practical and sane 16bit Unicode, at least for programming. Not for archeology. Only the common scripts, proper identifiers (without the bugs unfixed for decades), forbidden mark combinations for which a single symbol exists. Of course without emoji, but also without all the silly supplementary planes.
- flohofwoe 3y ago
  These are the situations where I restrict myself to 7-bit ASCII (and before the torches and pitchforks arrive, I'm not a native English speaker).
oaiey 3y ago
Title should be Unicode 88 (1998). The paper is a 10 years article
- cjs_ac 3y ago
  It's a 1988 article; the linked version is a scan of a 1998 reprinting.
robin_reala 3y ago
“A sequence of Unicodes (e.g. text file, string, or character stream) is called ‘Unitext’.” I like this, shame it didn’t catch on.
- hakfoo 3y ago
  It's a disappointment that we never used different conceptual "types" for text files with different encodings. You can sort of get there with MIME types and encoding data attached, but imagine if you had a filename convention like '.txt.ASCII' or '.txt.UTF8LE', or '.txt.ISO8859-1' so software would know exactly what to expect rather than trying to sniff byte-order marks or hueristics for guessing encoding and mojibake fixes.
  - wolfgang42 3y ago
    I think if someone had tried to make encoding-in-filename a common convention it would have just resulted in people writing “.txt.ASCII” without understanding what that meant, and we’d have the same charset sniffing we have now plus another mess on top. (For example: when you have a “config.ini.UTF-16” and a “config.ini.WINDOWS-1252”, which should your program prefer?) --- An anecdote: I was writing some code to send data to another company as an XML file. When I sent them a test file to validate, I got a complaint that it didn’t declare its encoding as UTF-8. “Yes, our database uses UCS-2 for exports,” I replied. “Why, can you not read it?” “Our system imported it just fine,“ they explained, “but it needs to say ‘<?xml version="1.0" encoding="UTF-8"?>’ at the top and when I looked at it with a text editor yours says something else there instead.” “Yes, it doesn’t say UTF-8 because it’s not UTF-8. Sounds like your system is OK with that, though?” “Yes, the rest of the file is fine, but we can’t move forward with go-live until you conform to the example in our implementation guide...” Actually changing the encoding proved difficult (for reasons I no longer recall), so I ended up shoving in a string replacement to make the file claim the wrong encoding, along with an apologetic comment; their end apparently did sniffing (and the BOM was unambiguous) so this worked out fine, but it goes to show just how confused people can get about what a character encoding a file actually is.
  - pwdisswordfishc 3y ago

57 points by robin_reala 3y ago

krispyfi 3y ago

"2.1 Sufficiency of 16 bits" Someone should go back in time and tell them...

netcruiser 3y ago
"it is certainly possible, albeit uninteresting, to come up with an unreasonable definition of "character" such there are more than 65536 of them". They probably didn't foresee the "unreasonable" and "uninteresting" use of emojis.
- krispyfi 3y ago
  I was thinking more of Han Unification, which we now realize was a bad idea, and to this day makes it difficult if not impossible to correctly render a block of text that contains characters in more than one CJK language.
thristian 3y ago
Their justification is: > Unicode aims in the first instance at the characters published in modern text (e.g. in the union of all newspapers and magazines printed in the world in 1988), whose number is far below 2¹⁴ = 16,384. ...and yeah, that's probably true. It wasn't until several years later that they joined forces with the ISO10646 project, which aimed to encode every character ever used by any human writing system, and 16 bits turned out not to be enough.
- jandrese 3y ago
  Adding all of the dead languages bloated the system very quickly. Shame UCS2 turned out to be such a mistake. Beyond being not just big enough, trading off space for compute power turned out to be the wrong trade, CPUs increased much more rapidly than memory sizes and data especially transmission rates.
  - WalterBright 3y ago
    UCS2 was not a mistake. Unicode forgot its mission.
    - otabdeveloper4 3y ago
      The idea of fixed-length characters was the mistake. Unicode is fine.
      - numpad0 3y ago
        24- or 32-bit fixed length would be probably fine… The issue is doing a unified codepoint right run into problems of accepting that zh-CN/zh-HK/zh-TW and ja-JP/kr-KR each uses slightly different definitions for every letters in common use, which is not just memory inefficient but also have to do with Taiwanese identity problem. Much easier to keep it to just zh-CN and zh-TW(recently renamed zh-Hans and zh-Hant) with zh-CN completely double defined with ja-JP, and kr-KR dealt as footnotes on ja-JP, which is what they did.
        lifthrasiir 3y ago
        > The issue is doing a unified codepoint right run into problems of accepting that zh-CN/zh-HK/zh-TW and ja-JP/kr-KR each uses slightly different definitions for every letters in common use, which is not just memory inefficient but also have to do with Taiwanese identity problem. I'm not sure what exactly you wanted to say, but all of six or seven variants of Han characters are substantially different from each other. Glyphwise there are three major clusters due to the different simplification history (of the lack thereof): PRC, Japan and everyone else. (Yes, Korean Hanja is much similar to traditional Chinese.) Even when Taiwan didn't exist at all there would have been three major clusters anyway. Also pedantry corner: ko-KR, not kr-KR. And zh-TW wasn't renamed to zh-Hant; it is valid to say zh-CN-Hant if PRC somehow wants to use traditional Chinese characters.
        numpad0 3y ago
- WalterBright 3y ago
  It's a pity that Unicode forgot its purpose and added: 1. formatting 2. fonts 3. semantic meaning 4. multiple encodings for the same code point 5. you vote for my invented character and I'll vote for your invented character 6. multiple code points for the same glyph 7. an incomprehensible document describing all this nonsense 8. an impossibility to implement it all
  - lifthrasiir 3y ago
    The only fault of Unicode is 4 (and we have eventually solved it anyway). Everything else is a natural complication from human writing systems.
    - WalterBright 3y ago
      Fonts, for example. That is a style, and should be set by the rendering instructions (like CSS) not the code point. Semantics: there's a code point for the letter 'a' and another code point for the mathematical symbol 'a', although the glyphs are identical. Unicode points should not have semantic meaning - semantic meaning comes from the context in which the points are used. You know, like the text in a printed book. And so on, for the other bullet points.
      - lifthrasiir 3y ago
        I have already explained why mathematical alphanumeric symbols exist in the first place [1]. They are an extremely small portion of Unicode; you need way more examples. By the way, if we indeed decide to encode characters according to glyphs, we would have to distinguish a double-storey `a` from a single-storey `a` (among others). They have clearly different glyphs, so why shouldn't we? Think about that. [1] https://news.ycombinator.com/item?id=28294032 https://news.ycombinator.com/item?id=28294032
        WalterBright 3y ago
        > I have already explained But somehow mathematical books have done just fine without any need for such encodings. Meaning in written language is inferred from the context, not the symbol. This is because meanings are infinite in variety, and symbols are finite. > Think about that I did. That's what fonts are for. Fonts come in endless variety, and so cannot be encoded into Unicode. They are an orthogonal property of rendered text. And so is formatting - it's an orthogonal property.
- wolverine876 3y ago
  It's not true if you count current languages of East Asia, such as Chinese. Or does the document address that omission somehow? EDIT: Skimming it, I don't see how the issue is addressed, except in a diagram that says, * Modern-Use Ideographs: 54x256 = 13,824 * Added Ideographs: 64x256 = 16,384 Also it talks about deduplicating identical characters in Chinese, the Japanese alphabets, and Korean (Hangul). More interesting is the full explanation of modern use: Distinction of "moderm-use" characters: Unicode gives higher priority to ensuring utility for the future than to preserving past antiquities. Unicode aims in the first instance at the characters published in modern text (e.g. in the union of all newspapers and magazines printed in the world in 1988), whose number is undoubtedly far below 2^14 = 16,384. Beyond those modern-use characters, all others may be defined to be obsolete or rare; these are better candidates for private-use registration than for congesting the public list of generally-useful Unicodes. It's very interesting, and revealing about the author, to define use as newspaper and magazine publishing.
  - lifthrasiir 3y ago
    Unicode 88 (wrongly) assumed that Han characters in current use are small enough (see p. 3 for specifics). This turned out to be massively incorrect because they have an extremely long tail. Same goes for Hangul. EDIT: > Beyond those modern-use characters, all others may be defined to be obsolete or rare; these are better candidates for private-use registration than for congesting the public list of generally-useful Unicodes. I do believe that the initial estimation was not very off. IICore 2.2 [1] contains 9,810 Han characters across all relevant countries and even assuming that every character should be encoded in two ways (to account for simplified-traditional splits) and IICore had some significant omissions we could have just encoded ~40,000 characters. If everything went well according to their plan. The reality however was that PUA for rare characters was an old concept---pretty much every East Asian character set has one---and proven so ineffective. PUA is by definition not interchangable, but rare characters are still in use and they have to be interchanged anyway. So people avoided them like a plague and argued for bigger character sets instad, East Asian character sets had to be ballooned up, and at the point of Unicode they became so large that a naive vision of Unicode 88 no longer worked. [1] https://en.wikipedia.org/wiki/International_Ideographs_Core https://en.wikipedia.org/wiki/International_Ideographs_Core

wolverine876 3y ago

In 1978, the initial proposal for a set of "Universal Signs" was made by Bob Belleville at Xerox PARC. Many persons contributed ideas to the development of a new encoding design. Beginning in 1980, these efforts evolved into the Xerox Character Code Standard (XCCS) by the present author, a multilingual encoding which has been maintained by Xerox as an internal corporate standard since 1982, through the efforts of Ed Smura, Ron Pellar, and others. Unicode arose as the result of eight years of working experience with XCCS. Its fundamental diferences rom XCCS were proposed oy Peter Fenwick and Dave Opstad (pure 16-bit codes), and by Lee Collins (ideographic character unification), Unicode retains the many features of XCCS whose utility have been proved over the years in an international line of communicating multilingual system products.

rurban 3y ago

Maybe when I have some time we'll have to go back to a practical and sane 16bit Unicode, at least for programming. Not for archeology. Only the common scripts, proper identifiers (without the bugs unfixed for decades), forbidden mark combinations for which a single symbol exists. Of course without emoji, but also without all the silly supplementary planes.

flohofwoe 3y ago
These are the situations where I restrict myself to 7-bit ASCII (and before the torches and pitchforks arrive, I'm not a native English speaker).

oaiey 3y ago

Title should be Unicode 88 (1998). The paper is a 10 years article

cjs_ac 3y ago
It's a 1988 article; the linked version is a scan of a 1998 reprinting.

robin_reala 3y ago

“A sequence of Unicodes (e.g. text file, string, or character stream) is called ‘Unitext’.” I like this, shame it didn’t catch on.

hakfoo 3y ago
It's a disappointment that we never used different conceptual "types" for text files with different encodings. You can sort of get there with MIME types and encoding data attached, but imagine if you had a filename convention like '.txt.ASCII' or '.txt.UTF8LE', or '.txt.ISO8859-1' so software would know exactly what to expect rather than trying to sniff byte-order marks or hueristics for guessing encoding and mojibake fixes.
- wolfgang42 3y ago
  I think if someone had tried to make encoding-in-filename a common convention it would have just resulted in people writing “.txt.ASCII” without understanding what that meant, and we’d have the same charset sniffing we have now plus another mess on top. (For example: when you have a “config.ini.UTF-16” and a “config.ini.WINDOWS-1252”, which should your program prefer?) --- An anecdote: I was writing some code to send data to another company as an XML file. When I sent them a test file to validate, I got a complaint that it didn’t declare its encoding as UTF-8. “Yes, our database uses UCS-2 for exports,” I replied. “Why, can you not read it?” “Our system imported it just fine,“ they explained, “but it needs to say ‘<?xml version="1.0" encoding="UTF-8"?>’ at the top and when I looked at it with a text editor yours says something else there instead.” “Yes, it doesn’t say UTF-8 because it’s not UTF-8. Sounds like your system is OK with that, though?” “Yes, the rest of the file is fine, but we can’t move forward with go-live until you conform to the example in our implementation guide...” Actually changing the encoding proved difficult (for reasons I no longer recall), so I ended up shoving in a string replacement to make the file claim the wrong encoding, along with an apologetic comment; their end apparently did sniffing (and the BOM was unambiguous) so this worked out fine, but it goes to show just how confused people can get about what a character encoding a file actually is.
- pwdisswordfishc 3y ago