5 ms·

There's an unbounded number of possible graphemes per Unicode, because they are formed of a start character and an unbounded number of combining characters foll

There's an unbounded number of possible graphemes per Unicode, because they are formed of a start character and an unbounded number of combining characters following them. As such, no fixed-length encoding can actually enumerate all possible graphemes.

protomyth 11y ago
What is the practicality of an unbounded number of possible graphemes? Does anyone in real writing of languages and not some concocted test case need this? I understood there is some conflict in the encoding of kanji into unicode, does this relate?
- masklinn 11y ago
  > What is the practicality of an unbounded number of possible graphemes? That question doesn't really make sense. Unbounded cluster sizes is simply a result of unicode design because there's no reason to bound it. > Does anyone in real writing of languages and not some concocted test case need this? In the real writing of languages there are languages with grapheme clusters of size 4+ e.g. क्षि (Devanagari kshi) is a single cluster made up of क, ्, ष and ि. > I understood there is some conflict in the encoding of kanji into unicode, does this relate? It's unrelated. Han Unification compressed the number of clusters by unifying han characters across simplified Chinese, traditional Chinese, Japanese and Korean. Han characters were an issue in the original 16 bits unicode because supporting all of them was estimated as possibly topping out 100,000 codepoints (where 16 bits unicode only supported 65k codepoints). AFAIK Han characters don't use much if any composition (outside of romanisation). Brahmic scripts, Hebrew and Arabic scripts, on the other hand, pile on diacritics.
  - protomyth 11y ago
    > That question doesn't really make sense. Unbounded cluster sizes is simply a result of unicode design because there's no reason to bound it. If I'm trying to draw a character on the screen, it makes a lot of sense. I guess I'm struck about how hard it would be to select the 4th through 8th character displayed on the screen from rows 3 through 13.
    - lambda 11y ago
      Displaying characters on the screen, and giving the ability to select text, is indeed quite a difficult and complex question in the face of full support of the world's written languages. Glyphs vary in width. There are combining characters. There are ligatures (and in some languages, ligatures are necessary for the text to be readable, not just stylistic). There is contextual reordering of characters. There's bi-directional text, which can switch direction mid-sentence when quoting some Roman text in Hebrew or vice versa. There's vertical text. There are a number of fonts, font weights, font styles, font sizes, font features like alternative characters, language dependent selection of glyphs, and so on, plus fallback if the user doesn't have the designated fonts or fonts that cover the given Unicode range installed. There's line breaking, which can't always operate on spaces because not all languages include spaces between words. There's justification of text, distributing the spacing between characters to make a block of text even on both sides. All of this complexity exists regardless of the encoding that you use. Simply picking a fixed width encoding will not do anything about the vast majority of the complexity; and the fixed width encoding will cause problems of its own (size bloat of strings in memory), as well as leading people astray in thinking that maybe they can get away with making some of the same assumptions that apply to ASCII on a fixed-width terminal (and even there, assuming that 1 character equals 1 unit of width fails in the case of tabs), when in fact they can't and most text handling beyond simply appending strings together is best handled by dedicated libraries that have had hundreds of developer-years of effort put into solving all of these problems.
  - weinzierl 11y ago
    >> What is the practicality of an unbounded number of >> possible graphemes? >That question doesn't really make sense. Unbounded >cluster sizes is simply a result of unicode design >because there's no reason to bound it. The standard itself doesn't mention any bounds but there is Unicode Standard Annex #15 - Unicode Normalization Forms which defines the Stream-Safe Text Format. UAX15-D3. Stream-Safe Text Format: A Unicode string is said to be in Stream-Safe Text Format if it would not contain any sequences of non-starters longer than 30 characters in length when normalized to NFKD. * Such a string can be normalized in buffered serialization with a buffer size of 32 characters, which would require no more than 128 bytes in any Unicode Encoding Form. * Incorrect buffer handling can introduce subtle errors in the results. Any buffered implementation should be carefully checked against the normalization test data. * The value of 30 is chosen to be significantly beyond what is required for any linguistic or technical usage. While it would have been feasible to chose a smaller number, this value provides a very wide margin, yet is well within the buffer size limits of practical implementations. * NFKD was chosen for the definition because it produces the potentially longest sequences of non-starters from the same text. Funny thing I noticed when looking this up: The The Unicode Standard Version 7.0.0 PDF has exactly 1000 pages. [1] http://www.unicode.org/reports/tr15/#Stream_Safe_Text_Format http://www.unicode.org/reports/tr15/#Stream_Safe_Text_Format
- geocar 11y ago
  Consider glitchr for an extreme case of what is meant by combining characters: https://twitter.com/glitchr_ https://twitter.com/glitchr_ Whether it needs to be unbounded is a different discussion: Programmers tend to avoid constants that aren't 0, 1, or a power of two. "Unbounded" is easier than guessing (e.g. UCS16) wrong.
  - protomyth 11y ago
    0,1,infinite is a nice rule, but it certainly complicates a spec. I just cannot comprehend something being unbounded when we are talking about characters we use to communicate. Given the number of possibilities of 64-bits or 128-bits, I'm having a bit of a tough time thinking that their isn't a bound on what we can type into the computer.
- lambda 11y ago
  We have not yet developed writing systems for all of the world's languages. So there is no practical bound you can put, because it is possible that in a future development of a writing system, there will either be entirely new characters invented, or new combinations of existing characters and diacritics. In addition, since the existing Unicode standard does not impose a bound, and implementations support stacking of large numbers of diacritics, people have used combinations of extremely large numbers of diacritics as a stylistic device, for example: http://stackoverflow.com/a/1732454/69755 http://stackoverflow.com/a/1732454/69755. Since examples like this exist in the wild, they need to be supported properly, and so the fundamental string types and operations in string handling libraries should be able to handle it.
  - protomyth 11y ago
    Yes, we have to live with what was created, but given 128-bits is boil the ocean territory and someone has to type this stuff, I don't see a compelling argument for unbounded.
    - lambda 11y ago
      But why bother with a fixed width 128 bit encoding? We have perfectly fine variable-width encodings, that take up 8 bits per character for the most common set of characters encoded, the ASCII range (even for CJK text, the fact that most markup languages like HTML and XML use ASCII means that actually, the bulk of documents fall within the ASCII range), and can handle arbitrarily complex stacks of diacritics. How would it be helpful to waste a huge amount of space for the vast majority of text, impose additional limitations on the number of combining characters, just for the benefit of fixed-width encoding of graphemes, which even in itself is not all that helpful since for most applications, you are either going to need to scan through the text linearly, or you could use byte offsets in the variable width encoding just as well as you could use grapheme offsets in a fixed width encoding?
      - protomyth 11y ago
        I used 128-bit as a max. I just don't like some unbounded thing when we know that there is a practical max. I wouldn't call something perfectly fine when getting the number of characters actually displayed on the screen is a chore.

protomyth 11y ago

What is the practicality of an unbounded number of possible graphemes? Does anyone in real writing of languages and not some concocted test case need this? I understood there is some conflict in the encoding of kanji into unicode, does this relate?

masklinn 11y ago
> What is the practicality of an unbounded number of possible graphemes? That question doesn't really make sense. Unbounded cluster sizes is simply a result of unicode design because there's no reason to bound it. > Does anyone in real writing of languages and not some concocted test case need this? In the real writing of languages there are languages with grapheme clusters of size 4+ e.g. क्षि (Devanagari kshi) is a single cluster made up of क, ्, ष and ि. > I understood there is some conflict in the encoding of kanji into unicode, does this relate? It's unrelated. Han Unification compressed the number of clusters by unifying han characters across simplified Chinese, traditional Chinese, Japanese and Korean. Han characters were an issue in the original 16 bits unicode because supporting all of them was estimated as possibly topping out 100,000 codepoints (where 16 bits unicode only supported 65k codepoints). AFAIK Han characters don't use much if any composition (outside of romanisation). Brahmic scripts, Hebrew and Arabic scripts, on the other hand, pile on diacritics.
- protomyth 11y ago
  > That question doesn't really make sense. Unbounded cluster sizes is simply a result of unicode design because there's no reason to bound it. If I'm trying to draw a character on the screen, it makes a lot of sense. I guess I'm struck about how hard it would be to select the 4th through 8th character displayed on the screen from rows 3 through 13.
  - lambda 11y ago
    Displaying characters on the screen, and giving the ability to select text, is indeed quite a difficult and complex question in the face of full support of the world's written languages. Glyphs vary in width. There are combining characters. There are ligatures (and in some languages, ligatures are necessary for the text to be readable, not just stylistic). There is contextual reordering of characters. There's bi-directional text, which can switch direction mid-sentence when quoting some Roman text in Hebrew or vice versa. There's vertical text. There are a number of fonts, font weights, font styles, font sizes, font features like alternative characters, language dependent selection of glyphs, and so on, plus fallback if the user doesn't have the designated fonts or fonts that cover the given Unicode range installed. There's line breaking, which can't always operate on spaces because not all languages include spaces between words. There's justification of text, distributing the spacing between characters to make a block of text even on both sides. All of this complexity exists regardless of the encoding that you use. Simply picking a fixed width encoding will not do anything about the vast majority of the complexity; and the fixed width encoding will cause problems of its own (size bloat of strings in memory), as well as leading people astray in thinking that maybe they can get away with making some of the same assumptions that apply to ASCII on a fixed-width terminal (and even there, assuming that 1 character equals 1 unit of width fails in the case of tabs), when in fact they can't and most text handling beyond simply appending strings together is best handled by dedicated libraries that have had hundreds of developer-years of effort put into solving all of these problems.
- weinzierl 11y ago
  >> What is the practicality of an unbounded number of >> possible graphemes? >That question doesn't really make sense. Unbounded >cluster sizes is simply a result of unicode design >because there's no reason to bound it. The standard itself doesn't mention any bounds but there is Unicode Standard Annex #15 - Unicode Normalization Forms which defines the Stream-Safe Text Format. UAX15-D3. Stream-Safe Text Format: A Unicode string is said to be in Stream-Safe Text Format if it would not contain any sequences of non-starters longer than 30 characters in length when normalized to NFKD. * Such a string can be normalized in buffered serialization with a buffer size of 32 characters, which would require no more than 128 bytes in any Unicode Encoding Form. * Incorrect buffer handling can introduce subtle errors in the results. Any buffered implementation should be carefully checked against the normalization test data. * The value of 30 is chosen to be significantly beyond what is required for any linguistic or technical usage. While it would have been feasible to chose a smaller number, this value provides a very wide margin, yet is well within the buffer size limits of practical implementations. * NFKD was chosen for the definition because it produces the potentially longest sequences of non-starters from the same text. Funny thing I noticed when looking this up: The The Unicode Standard Version 7.0.0 PDF has exactly 1000 pages. [1] http://www.unicode.org/reports/tr15/#Stream_Safe_Text_Format http://www.unicode.org/reports/tr15/#Stream_Safe_Text_Format
geocar 11y ago
Consider glitchr for an extreme case of what is meant by combining characters: https://twitter.com/glitchr_ https://twitter.com/glitchr_ Whether it needs to be unbounded is a different discussion: Programmers tend to avoid constants that aren't 0, 1, or a power of two. "Unbounded" is easier than guessing (e.g. UCS16) wrong.
- protomyth 11y ago
  0,1,infinite is a nice rule, but it certainly complicates a spec. I just cannot comprehend something being unbounded when we are talking about characters we use to communicate. Given the number of possibilities of 64-bits or 128-bits, I'm having a bit of a tough time thinking that their isn't a bound on what we can type into the computer.
lambda 11y ago
We have not yet developed writing systems for all of the world's languages. So there is no practical bound you can put, because it is possible that in a future development of a writing system, there will either be entirely new characters invented, or new combinations of existing characters and diacritics. In addition, since the existing Unicode standard does not impose a bound, and implementations support stacking of large numbers of diacritics, people have used combinations of extremely large numbers of diacritics as a stylistic device, for example: http://stackoverflow.com/a/1732454/69755 http://stackoverflow.com/a/1732454/69755. Since examples like this exist in the wild, they need to be supported properly, and so the fundamental string types and operations in string handling libraries should be able to handle it.
- protomyth 11y ago
  Yes, we have to live with what was created, but given 128-bits is boil the ocean territory and someone has to type this stuff, I don't see a compelling argument for unbounded.
  - lambda 11y ago
    But why bother with a fixed width 128 bit encoding? We have perfectly fine variable-width encodings, that take up 8 bits per character for the most common set of characters encoded, the ASCII range (even for CJK text, the fact that most markup languages like HTML and XML use ASCII means that actually, the bulk of documents fall within the ASCII range), and can handle arbitrarily complex stacks of diacritics. How would it be helpful to waste a huge amount of space for the vast majority of text, impose additional limitations on the number of combining characters, just for the benefit of fixed-width encoding of graphemes, which even in itself is not all that helpful since for most applications, you are either going to need to scan through the text linearly, or you could use byte offsets in the variable width encoding just as well as you could use grapheme offsets in a fixed width encoding?
    - protomyth 11y ago
      I used 128-bit as a max. I just don't like some unbounded thing when we know that there is a practical max. I wouldn't call something perfectly fine when getting the number of characters actually displayed on the screen is a chore.