4 ms·

This article is pretty great, however I would like to caution against using the word "character" in discussions about Unicode. The post does say that it's a fuz

This article is pretty great, however I would like to caution against using the word "character" in discussions about Unicode. The post does say that it's a fuzzy concept, but also says that it's "Basically a thing in the Unicode table somewhere.", which is the definition of a code point. Unicode itself gives multiple incompatible meanings for the term (http://unicode.org/glossary/#character http://unicode.org/glossary/#character) and only seems to use it in non-normative text -- so the post isn't wrong, but has the potential to cause confusion. Often when you want to say character you really mean "grapheme cluster"[1]. A Devanagri consonant cluster (of one or more consonants) with optional vowel (and misc diacritics) is a grapheme cluster. A latin letter with accent mark(s) is a grapheme cluster. A Hangul Jamo (precomposed or otherwise) is a grapheme cluster. A flag or multicultural family emoji is a grapheme cluster. When talking about characters, the question arises if a Hangul Jamo made of decomposed characters is one character or three. Or if the combination of a [devanagri consonant + virama] "character" and [devanagri consonant + vowel] "character" is one character or two. This is specified in the case of grapheme clusters, and this usually maps to where the notion of a character is important -- text selection and offsets in editing, etc. "code point" does not universally map to any tangible concept -- it's a concept made up for the sake of specifying unicode. You only care about code points when dealing with UTF32 strings or when implementing operations on unicode text. "glyph" is also sometimes what we mean when we say "character", though that's more useful on the rendering end of things. [1]: to be pedantic, "extended grapheme cluster", because unicode gave a rigorous definition of grapheme cluster and later decided to change it.

d0mine 10y ago
I find the following hierarchy helpful: bytes -> code units -> code points -> extended grapheme clusters -> user-perceived characters In Python, Unicode string is an immutable sequence of Unicode code points. It has nothing to do with UTF32 (Python uses flexible internal representation). To get "user-perceived characters" (approximated by eXtended graphemes clusters): chars = regex.findall('\X', unicode_text) width_in_terminal = wcwidth.wcswidth(unicode_text) In practice, whatever is produced by the default method of iterating over a string is often called a character in that programming language (code point, or UTF16 code unit, or even a byte). https://pypi.python.org/pypi/regex https://pypi.python.org/pypi/regex https://pypi.python.org/pypi/wcwidth https://pypi.python.org/pypi/wcwidth
- labster 10y ago
  In Perl 6: $width-in-terminal = $text.chars; $codepoints = $text.codes; $bytes = $text.encode('utf8').bytes; The .chars method should be the fastest, because Perl 6 internally uses strings of fully composed characters (normalized form grapheme). It's much better than having to do regex hacks like in Python.
  - dom0 10y ago
    $width-in-terminal = $text.chars; ^- very likely wrong. wcswidth computes the cell-width (or column-count) of a Unicode string, which is unrelated to the count of graphemes, EGCs or code points. For example, Latin characters are one cell/column wide, while for example many CJK characters occupy two cells/columns, while they are still one EGC. A typical application is printing CJK things to a terminal mask, progress display or similar.
    - labster 10y ago
      Ah right. I guess we were meaning different things by width. And you're probably more correct here.
  - Someone 10y ago
    Given that one can make graphemes that contain many Unicode code points (213 in this extreme example: https://www.reddit.com/r/Unicode/comments/4yie0a/tallest_longest_unicode_character/ https://www.reddit.com/r/Unicode/comments/4yie0a/tallest_lon...), I wondered whether that can be correct for any reasonable definition of "fully composed character". As far as I can tell, it turned out to be correct, though. Perl 6 has its own Inocode normalization variant called NFG (https://design.perl6.org/S15.html#NFG https://design.perl6.org/S15.html#NFG): "Formally Perl 6 graphemes are defined exactly according to Unicode Grapheme Cluster Boundaries at level "extended" (in contrast to "tailored" or "legacy"), see Unicode Standard Annex #29 UNICODE TEXT SEGMENTATION 3 Grapheme Cluster Boundaries>. This is the same as the Perl 5 character class \X Match Unicode "eXtended grapheme cluster" With NFG, strings start by being run through the normal NFC process, compressing any given character sequences into precomposed characters. Any graphemes remaining without precomposed characters, such as ậ or नि, are given their own internal designation to refer to them, at least 32 bits in length, in such a way that they avoid clashing with any potential future changes to Unicode. The mapping between these internal designations and graphemes in this form is not guaranteed constant, even between strings in the same process." From that, I guess Perl 6 extends its mapping between NFG code points (an extension of Unicode code points) and Unicode graphemes clusters whenever it encounters a grapheme cluster it hasn't seen before. Ignoring performance concerns (might not be bad, but I'm not sure about that), that seems a nice approach.
jasode 10y ago
>, but has the potential to cause confusion. On a related note, one of the best examples that reveal the confusing intricacies of Unicode was a previous HN thread about encoding Bengali: https://news.ycombinator.com/item?id=9219162 https://news.ycombinator.com/item?id=9219162 Lots of intelligent comments from many contributors in that high quality thread. For me, the takeaway was the truly understanding Unicode requires holding 5 different layers of concepts in your head simultaneously and the oft-cited Joel Software "Absolute Minimum You Must Know About Unicode" only covers 2 of them. (Which to be fair to Joel, his title emphasizes "minimum to know" and not "everything to know"). Also, at the highest level of abstraction, intelligent people can have philosophical disagreements on _what_ to encode in Unicode.
brudgers 10y ago
The problem with "character" is the same as the problem with "string." There is a formal computer science definition that comes from the discipline of automata, is mathematical, and useful for building compilers and state machines and why characters and strings in C are bytes and streams of bytes. Because ASCII abstracts onto bytes well but text does not, layering an abstraction for 'text' that decomposes into characters and strings is where the problem lies. That's where, for example, Python 2 ran into trouble. Perl's set of compromises made its text abstraction useful, but creating them required creating them to be a principle focus of the language. A stream of bytes is a stream of bytes, short of magic there's no way to generate a correct interpretation but for a predesignated protocol. And the only way to get a predesignated protocol for text is to make a deep study of human language and to choose to live with some compromises and not to live with others.

d0mine 10y ago

I find the following hierarchy helpful: bytes -> code units -> code points -> extended grapheme clusters -> user-perceived characters In Python, Unicode string is an immutable sequence of Unicode code points. It has nothing to do with UTF32 (Python uses flexible internal representation). To get "user-perceived characters" (approximated by eXtended graphemes clusters): chars = regex.findall('\X', unicode_text) width_in_terminal = wcwidth.wcswidth(unicode_text) In practice, whatever is produced by the default method of iterating over a string is often called a character in that programming language (code point, or UTF16 code unit, or even a byte). https://pypi.python.org/pypi/regex https://pypi.python.org/pypi/regex https://pypi.python.org/pypi/wcwidth https://pypi.python.org/pypi/wcwidth

labster 10y ago
In Perl 6: $width-in-terminal = $text.chars; $codepoints = $text.codes; $bytes = $text.encode('utf8').bytes; The .chars method should be the fastest, because Perl 6 internally uses strings of fully composed characters (normalized form grapheme). It's much better than having to do regex hacks like in Python.
- dom0 10y ago
  $width-in-terminal = $text.chars; ^- very likely wrong. wcswidth computes the cell-width (or column-count) of a Unicode string, which is unrelated to the count of graphemes, EGCs or code points. For example, Latin characters are one cell/column wide, while for example many CJK characters occupy two cells/columns, while they are still one EGC. A typical application is printing CJK things to a terminal mask, progress display or similar.
  - labster 10y ago
    Ah right. I guess we were meaning different things by width. And you're probably more correct here.
- Someone 10y ago
  Given that one can make graphemes that contain many Unicode code points (213 in this extreme example: https://www.reddit.com/r/Unicode/comments/4yie0a/tallest_longest_unicode_character/ https://www.reddit.com/r/Unicode/comments/4yie0a/tallest_lon...), I wondered whether that can be correct for any reasonable definition of "fully composed character". As far as I can tell, it turned out to be correct, though. Perl 6 has its own Inocode normalization variant called NFG (https://design.perl6.org/S15.html#NFG https://design.perl6.org/S15.html#NFG): "Formally Perl 6 graphemes are defined exactly according to Unicode Grapheme Cluster Boundaries at level "extended" (in contrast to "tailored" or "legacy"), see Unicode Standard Annex #29 UNICODE TEXT SEGMENTATION 3 Grapheme Cluster Boundaries>. This is the same as the Perl 5 character class \X Match Unicode "eXtended grapheme cluster" With NFG, strings start by being run through the normal NFC process, compressing any given character sequences into precomposed characters. Any graphemes remaining without precomposed characters, such as ậ or नि, are given their own internal designation to refer to them, at least 32 bits in length, in such a way that they avoid clashing with any potential future changes to Unicode. The mapping between these internal designations and graphemes in this form is not guaranteed constant, even between strings in the same process." From that, I guess Perl 6 extends its mapping between NFG code points (an extension of Unicode code points) and Unicode graphemes clusters whenever it encounters a grapheme cluster it hasn't seen before. Ignoring performance concerns (might not be bad, but I'm not sure about that), that seems a nice approach.

jasode 10y ago

>, but has the potential to cause confusion. On a related note, one of the best examples that reveal the confusing intricacies of Unicode was a previous HN thread about encoding Bengali: https://news.ycombinator.com/item?id=9219162 https://news.ycombinator.com/item?id=9219162 Lots of intelligent comments from many contributors in that high quality thread. For me, the takeaway was the truly understanding Unicode requires holding 5 different layers of concepts in your head simultaneously and the oft-cited Joel Software "Absolute Minimum You Must Know About Unicode" only covers 2 of them. (Which to be fair to Joel, his title emphasizes "minimum to know" and not "everything to know"). Also, at the highest level of abstraction, intelligent people can have philosophical disagreements on _what_ to encode in Unicode.

brudgers 10y ago

The problem with "character" is the same as the problem with "string." There is a formal computer science definition that comes from the discipline of automata, is mathematical, and useful for building compilers and state machines and why characters and strings in C are bytes and streams of bytes. Because ASCII abstracts onto bytes well but text does not, layering an abstraction for 'text' that decomposes into characters and strings is where the problem lies. That's where, for example, Python 2 ran into trouble. Perl's set of compromises made its text abstraction useful, but creating them required creating them to be a principle focus of the language. A stream of bytes is a stream of bytes, short of magic there's no way to generate a correct interpretation but for a predesignated protocol. And the only way to get a predesignated protocol for text is to make a deep study of human language and to choose to live with some compromises and not to live with others.