6 ms·

> I was just thinking how convenient it is to type emojis right into strings in Python and print them. I assumed C didn't have much unicode compatibility, thoug

> I was just thinking how convenient it is to type emojis right into strings in Python and print them. I assumed C didn't have much unicode compatibility, though I didn't research it. Libgrapheme is a nice library, but it doesn't really have anything to do with this. Almost all modern terminal emulators use the UTF-8 character encoding. In order to successfully output Unicode characters, your programming language doesn't actually need much "Unicode support"; it just needs to be able to send UTF-8-encoded bytes to stdout. (That's why many modern programming languages like Go and Zig define strings as a simple array of bytes.) Modern C compilers allow you to printf("é") and get the appropriate behavior. As you mention, the terminal emulator also needs to be able to decode and display those UTF-8 bytes correctly, and a lot of terminals don't get it right in some situations. Off the top of my head, I don't know of a terminal that actually implements the entire (very complex) set of Unicode text rendering behaviors; maybe one of the web-based ones that run in Electron? macOS's Terminal.app is also pretty good IIRC. Where libgrapheme comes in is if you want to analyze or manipulate a UTF-8-encoded string. It provides operations like "split into words" and "convert to uppercase". A surprising number of programs never need to do that stuff, but if you do, libgrapheme will give you a Unicode-compatible implementation. (Many more basic operations, like concatenating two strings, will work just fine without libgrapheme.)

mananaysiempre 4y ago
(Not a language or Unicode expert, the following likely has important mistakes.) > Off the top of my head, I don't know of a terminal that actually implements the entire (very complex) set of Unicode text rendering behaviors There are at least two reasons for this: First, nobody actually seems to know how bidirectional text should interact with terminal control sequences, or indeed how it should be typeset on a terminal in the first place (what is the primary direction? where are the reordering boundaries?). There is the pre-Unicode bidirectional support mode (BDSM, I kid you not) in ECMA-48[1] and TR/53[2], which AFAIK nobody implements nor cares about; there are terminal emulators endorsed by bidi-language users[3], which AFAIK nobody has written down the behaviour of; there is the Freedesktop bidi terminal spec[4], which is a draft and AFAIK nobody implements yet either but at least some people care about; finally, there are bidi-language users who say that spec is a mistake[5]. Second, aside from bidi and a smattering of other things such as emoji, there is no detailed “Unicode text rendering behaviour”, only standards specific to font formats—the most recent among them being OpenType, which is dubiously compatible across implementations, decently documented only through painstaking reverse engineering (sometimes in words[6], sometimes only in Freetype library code), and generally full of snakes[7]. And it has no notion of a monospace font—only of a (proportional) font where all Lat/Cyr/Grk characters just happen to have the same advance. AFAICT that is not negligence or an oversight, but rather a concession to the fact that there are scripts which don’t really have a notion of monospace in the typographic tradition and in fact are written such that it’s extremely unclear what monospace would even mean—certainly not one or two cells per codepoint (e.g. Burmese or Tibetan; apparently there are Arabic monospace fonts[8] but I’ve no idea how the hell they work). Not coincidentally, those are the scripts where you really, really need that shaper, otherwise nothing looks anywhere close to correct. [This post could have been titled “Contra Muratori on Unicode in terminal emulators”.] [1] https://www.ecma-international.org/publications-and-standards/standards/ecma-48/ https://www.ecma-international.org/publications-and-standard... [2] https://www.ecma-international.org/publications-and-standards/technical-reports/ecma-tr-53/ https://www.ecma-international.org/publications-and-standard... [3] https://news.ycombinator.com/item?id=8086417 https://news.ycombinator.com/item?id=8086417 [4] https://terminal-wg.pages.freedesktop.org/bidi/ https://terminal-wg.pages.freedesktop.org/bidi/ [5] http://litcave.rudi.ir/ http://litcave.rudi.ir/ [6] https://github.com/n8willis/opentype-shaping-documents https://github.com/n8willis/opentype-shaping-documents [7] https://litherum.blogspot.com/2019/03/addition-font.html https://litherum.blogspot.com/2019/03/addition-font.html [8] https://news.ycombinator.com/item?id=10395464 https://news.ycombinator.com/item?id=10395464
- csande17 4y ago
  Here's another fun Unicode pitfall: does any terminal provide a way to display Chinese and Japanese text simultaneously, using the appropriate versions of the glyphs for each language's characters?
  - mananaysiempre 4y ago
    As far as existing terminals are concerned, I don’t know. FWIW, there are similar problems (though only to the point of looking wrong, not of misunderstanding) in other scripts: Cyrillic as used in Bulgarian and a number of other languages[1] and even Latin as used in Polish[2]. Even the Han version, though, does not seem to me to be the sort of “what does it even mean?” problem like those I listed above; more like what you want the input to be. You can make your terminal keep language state, e.g. using the deprecated language tags. Pro: some form of this likely already needs to happen for bidi support; similar to what HTML does. Con: no text file or program ever did this; your nice UTF-8-only terminal is now stateful and goes mad after `head /dev/urandom`. Alternatively, you can require the driving program to emit variation selectors for each Han character. Pro: the state and the ensuing madness is now limited; you can still pretend you’re looking at a stream of characters. Con: no text file or program ever did this; neither does HTML although it theorerically could. [1] https://commons.wikimedia.org/wiki/File:Cyrillic_alternates.svg https://commons.wikimedia.org/wiki/File:Cyrillic_alternates.... [2] https://www.twardoch.com/download/polishhowto/kreska.html https://www.twardoch.com/download/polishhowto/kreska.html
  - a1369209993 4y ago
    > does any terminal provide a way to display Chinese and Japanese text simultaneously, using the appropriate versions of the glyphs Does any terminal provide a way to display (say) two consecutive lines in different fonts? Because according to Unicode (and for once I don't particularly disagree), Chinese versus Japanese ideographs are the same characters, drawn in two different fonts.
    - hnfong 4y ago
      https://www.facebook.com/appliedphilosophy/photos/a.828540073892609/891825587564057 https://www.facebook.com/appliedphilosophy/photos/a.82854007... Many years ago I thought the highlighted text was a light hearted joke. Now I see what it is actually quite serious. (And now I understand why, among all people, a Japanese philosopher brought up the issue) I personally think Unicode didn't go essentialist enough though. There's lots of obscure CJK characters in unicode that, when you look up a dictionary, just says it's another form of a common character. What I heard from people in the know was that the CJK committee basically made an exception for the Japanese because they were threatening to withdraw from the Unicode effort unless their demands were met, and for the most part they got what they wanted. The "Chinese versus Japanese ideographs are the same characters, drawn in two different fonts" is actually a minority of those cases, which I personally interpret as clerical errors instead of a philosophical disagreement. For example if Unicode were actually thorough with the essentialist philosophy, 關 and 関 would have the same code point.
arcticbull 4y ago
> That's why many modern programming languages like Go and Zig define strings as a simple array of bytes. I'd argue this is not in fact a modern approach but rather a pretty dated one. Modern language string API design instead leans into 'views' of the underlying depending on what you are trying to do with it. - Transmission and storage: UTF-8 bytes. - Parsing: code points. - Editing: grapheme clusters. - Rendering: pixels. It doesn't make sense to consider a random assortment of bytes to be a 'string', that's just an array of random bytes - and the language almost certainly already has a better type for that like a vector of 8-bit unsigned values or a slice type. Strings should be opaque - but always valid - encoded data to be sensible and differentiated from a slice type. Zig seems to be even worse than Go, requiring null-termination instead of storing them as Pascal strings. This means that a valid substring requires a copy so that the substring can itself be null-terminated to be valid. Inner pointers are not valid unless to the end of the parent and therefore substrings are very expensive relatively speaking.
- kristoff_it 4y ago
  > Zig seems to be even worse than Go, requiring null-termination instead of storing them as Pascal strings. This means that a valid substring requires a copy so that the substring can itself be null-terminated to be valid. Inner pointers are not valid unless to the end of the parent and therefore substrings are very expensive relatively speaking. No, Zig uses slices normally (ie ptr + len). Terminators are part of the type system for the (relatively rare, mostly related to C interop) cases where you really want a sentinel value at the end of your array.
  - arcticbull 4y ago
    I see, I misread the docs. However everything else I stand by, after all it allows you to embed invalid UTF-8 content.

mananaysiempre 4y ago

(Not a language or Unicode expert, the following likely has important mistakes.) > Off the top of my head, I don't know of a terminal that actually implements the entire (very complex) set of Unicode text rendering behaviors There are at least two reasons for this: First, nobody actually seems to know how bidirectional text should interact with terminal control sequences, or indeed how it should be typeset on a terminal in the first place (what is the primary direction? where are the reordering boundaries?). There is the pre-Unicode bidirectional support mode (BDSM, I kid you not) in ECMA-48[1] and TR/53[2], which AFAIK nobody implements nor cares about; there are terminal emulators endorsed by bidi-language users[3], which AFAIK nobody has written down the behaviour of; there is the Freedesktop bidi terminal spec[4], which is a draft and AFAIK nobody implements yet either but at least some people care about; finally, there are bidi-language users who say that spec is a mistake[5]. Second, aside from bidi and a smattering of other things such as emoji, there is no detailed “Unicode text rendering behaviour”, only standards specific to font formats—the most recent among them being OpenType, which is dubiously compatible across implementations, decently documented only through painstaking reverse engineering (sometimes in words[6], sometimes only in Freetype library code), and generally full of snakes[7]. And it has no notion of a monospace font—only of a (proportional) font where all Lat/Cyr/Grk characters just happen to have the same advance. AFAICT that is not negligence or an oversight, but rather a concession to the fact that there are scripts which don’t really have a notion of monospace in the typographic tradition and in fact are written such that it’s extremely unclear what monospace would even mean—certainly not one or two cells per codepoint (e.g. Burmese or Tibetan; apparently there are Arabic monospace fonts[8] but I’ve no idea how the hell they work). Not coincidentally, those are the scripts where you really, really need that shaper, otherwise nothing looks anywhere close to correct. [This post could have been titled “Contra Muratori on Unicode in terminal emulators”.] [1] https://www.ecma-international.org/publications-and-standards/standards/ecma-48/ https://www.ecma-international.org/publications-and-standard... [2] https://www.ecma-international.org/publications-and-standards/technical-reports/ecma-tr-53/ https://www.ecma-international.org/publications-and-standard... [3] https://news.ycombinator.com/item?id=8086417 https://news.ycombinator.com/item?id=8086417 [4] https://terminal-wg.pages.freedesktop.org/bidi/ https://terminal-wg.pages.freedesktop.org/bidi/ [5] http://litcave.rudi.ir/ http://litcave.rudi.ir/ [6] https://github.com/n8willis/opentype-shaping-documents https://github.com/n8willis/opentype-shaping-documents [7] https://litherum.blogspot.com/2019/03/addition-font.html https://litherum.blogspot.com/2019/03/addition-font.html [8] https://news.ycombinator.com/item?id=10395464 https://news.ycombinator.com/item?id=10395464

csande17 4y ago
Here's another fun Unicode pitfall: does any terminal provide a way to display Chinese and Japanese text simultaneously, using the appropriate versions of the glyphs for each language's characters?
- mananaysiempre 4y ago
  As far as existing terminals are concerned, I don’t know. FWIW, there are similar problems (though only to the point of looking wrong, not of misunderstanding) in other scripts: Cyrillic as used in Bulgarian and a number of other languages[1] and even Latin as used in Polish[2]. Even the Han version, though, does not seem to me to be the sort of “what does it even mean?” problem like those I listed above; more like what you want the input to be. You can make your terminal keep language state, e.g. using the deprecated language tags. Pro: some form of this likely already needs to happen for bidi support; similar to what HTML does. Con: no text file or program ever did this; your nice UTF-8-only terminal is now stateful and goes mad after `head /dev/urandom`. Alternatively, you can require the driving program to emit variation selectors for each Han character. Pro: the state and the ensuing madness is now limited; you can still pretend you’re looking at a stream of characters. Con: no text file or program ever did this; neither does HTML although it theorerically could. [1] https://commons.wikimedia.org/wiki/File:Cyrillic_alternates.svg https://commons.wikimedia.org/wiki/File:Cyrillic_alternates.... [2] https://www.twardoch.com/download/polishhowto/kreska.html https://www.twardoch.com/download/polishhowto/kreska.html
- a1369209993 4y ago
  > does any terminal provide a way to display Chinese and Japanese text simultaneously, using the appropriate versions of the glyphs Does any terminal provide a way to display (say) two consecutive lines in different fonts? Because according to Unicode (and for once I don't particularly disagree), Chinese versus Japanese ideographs are the same characters, drawn in two different fonts.
  - hnfong 4y ago
    https://www.facebook.com/appliedphilosophy/photos/a.828540073892609/891825587564057 https://www.facebook.com/appliedphilosophy/photos/a.82854007... Many years ago I thought the highlighted text was a light hearted joke. Now I see what it is actually quite serious. (And now I understand why, among all people, a Japanese philosopher brought up the issue) I personally think Unicode didn't go essentialist enough though. There's lots of obscure CJK characters in unicode that, when you look up a dictionary, just says it's another form of a common character. What I heard from people in the know was that the CJK committee basically made an exception for the Japanese because they were threatening to withdraw from the Unicode effort unless their demands were met, and for the most part they got what they wanted. The "Chinese versus Japanese ideographs are the same characters, drawn in two different fonts" is actually a minority of those cases, which I personally interpret as clerical errors instead of a philosophical disagreement. For example if Unicode were actually thorough with the essentialist philosophy, 關 and 関 would have the same code point.

arcticbull 4y ago

> That's why many modern programming languages like Go and Zig define strings as a simple array of bytes. I'd argue this is not in fact a modern approach but rather a pretty dated one. Modern language string API design instead leans into 'views' of the underlying depending on what you are trying to do with it. - Transmission and storage: UTF-8 bytes. - Parsing: code points. - Editing: grapheme clusters. - Rendering: pixels. It doesn't make sense to consider a random assortment of bytes to be a 'string', that's just an array of random bytes - and the language almost certainly already has a better type for that like a vector of 8-bit unsigned values or a slice type. Strings should be opaque - but always valid - encoded data to be sensible and differentiated from a slice type. Zig seems to be even worse than Go, requiring null-termination instead of storing them as Pascal strings. This means that a valid substring requires a copy so that the substring can itself be null-terminated to be valid. Inner pointers are not valid unless to the end of the parent and therefore substrings are very expensive relatively speaking.

kristoff_it 4y ago
> Zig seems to be even worse than Go, requiring null-termination instead of storing them as Pascal strings. This means that a valid substring requires a copy so that the substring can itself be null-terminated to be valid. Inner pointers are not valid unless to the end of the parent and therefore substrings are very expensive relatively speaking. No, Zig uses slices normally (ie ptr + len). Terminators are part of the type system for the (relatively rare, mostly related to C interop) cases where you really want a sentinel value at the end of your array.
- arcticbull 4y ago
  I see, I misread the docs. However everything else I stand by, after all it allows you to embed invalid UTF-8 content.