8 ms·

I've been trying to get into C this past week, and this is a great coincidence to see this today! I was just thinking how convenient it is to type emojis right

I've been trying to get into C this past week, and this is a great coincidence to see this today! I was just thinking how convenient it is to type emojis right into strings in Python and print them. I assumed C didn't have much unicode compatibility, though I didn't research it. I gave libgrapheme a try, and it compiled just as the instructions said it would. The hello-world program also mostly worked, but in my terminal it malformed several things. For example the American flag emoji rendered in my terminal as [U][S], and the family emoji rendered as three distinct emoji faces (side-by-side) rather than one grouped one. I went to a website that lets me copy emojis to my clipboard, and I directly copy-pasted the American flag into my terminal, and I still got [U][S], so I think the problem is just with the terminal and not the library. edit: Indeed, this is a problem in Gnome terminal. I found a Bugzilla link[0] that is still open. The official name for the grouped emoji type is "ZWJ sequence"[1], short for Zero-Width Joiner, and it appears not a lot of terminals support them. If anyone knows of a good one for Linux, please let me know! Great stuff, thank you for sharing! References: [0]: https://gitlab.gnome.org/GNOME/vte/-/issues/2317 https://gitlab.gnome.org/GNOME/vte/-/issues/2317 [1]: https://emojipedia.org/emoji-zwj-sequence/ https://emojipedia.org/emoji-zwj-sequence/

csande17 4y ago
> I was just thinking how convenient it is to type emojis right into strings in Python and print them. I assumed C didn't have much unicode compatibility, though I didn't research it. Libgrapheme is a nice library, but it doesn't really have anything to do with this. Almost all modern terminal emulators use the UTF-8 character encoding. In order to successfully output Unicode characters, your programming language doesn't actually need much "Unicode support"; it just needs to be able to send UTF-8-encoded bytes to stdout. (That's why many modern programming languages like Go and Zig define strings as a simple array of bytes.) Modern C compilers allow you to printf("é") and get the appropriate behavior. As you mention, the terminal emulator also needs to be able to decode and display those UTF-8 bytes correctly, and a lot of terminals don't get it right in some situations. Off the top of my head, I don't know of a terminal that actually implements the entire (very complex) set of Unicode text rendering behaviors; maybe one of the web-based ones that run in Electron? macOS's Terminal.app is also pretty good IIRC. Where libgrapheme comes in is if you want to analyze or manipulate a UTF-8-encoded string. It provides operations like "split into words" and "convert to uppercase". A surprising number of programs never need to do that stuff, but if you do, libgrapheme will give you a Unicode-compatible implementation. (Many more basic operations, like concatenating two strings, will work just fine without libgrapheme.)
- mananaysiempre 4y ago
  (Not a language or Unicode expert, the following likely has important mistakes.) > Off the top of my head, I don't know of a terminal that actually implements the entire (very complex) set of Unicode text rendering behaviors There are at least two reasons for this: First, nobody actually seems to know how bidirectional text should interact with terminal control sequences, or indeed how it should be typeset on a terminal in the first place (what is the primary direction? where are the reordering boundaries?). There is the pre-Unicode bidirectional support mode (BDSM, I kid you not) in ECMA-48[1] and TR/53[2], which AFAIK nobody implements nor cares about; there are terminal emulators endorsed by bidi-language users[3], which AFAIK nobody has written down the behaviour of; there is the Freedesktop bidi terminal spec[4], which is a draft and AFAIK nobody implements yet either but at least some people care about; finally, there are bidi-language users who say that spec is a mistake[5]. Second, aside from bidi and a smattering of other things such as emoji, there is no detailed “Unicode text rendering behaviour”, only standards specific to font formats—the most recent among them being OpenType, which is dubiously compatible across implementations, decently documented only through painstaking reverse engineering (sometimes in words[6], sometimes only in Freetype library code), and generally full of snakes[7]. And it has no notion of a monospace font—only of a (proportional) font where all Lat/Cyr/Grk characters just happen to have the same advance. AFAICT that is not negligence or an oversight, but rather a concession to the fact that there are scripts which don’t really have a notion of monospace in the typographic tradition and in fact are written such that it’s extremely unclear what monospace would even mean—certainly not one or two cells per codepoint (e.g. Burmese or Tibetan; apparently there are Arabic monospace fonts[8] but I’ve no idea how the hell they work). Not coincidentally, those are the scripts where you really, really need that shaper, otherwise nothing looks anywhere close to correct. [This post could have been titled “Contra Muratori on Unicode in terminal emulators”.] [1] https://www.ecma-international.org/publications-and-standards/standards/ecma-48/ https://www.ecma-international.org/publications-and-standard... [2] https://www.ecma-international.org/publications-and-standards/technical-reports/ecma-tr-53/ https://www.ecma-international.org/publications-and-standard... [3] https://news.ycombinator.com/item?id=8086417 https://news.ycombinator.com/item?id=8086417 [4] https://terminal-wg.pages.freedesktop.org/bidi/ https://terminal-wg.pages.freedesktop.org/bidi/ [5] http://litcave.rudi.ir/ http://litcave.rudi.ir/ [6] https://github.com/n8willis/opentype-shaping-documents https://github.com/n8willis/opentype-shaping-documents [7] https://litherum.blogspot.com/2019/03/addition-font.html https://litherum.blogspot.com/2019/03/addition-font.html [8] https://news.ycombinator.com/item?id=10395464 https://news.ycombinator.com/item?id=10395464
kevin_thibedeau 4y ago
C11 has full support for Unicode in source code for string and char literals and comments. Any runtime Unicode processing, however, has to be provided by outside code.
5e92cb50239222b 4y ago
> If anyone knows of a good one for Linux, please let me know! kitty has excellent support for this stuff, and is much more performant than anything based on vte (like gnome-terminal). https://github.com/kovidgoyal/kitty https://github.com/kovidgoyal/kitty

csande17 4y ago

> I was just thinking how convenient it is to type emojis right into strings in Python and print them. I assumed C didn't have much unicode compatibility, though I didn't research it. Libgrapheme is a nice library, but it doesn't really have anything to do with this. Almost all modern terminal emulators use the UTF-8 character encoding. In order to successfully output Unicode characters, your programming language doesn't actually need much "Unicode support"; it just needs to be able to send UTF-8-encoded bytes to stdout. (That's why many modern programming languages like Go and Zig define strings as a simple array of bytes.) Modern C compilers allow you to printf("é") and get the appropriate behavior. As you mention, the terminal emulator also needs to be able to decode and display those UTF-8 bytes correctly, and a lot of terminals don't get it right in some situations. Off the top of my head, I don't know of a terminal that actually implements the entire (very complex) set of Unicode text rendering behaviors; maybe one of the web-based ones that run in Electron? macOS's Terminal.app is also pretty good IIRC. Where libgrapheme comes in is if you want to analyze or manipulate a UTF-8-encoded string. It provides operations like "split into words" and "convert to uppercase". A surprising number of programs never need to do that stuff, but if you do, libgrapheme will give you a Unicode-compatible implementation. (Many more basic operations, like concatenating two strings, will work just fine without libgrapheme.)

mananaysiempre 4y ago
(Not a language or Unicode expert, the following likely has important mistakes.) > Off the top of my head, I don't know of a terminal that actually implements the entire (very complex) set of Unicode text rendering behaviors There are at least two reasons for this: First, nobody actually seems to know how bidirectional text should interact with terminal control sequences, or indeed how it should be typeset on a terminal in the first place (what is the primary direction? where are the reordering boundaries?). There is the pre-Unicode bidirectional support mode (BDSM, I kid you not) in ECMA-48[1] and TR/53[2], which AFAIK nobody implements nor cares about; there are terminal emulators endorsed by bidi-language users[3], which AFAIK nobody has written down the behaviour of; there is the Freedesktop bidi terminal spec[4], which is a draft and AFAIK nobody implements yet either but at least some people care about; finally, there are bidi-language users who say that spec is a mistake[5]. Second, aside from bidi and a smattering of other things such as emoji, there is no detailed “Unicode text rendering behaviour”, only standards specific to font formats—the most recent among them being OpenType, which is dubiously compatible across implementations, decently documented only through painstaking reverse engineering (sometimes in words[6], sometimes only in Freetype library code), and generally full of snakes[7]. And it has no notion of a monospace font—only of a (proportional) font where all Lat/Cyr/Grk characters just happen to have the same advance. AFAICT that is not negligence or an oversight, but rather a concession to the fact that there are scripts which don’t really have a notion of monospace in the typographic tradition and in fact are written such that it’s extremely unclear what monospace would even mean—certainly not one or two cells per codepoint (e.g. Burmese or Tibetan; apparently there are Arabic monospace fonts[8] but I’ve no idea how the hell they work). Not coincidentally, those are the scripts where you really, really need that shaper, otherwise nothing looks anywhere close to correct. [This post could have been titled “Contra Muratori on Unicode in terminal emulators”.] [1] https://www.ecma-international.org/publications-and-standards/standards/ecma-48/ https://www.ecma-international.org/publications-and-standard... [2] https://www.ecma-international.org/publications-and-standards/technical-reports/ecma-tr-53/ https://www.ecma-international.org/publications-and-standard... [3] https://news.ycombinator.com/item?id=8086417 https://news.ycombinator.com/item?id=8086417 [4] https://terminal-wg.pages.freedesktop.org/bidi/ https://terminal-wg.pages.freedesktop.org/bidi/ [5] http://litcave.rudi.ir/ http://litcave.rudi.ir/ [6] https://github.com/n8willis/opentype-shaping-documents https://github.com/n8willis/opentype-shaping-documents [7] https://litherum.blogspot.com/2019/03/addition-font.html https://litherum.blogspot.com/2019/03/addition-font.html [8] https://news.ycombinator.com/item?id=10395464 https://news.ycombinator.com/item?id=10395464

kevin_thibedeau 4y ago

C11 has full support for Unicode in source code for string and char literals and comments. Any runtime Unicode processing, however, has to be provided by outside code.

5e92cb50239222b 4y ago

> If anyone knows of a good one for Linux, please let me know! kitty has excellent support for this stuff, and is much more performant than anything based on vte (like gnome-terminal). https://github.com/kovidgoyal/kitty https://github.com/kovidgoyal/kitty