7 ms·

Libgrapheme: A simple freestanding C99 library for Unicode

108 points by harporoeder 4y ago

manifoldgeo 4y ago
I've been trying to get into C this past week, and this is a great coincidence to see this today! I was just thinking how convenient it is to type emojis right into strings in Python and print them. I assumed C didn't have much unicode compatibility, though I didn't research it. I gave libgrapheme a try, and it compiled just as the instructions said it would. The hello-world program also mostly worked, but in my terminal it malformed several things. For example the American flag emoji rendered in my terminal as [U][S], and the family emoji rendered as three distinct emoji faces (side-by-side) rather than one grouped one. I went to a website that lets me copy emojis to my clipboard, and I directly copy-pasted the American flag into my terminal, and I still got [U][S], so I think the problem is just with the terminal and not the library. edit: Indeed, this is a problem in Gnome terminal. I found a Bugzilla link[0] that is still open. The official name for the grouped emoji type is "ZWJ sequence"[1], short for Zero-Width Joiner, and it appears not a lot of terminals support them. If anyone knows of a good one for Linux, please let me know! Great stuff, thank you for sharing! References: [0]: https://gitlab.gnome.org/GNOME/vte/-/issues/2317 https://gitlab.gnome.org/GNOME/vte/-/issues/2317 [1]: https://emojipedia.org/emoji-zwj-sequence/ https://emojipedia.org/emoji-zwj-sequence/
- csande17 4y ago
  > I was just thinking how convenient it is to type emojis right into strings in Python and print them. I assumed C didn't have much unicode compatibility, though I didn't research it. Libgrapheme is a nice library, but it doesn't really have anything to do with this. Almost all modern terminal emulators use the UTF-8 character encoding. In order to successfully output Unicode characters, your programming language doesn't actually need much "Unicode support"; it just needs to be able to send UTF-8-encoded bytes to stdout. (That's why many modern programming languages like Go and Zig define strings as a simple array of bytes.) Modern C compilers allow you to printf("é") and get the appropriate behavior. As you mention, the terminal emulator also needs to be able to decode and display those UTF-8 bytes correctly, and a lot of terminals don't get it right in some situations. Off the top of my head, I don't know of a terminal that actually implements the entire (very complex) set of Unicode text rendering behaviors; maybe one of the web-based ones that run in Electron? macOS's Terminal.app is also pretty good IIRC. Where libgrapheme comes in is if you want to analyze or manipulate a UTF-8-encoded string. It provides operations like "split into words" and "convert to uppercase". A surprising number of programs never need to do that stuff, but if you do, libgrapheme will give you a Unicode-compatible implementation. (Many more basic operations, like concatenating two strings, will work just fine without libgrapheme.)
  - mananaysiempre 4y ago
    (Not a language or Unicode expert, the following likely has important mistakes.) > Off the top of my head, I don't know of a terminal that actually implements the entire (very complex) set of Unicode text rendering behaviors There are at least two reasons for this: First, nobody actually seems to know how bidirectional text should interact with terminal control sequences, or indeed how it should be typeset on a terminal in the first place (what is the primary direction? where are the reordering boundaries?). There is the pre-Unicode bidirectional support mode (BDSM, I kid you not) in ECMA-48[1] and TR/53[2], which AFAIK nobody implements nor cares about; there are terminal emulators endorsed by bidi-language users[3], which AFAIK nobody has written down the behaviour of; there is the Freedesktop bidi terminal spec[4], which is a draft and AFAIK nobody implements yet either but at least some people care about; finally, there are bidi-language users who say that spec is a mistake[5]. Second, aside from bidi and a smattering of other things such as emoji, there is no detailed “Unicode text rendering behaviour”, only standards specific to font formats—the most recent among them being OpenType, which is dubiously compatible across implementations, decently documented only through painstaking reverse engineering (sometimes in words[6], sometimes only in Freetype library code), and generally full of snakes[7]. And it has no notion of a monospace font—only of a (proportional) font where all Lat/Cyr/Grk characters just happen to have the same advance. AFAICT that is not negligence or an oversight, but rather a concession to the fact that there are scripts which don’t really have a notion of monospace in the typographic tradition and in fact are written such that it’s extremely unclear what monospace would even mean—certainly not one or two cells per codepoint (e.g. Burmese or Tibetan; apparently there are Arabic monospace fonts[8] but I’ve no idea how the hell they work). Not coincidentally, those are the scripts where you really, really need that shaper, otherwise nothing looks anywhere close to correct. [This post could have been titled “Contra Muratori on Unicode in terminal emulators”.] [1] https://www.ecma-international.org/publications-and-standards/standards/ecma-48/ https://www.ecma-international.org/publications-and-standard... [2] https://www.ecma-international.org/publications-and-standards/technical-reports/ecma-tr-53/ https://www.ecma-international.org/publications-and-standard... [3] https://news.ycombinator.com/item?id=8086417 https://news.ycombinator.com/item?id=8086417 [4] https://terminal-wg.pages.freedesktop.org/bidi/ https://terminal-wg.pages.freedesktop.org/bidi/ [5] http://litcave.rudi.ir/ http://litcave.rudi.ir/ [6] https://github.com/n8willis/opentype-shaping-documents https://github.com/n8willis/opentype-shaping-documents [7] https://litherum.blogspot.com/2019/03/addition-font.html https://litherum.blogspot.com/2019/03/addition-font.html [8] https://news.ycombinator.com/item?id=10395464 https://news.ycombinator.com/item?id=10395464
- kevin_thibedeau 4y ago
  C11 has full support for Unicode in source code for string and char literals and comments. Any runtime Unicode processing, however, has to be provided by outside code.
- 5e92cb50239222b 4y ago
  > If anyone knows of a good one for Linux, please let me know! kitty has excellent support for this stuff, and is much more performant than anything based on vte (like gnome-terminal). https://github.com/kovidgoyal/kitty https://github.com/kovidgoyal/kitty
sylware 4y ago
- tomcam 4y ago
  - dang 4y ago
    Personal attacks will get you banned here, regardless of what you're replying to. If you'd please review https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html and not post like this to HN, we'd appreciate it.
daaaaaaan 4y ago
- raspyberr 4y ago
  https://en.wikipedia.org/wiki/Talk:Suckless.org https://en.wikipedia.org/wiki/Talk:Suckless.org
- copperwater69 4y ago
- platinumrad 4y ago
  You can walk at night holding a torch without being a Nazi.
gigel82 4y ago
This is interesting, particularly for implementing Intl in JS engines without the mega-heavy ICU. But I wonder how portable it really is. Sometimes I have to dig very deep to find that what folks call "portable C" is actually POSIX-dependent. It doesn't appear to be the case after going through the code for a bit, so that's promising.
- mananaysiempre 4y ago
  You can also refer to the Unicode routines of other small JS engines[1,2], those don’t use ICU either, although the implementations are mercilessly size-optimized (to put it politely) and restricted to what the target JS version requires (e.g. Duktape does casemapping but no normalization). Still, Bellard’s in particular look like he had a small Unicode processing library lying around and just copied it into the tree, not like he was forced to write the absolute minimum to do a JS inteprerer, so they can even be compared with dedicated libraries like libgrapheme, libutf8proc or libutf. [1] https://github.com/bellard/quickjs/blob/master/libunicode.c https://github.com/bellard/quickjs/blob/master/libunicode.c [2] https://github.com/svaarala/duktape/blob/master/src-input/duk_unicode.h https://github.com/svaarala/duktape/blob/master/src-input/du...
- dochtman 4y ago
  Maybe a comparison to ICU4X is more interesting.
- layer8 4y ago
  The functionality they’re implementing is purely string/byte processing, I don’t see what should introduce use of non-POSIX features.
- dmitriid 4y ago
  > This is interesting, particularly for implementing Intl in JS engines without the mega-heavy ICU. On the contrary, every single programming language and platform should by default include ICU and provide easy bindings to its functionality. Because even normalization and equality comparisons are often done poorly or not at all in these "simple" libraries.
__d 4y ago
Does anyone know offhand whether this does comparisons? And normalization?
- cryptonector 4y ago
  It looks like "no", just browsing its docs.
  - __d 4y ago
    Yeah, I didn't see anything in the man pages. For my purposes, a Unicode library that doesn't do string comparisons is, well ... incomplete. It's a fairly fundamental thing that you need to do: I would have thought well before you'd want eg. up/down-casing, for instance.
    - cryptonector 4y ago
      Illumos has one: https://src.illumos.org/source/xref/illumos-gate/usr/src/common/unicode/u8_textprep.c?r=85bb5f1d#1728 https://src.illumos.org/source/xref/illumos-gate/usr/src/com...
HexDecOctBin 4y ago
Is there any similar C library that deals with Normalisation and Collation?
- rockwotj 4y ago
  https://github.com/JuliaStrings/utf8proc https://github.com/JuliaStrings/utf8proc does normalization
  - HexDecOctBin 4y ago
    I was hoping for some alternative since the linked article mentions utf8proc as such: > Some libraries, like libutf8proc and libunistring, are incorrect by basing their API on assumptions that haven't been true for years (e.g. offering stateless grapheme cluster segmentation even though the underlying algorithm is not stateless). As an additional factor, libutf8proc's UTF-8-decoder is unsafe, as it allows overlong encodings that can be easily used for exploits.
    - rockwotj 4y ago
      There is a stateful API too, and I'm surprised they mention it's decoder is unsafe, utf8proc is fuzzed by oss fuzz, which I assume would catch something like that.
1letterunixname 4y ago
Oh my. Don't roll your own crypto or Unicode library. There are too many edge cases that "simplicity" simply doesn't scale well with. Use what works and what's already been done rather than reinventing the wheel without a clear and necessary purpose.
- up2isomorphism 4y ago
  This is very general argument that can be used when anybody is trying to write another software that does same thing as the existing ones.
  - einmaliger 4y ago
    I think it specifically applies to implementing standards that are known to be more difficult to implement and easily break.
hnfong 4y ago
The claims about "word segmentation" caught my eye, because it's not even well defined in my native language. (one example among the difficulties: there's a lot of constructs like "rid-fucking-iculous". Should it be "rid/fucking/iculous" or "fucking ridiculous"?) After checking the source to see how they perform this literally impossible feat, it turns out they were implementing a Unicode standard that basically tries to do something useful (for some families of languages) but has a whole wall of caveats: https://unicode.org/reports/tr29/#Word_Boundary_Rules https://unicode.org/reports/tr29/#Word_Boundary_Rules I can see users of the word segmentation function shooting their feet really badly with at least CJK languages, especially with a deceptively simple API like that. In general "word segmentation" doesn't make sense on a language-neutral level. libgrapheme even admits to this somewhat: """ For some languages, for instance, it is necessary to have a dictionary on hand to always accurately determine when a word begins and ends. The defaults provided by the standard, though, already do a great job respecting the language's boundaries in the general case """ I disagree about the "great job" part. It's probably the best job a Unicode spec can do, but it's not going to be great for like at least 20% of the world's population... (Also a minor nit is that no dictionary is accurate, so even with a dictionary the results are not really "always accurate", just mostly accurate, unless you hit edge cases [for CJK at least], upon which advanced NLP techniques are needed.) IMHO they should have a big disclaimer in their GRAPHEME_NEXT_WORD_BREAK(3) manual page warning users about the caveats (like, "if you're using this for anything other than $(these families of languages) make sure you really know what you're doing before using this function").

108 points by harporoeder 4y ago

I've been trying to get into C this past week, and this is a great coincidence to see this today! I was just thinking how convenient it is to type emojis right into strings in Python and print them. I assumed C didn't have much unicode compatibility, though I didn't research it. I gave libgrapheme a try, and it compiled just as the instructions said it would. The hello-world program also mostly worked, but in my terminal it malformed several things. For example the American flag emoji rendered in my terminal as [U][S], and the family emoji rendered as three distinct emoji faces (side-by-side) rather than one grouped one. I went to a website that lets me copy emojis to my clipboard, and I directly copy-pasted the American flag into my terminal, and I still got [U][S], so I think the problem is just with the terminal and not the library. edit: Indeed, this is a problem in Gnome terminal. I found a Bugzilla link[0] that is still open. The official name for the grouped emoji type is "ZWJ sequence"[1], short for Zero-Width Joiner, and it appears not a lot of terminals support them. If anyone knows of a good one for Linux, please let me know! Great stuff, thank you for sharing! References: [0]: https://gitlab.gnome.org/GNOME/vte/-/issues/2317 https://gitlab.gnome.org/GNOME/vte/-/issues/2317 [1]: https://emojipedia.org/emoji-zwj-sequence/ https://emojipedia.org/emoji-zwj-sequence/

csande17 4y ago
> I was just thinking how convenient it is to type emojis right into strings in Python and print them. I assumed C didn't have much unicode compatibility, though I didn't research it. Libgrapheme is a nice library, but it doesn't really have anything to do with this. Almost all modern terminal emulators use the UTF-8 character encoding. In order to successfully output Unicode characters, your programming language doesn't actually need much "Unicode support"; it just needs to be able to send UTF-8-encoded bytes to stdout. (That's why many modern programming languages like Go and Zig define strings as a simple array of bytes.) Modern C compilers allow you to printf("é") and get the appropriate behavior. As you mention, the terminal emulator also needs to be able to decode and display those UTF-8 bytes correctly, and a lot of terminals don't get it right in some situations. Off the top of my head, I don't know of a terminal that actually implements the entire (very complex) set of Unicode text rendering behaviors; maybe one of the web-based ones that run in Electron? macOS's Terminal.app is also pretty good IIRC. Where libgrapheme comes in is if you want to analyze or manipulate a UTF-8-encoded string. It provides operations like "split into words" and "convert to uppercase". A surprising number of programs never need to do that stuff, but if you do, libgrapheme will give you a Unicode-compatible implementation. (Many more basic operations, like concatenating two strings, will work just fine without libgrapheme.)
- mananaysiempre 4y ago
  (Not a language or Unicode expert, the following likely has important mistakes.) > Off the top of my head, I don't know of a terminal that actually implements the entire (very complex) set of Unicode text rendering behaviors There are at least two reasons for this: First, nobody actually seems to know how bidirectional text should interact with terminal control sequences, or indeed how it should be typeset on a terminal in the first place (what is the primary direction? where are the reordering boundaries?). There is the pre-Unicode bidirectional support mode (BDSM, I kid you not) in ECMA-48[1] and TR/53[2], which AFAIK nobody implements nor cares about; there are terminal emulators endorsed by bidi-language users[3], which AFAIK nobody has written down the behaviour of; there is the Freedesktop bidi terminal spec[4], which is a draft and AFAIK nobody implements yet either but at least some people care about; finally, there are bidi-language users who say that spec is a mistake[5]. Second, aside from bidi and a smattering of other things such as emoji, there is no detailed “Unicode text rendering behaviour”, only standards specific to font formats—the most recent among them being OpenType, which is dubiously compatible across implementations, decently documented only through painstaking reverse engineering (sometimes in words[6], sometimes only in Freetype library code), and generally full of snakes[7]. And it has no notion of a monospace font—only of a (proportional) font where all Lat/Cyr/Grk characters just happen to have the same advance. AFAICT that is not negligence or an oversight, but rather a concession to the fact that there are scripts which don’t really have a notion of monospace in the typographic tradition and in fact are written such that it’s extremely unclear what monospace would even mean—certainly not one or two cells per codepoint (e.g. Burmese or Tibetan; apparently there are Arabic monospace fonts[8] but I’ve no idea how the hell they work). Not coincidentally, those are the scripts where you really, really need that shaper, otherwise nothing looks anywhere close to correct. [This post could have been titled “Contra Muratori on Unicode in terminal emulators”.] [1] https://www.ecma-international.org/publications-and-standards/standards/ecma-48/ https://www.ecma-international.org/publications-and-standard... [2] https://www.ecma-international.org/publications-and-standards/technical-reports/ecma-tr-53/ https://www.ecma-international.org/publications-and-standard... [3] https://news.ycombinator.com/item?id=8086417 https://news.ycombinator.com/item?id=8086417 [4] https://terminal-wg.pages.freedesktop.org/bidi/ https://terminal-wg.pages.freedesktop.org/bidi/ [5] http://litcave.rudi.ir/ http://litcave.rudi.ir/ [6] https://github.com/n8willis/opentype-shaping-documents https://github.com/n8willis/opentype-shaping-documents [7] https://litherum.blogspot.com/2019/03/addition-font.html https://litherum.blogspot.com/2019/03/addition-font.html [8] https://news.ycombinator.com/item?id=10395464 https://news.ycombinator.com/item?id=10395464
kevin_thibedeau 4y ago
C11 has full support for Unicode in source code for string and char literals and comments. Any runtime Unicode processing, however, has to be provided by outside code.
5e92cb50239222b 4y ago
> If anyone knows of a good one for Linux, please let me know! kitty has excellent support for this stuff, and is much more performant than anything based on vte (like gnome-terminal). https://github.com/kovidgoyal/kitty https://github.com/kovidgoyal/kitty

sylware 4y ago

tomcam 4y ago
- dang 4y ago
  Personal attacks will get you banned here, regardless of what you're replying to. If you'd please review https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html and not post like this to HN, we'd appreciate it.

daaaaaaan 4y ago

raspyberr 4y ago
https://en.wikipedia.org/wiki/Talk:Suckless.org https://en.wikipedia.org/wiki/Talk:Suckless.org
copperwater69 4y ago
platinumrad 4y ago
You can walk at night holding a torch without being a Nazi.

gigel82 4y ago

This is interesting, particularly for implementing Intl in JS engines without the mega-heavy ICU. But I wonder how portable it really is. Sometimes I have to dig very deep to find that what folks call "portable C" is actually POSIX-dependent. It doesn't appear to be the case after going through the code for a bit, so that's promising.

mananaysiempre 4y ago
You can also refer to the Unicode routines of other small JS engines[1,2], those don’t use ICU either, although the implementations are mercilessly size-optimized (to put it politely) and restricted to what the target JS version requires (e.g. Duktape does casemapping but no normalization). Still, Bellard’s in particular look like he had a small Unicode processing library lying around and just copied it into the tree, not like he was forced to write the absolute minimum to do a JS inteprerer, so they can even be compared with dedicated libraries like libgrapheme, libutf8proc or libutf. [1] https://github.com/bellard/quickjs/blob/master/libunicode.c https://github.com/bellard/quickjs/blob/master/libunicode.c [2] https://github.com/svaarala/duktape/blob/master/src-input/duk_unicode.h https://github.com/svaarala/duktape/blob/master/src-input/du...
dochtman 4y ago
Maybe a comparison to ICU4X is more interesting.
layer8 4y ago
The functionality they’re implementing is purely string/byte processing, I don’t see what should introduce use of non-POSIX features.
dmitriid 4y ago
> This is interesting, particularly for implementing Intl in JS engines without the mega-heavy ICU. On the contrary, every single programming language and platform should by default include ICU and provide easy bindings to its functionality. Because even normalization and equality comparisons are often done poorly or not at all in these "simple" libraries.

__d 4y ago

Does anyone know offhand whether this does comparisons? And normalization?

cryptonector 4y ago
It looks like "no", just browsing its docs.
- __d 4y ago
  Yeah, I didn't see anything in the man pages. For my purposes, a Unicode library that doesn't do string comparisons is, well ... incomplete. It's a fairly fundamental thing that you need to do: I would have thought well before you'd want eg. up/down-casing, for instance.
  - cryptonector 4y ago
    Illumos has one: https://src.illumos.org/source/xref/illumos-gate/usr/src/common/unicode/u8_textprep.c?r=85bb5f1d#1728 https://src.illumos.org/source/xref/illumos-gate/usr/src/com...

HexDecOctBin 4y ago

Is there any similar C library that deals with Normalisation and Collation?

rockwotj 4y ago
https://github.com/JuliaStrings/utf8proc https://github.com/JuliaStrings/utf8proc does normalization
- HexDecOctBin 4y ago
  I was hoping for some alternative since the linked article mentions utf8proc as such: > Some libraries, like libutf8proc and libunistring, are incorrect by basing their API on assumptions that haven't been true for years (e.g. offering stateless grapheme cluster segmentation even though the underlying algorithm is not stateless). As an additional factor, libutf8proc's UTF-8-decoder is unsafe, as it allows overlong encodings that can be easily used for exploits.
  - rockwotj 4y ago
    There is a stateful API too, and I'm surprised they mention it's decoder is unsafe, utf8proc is fuzzed by oss fuzz, which I assume would catch something like that.

1letterunixname 4y ago

Oh my. Don't roll your own crypto or Unicode library. There are too many edge cases that "simplicity" simply doesn't scale well with. Use what works and what's already been done rather than reinventing the wheel without a clear and necessary purpose.

up2isomorphism 4y ago
This is very general argument that can be used when anybody is trying to write another software that does same thing as the existing ones.
- einmaliger 4y ago
  I think it specifically applies to implementing standards that are known to be more difficult to implement and easily break.

hnfong 4y ago

The claims about "word segmentation" caught my eye, because it's not even well defined in my native language. (one example among the difficulties: there's a lot of constructs like "rid-fucking-iculous". Should it be "rid/fucking/iculous" or "fucking ridiculous"?) After checking the source to see how they perform this literally impossible feat, it turns out they were implementing a Unicode standard that basically tries to do something useful (for some families of languages) but has a whole wall of caveats: https://unicode.org/reports/tr29/#Word_Boundary_Rules https://unicode.org/reports/tr29/#Word_Boundary_Rules I can see users of the word segmentation function shooting their feet really badly with at least CJK languages, especially with a deceptively simple API like that. In general "word segmentation" doesn't make sense on a language-neutral level. libgrapheme even admits to this somewhat: """ For some languages, for instance, it is necessary to have a dictionary on hand to always accurately determine when a word begins and ends. The defaults provided by the standard, though, already do a great job respecting the language's boundaries in the general case """ I disagree about the "great job" part. It's probably the best job a Unicode spec can do, but it's not going to be great for like at least 20% of the world's population... (Also a minor nit is that no dictionary is accurate, so even with a dictionary the results are not really "always accurate", just mostly accurate, unless you hit edge cases [for CJK at least], upon which advanced NLP techniques are needed.) IMHO they should have a big disclaimer in their GRAPHEME_NEXT_WORD_BREAK(3) manual page warning users about the caveats (like, "if you're using this for anything other than $(these families of languages) make sure you really know what you're doing before using this function").