4 ms·
This is just an FYI. I don't mean to say much to your overall point, although, as someone else who has spent a lot of time doing Unicode-y things, I do tend to
by burntsushi 4y ago
This is just an FYI. I don't mean to say much to your overall point, although, as someone else who has spent a lot of time doing Unicode-y things, I do tend to agree with you. I had a very similar discussion a bit ago.[1] (A quick glance at that discussion might suggest that I disagree with you, but see the last paragraph of my big comment in that thread.)
Putting that aside, at least with respect to grapheme segmentation, it might be a little simpler than you think. But maybe only a little. The unicode-segmentation crate also does word segmentation, which is quite a bit more complicated than grapheme segmentation. For example, you can write a regex to parse graphemes without too much fuss[2]. (Compare that with the word segmentation regex, much to my chagrin.[3]) Once you build the regex, actually using it is basically as simple as running the regex.[4]
Sadly, not all regex engines will be able to parse that regex due to its use of somewhat obscure Unicode properties. But the Rust regex crate can. :-)
And of course, this somewhat shifts code size to heap size. So there's that too. But bottom line is, if you have a nice regex engine available to you, you can whip up a grapheme segmenter pretty quickly. And some regex engines even have grapheme segmentation built in via \X.
[1]: https://github.com/BurntSushi/aho-corasick/issues/72 https://github.com/BurntSushi/aho-corasick/issues/72
[2]: https://github.com/BurntSushi/bstr/blob/e38e7a7ca986f9499b30202f49d79e531d14d192/scripts/regex/grapheme.sh https://github.com/BurntSushi/bstr/blob/e38e7a7ca986f9499b30...
[3]: https://github.com/BurntSushi/bstr/blob/e38e7a7ca986f9499b30202f49d79e531d14d192/scripts/regex/word.sh#L1 https://github.com/BurntSushi/bstr/blob/e38e7a7ca986f9499b30...
[4]: https://github.com/BurntSushi/bstr/blob/e38e7a7ca986f9499b30202f49d79e531d14d192/src/unicode/grapheme.rs#L185-L204 https://github.com/BurntSushi/bstr/blob/e38e7a7ca986f9499b30...