4 ms·

Yeah I agree that it should be possible to use any of the indexing systems depending on your needs. My criticism of grapheme cluster based indexing being the "d

Yeah I agree that it should be possible to use any of the indexing systems depending on your needs. My criticism of grapheme cluster based indexing being the "default" is that it only really makes sense for UI code. In other situations (eg, passing JSON text over the network) it adds a lot of complexity and code size, with no benefit. If I'm writing a regex parsing system, or a JSON parser, or an event logger, or a compiler, I'll use a lot of strings. But my code probably doesn't care very much about grapheme cluster boundaries. Handling grapheme cluster boundaries is very complex code. (Rust's unicode-segmentation crate is just shy of 100kb!). When parsing grapheme boundaries isn't needed, all that code is pure overhead.

burntsushi 4y ago
This is just an FYI. I don't mean to say much to your overall point, although, as someone else who has spent a lot of time doing Unicode-y things, I do tend to agree with you. I had a very similar discussion a bit ago.[1] (A quick glance at that discussion might suggest that I disagree with you, but see the last paragraph of my big comment in that thread.) Putting that aside, at least with respect to grapheme segmentation, it might be a little simpler than you think. But maybe only a little. The unicode-segmentation crate also does word segmentation, which is quite a bit more complicated than grapheme segmentation. For example, you can write a regex to parse graphemes without too much fuss[2]. (Compare that with the word segmentation regex, much to my chagrin.[3]) Once you build the regex, actually using it is basically as simple as running the regex.[4] Sadly, not all regex engines will be able to parse that regex due to its use of somewhat obscure Unicode properties. But the Rust regex crate can. :-) And of course, this somewhat shifts code size to heap size. So there's that too. But bottom line is, if you have a nice regex engine available to you, you can whip up a grapheme segmenter pretty quickly. And some regex engines even have grapheme segmentation built in via \X. [1]: https://github.com/BurntSushi/aho-corasick/issues/72 https://github.com/BurntSushi/aho-corasick/issues/72 [2]: https://github.com/BurntSushi/bstr/blob/e38e7a7ca986f9499b30202f49d79e531d14d192/scripts/regex/grapheme.sh https://github.com/BurntSushi/bstr/blob/e38e7a7ca986f9499b30... [3]: https://github.com/BurntSushi/bstr/blob/e38e7a7ca986f9499b30202f49d79e531d14d192/scripts/regex/word.sh#L1 https://github.com/BurntSushi/bstr/blob/e38e7a7ca986f9499b30... [4]: https://github.com/BurntSushi/bstr/blob/e38e7a7ca986f9499b30202f49d79e531d14d192/src/unicode/grapheme.rs#L185-L204 https://github.com/BurntSushi/bstr/blob/e38e7a7ca986f9499b30...
marcosdumay 4y ago
Well, it's for human language, not only UI. If you are indexing it for search, pattern matching, or validating, you will want graphemes. But yes, computer languages are normally designed so you can parse them at the byte view. Except for a small, but still surprising common set of them that are tokenized at the byte view, but parsed after some human language dependent transformation that happens over graphemes. (And those do usually have problem with internationalization.)
- josephg 4y ago
  If your parsing code cares about grapheme cluster boundaries, it'll behave differently on the same string when the unicode spec changes. So it may work differently on different operating systems, or with different versions of your unicode libraries. And that means the output of your validation methods or tokenizers will randomly change! I'm happy for the rendering of strings to change over time. But in my business logic, compilers and database code: That sounds like an unnecessary source of bugs.

burntsushi 4y ago

This is just an FYI. I don't mean to say much to your overall point, although, as someone else who has spent a lot of time doing Unicode-y things, I do tend to agree with you. I had a very similar discussion a bit ago.[1] (A quick glance at that discussion might suggest that I disagree with you, but see the last paragraph of my big comment in that thread.) Putting that aside, at least with respect to grapheme segmentation, it might be a little simpler than you think. But maybe only a little. The unicode-segmentation crate also does word segmentation, which is quite a bit more complicated than grapheme segmentation. For example, you can write a regex to parse graphemes without too much fuss[2]. (Compare that with the word segmentation regex, much to my chagrin.[3]) Once you build the regex, actually using it is basically as simple as running the regex.[4] Sadly, not all regex engines will be able to parse that regex due to its use of somewhat obscure Unicode properties. But the Rust regex crate can. :-) And of course, this somewhat shifts code size to heap size. So there's that too. But bottom line is, if you have a nice regex engine available to you, you can whip up a grapheme segmenter pretty quickly. And some regex engines even have grapheme segmentation built in via \X. [1]: https://github.com/BurntSushi/aho-corasick/issues/72 https://github.com/BurntSushi/aho-corasick/issues/72 [2]: https://github.com/BurntSushi/bstr/blob/e38e7a7ca986f9499b30202f49d79e531d14d192/scripts/regex/grapheme.sh https://github.com/BurntSushi/bstr/blob/e38e7a7ca986f9499b30... [3]: https://github.com/BurntSushi/bstr/blob/e38e7a7ca986f9499b30202f49d79e531d14d192/scripts/regex/word.sh#L1 https://github.com/BurntSushi/bstr/blob/e38e7a7ca986f9499b30... [4]: https://github.com/BurntSushi/bstr/blob/e38e7a7ca986f9499b30202f49d79e531d14d192/src/unicode/grapheme.rs#L185-L204 https://github.com/BurntSushi/bstr/blob/e38e7a7ca986f9499b30...

marcosdumay 4y ago

Well, it's for human language, not only UI. If you are indexing it for search, pattern matching, or validating, you will want graphemes. But yes, computer languages are normally designed so you can parse them at the byte view. Except for a small, but still surprising common set of them that are tokenized at the byte view, but parsed after some human language dependent transformation that happens over graphemes. (And those do usually have problem with internationalization.)

josephg 4y ago
If your parsing code cares about grapheme cluster boundaries, it'll behave differently on the same string when the unicode spec changes. So it may work differently on different operating systems, or with different versions of your unicode libraries. And that means the output of your validation methods or tokenizers will randomly change! I'm happy for the rendering of strings to change over time. But in my business logic, compilers and database code: That sounds like an unnecessary source of bugs.