4 ms·
It's not sufficient to use code points right? Some characters, for instance your example of the black thumbs up emoji, are grapheme clusters [1] composed of mul
by arcticbull 7y ago
It's not sufficient to use code points right? Some characters, for instance your example of the black thumbs up emoji, are grapheme clusters [1] composed of multiple code points. I think you have to iterate in grapheme clusters and convert that back to an offset in the original underlying encoding.
If you just rely on code points you risk splitting up a grapheme cluster into (in your example) two graphemes, one in each sub-string, the left representing "black" and the right representing "thumbs up." Further, you actually need to utilize one of the unicode normalization forms to perform meaningful operations like comparison or sorting.
This is one thing Rust's string API gets right, allowing you to iterate over a string as UTF-8 bytes in constant time -- and, by walking, your choice of codepoints and (currently unstable, or in the unicode-segmentation crate) grapheme clusters. Even that though is a partial solution. [2]
Definitely a tough problem!
[1] https://mathias.gaunard.com/unicode/doc/html/unicode/introduction_to_unicode.html https://mathias.gaunard.com/unicode/doc/html/unicode/introdu...
[2] https://internals.rust-lang.org/t/support-for-grapheme-clusters-in-std/7339/4 https://internals.rust-lang.org/t/support-for-grapheme-clust...
- clauderoux 7y agoExactly my point. Most modern emojis cannot rely on pure codepoints to be extracted.
- arcticbull 7y agoMakes sense! Did you end up implementing normalization for your sub-string find, or did you work around it some other way? I couldn't seem to see it when skimming.
- clauderoux 7y agoYou can have a look on: s_is_emoji...
- clauderoux 7y agoIn https://github.com/naver/tamgu/blob/master/include/conversion.h https://github.com/naver/tamgu/blob/master/include/conversio..., I have implemented a class: agnostring which derives from "std::string". There are some methods to traverse a UTF8 string: begin(): to initialize the traversal end() : is true when the string is fully traversed next(): which goes to the next character and returns the current character. s.begin(); while (!s.end()) { u = s.next(); }
- arcticbull 7y agoVery cool, thanks!