3 ms·
> Linear-time indexing This is great for ASCII when you know there's no such thing as combining characters/etc, but I would like to remind everyone reading thi
by ninkendo 7y ago
> Linear-time indexing
This is great for ASCII when you know there's no such thing as combining characters/etc, but I would like to remind everyone reading this that there's no such thing as "linear indexing" of user-perceived characters in Unicode.
User-perceived characters need processing in order to index due to Grapheme clusters potentially using many code points together. (https://unicode.org/reports/tr29/#Grapheme_Cluster_Boundaries https://unicode.org/reports/tr29/#Grapheme_Cluster_Boundarie...)
For instance, (on my machine, a dark-skinned male teacher) is a combination of these characters:
- U+1F468 Man
- U+1F3FE Medium-Dark Skin Tone
- U+200D Zero Width Joiner
- U+1F3EB School
And knowing what the byte index of the start/end of that character in a string cannot be done by just multiplying an offset by some multiple of number of bytes per character.
- cygx 7y agoI would like to remind everyone reading this that there's no such thing as "linear indexing" of user-perceived characters in Unicode. There is if you dynamically assign virtual code points to grapheme clusters.
- nitrogen 7y agoIsn't ASCII indexing constant time and Unicode grapheme indexing ~linear time?
- ninkendo 7y agoCorrect, i seem to have quoted the wrong part of the GP comment... I meant to address this quote: > adding a special flag to indicate all characters in the string are ASCII, so that we can still use O(1) indexing in this case But I got confused and addressed the linear indexing part. But yeah, ASCII is constant time, but you can’t assume even UTF-32 can be constant time due to variable length grapheme clusters.
- bradleyjg 7y agoAfter almost 4000 years have we concluded that the alphabet was a bad idea after all?