3 ms·
From the article: * Gecko is huge and it uses TwoByte strings in most places. Converting all of Gecko to use UTF8 strings is a much bigger project and has its
by Macha 7y ago
From the article:
* Gecko is huge and it uses TwoByte strings in most places. Converting all of Gecko to use UTF8 strings is a much bigger project and has its own risks. As described below, we currently inflate Latin1 strings to TwoByte Gecko strings and that was also a potential performance risk, but inflating Latin1 is much faster than inflating UTF8.
* Linear-time indexing: operations like charAt require character indexing to be fast. We discussed solving this by adding a special flag to indicate all characters in the string are ASCII, so that we can still use O(1) indexing in this case. This scheme will only work for ASCII strings, though, so it’s a potential performance risk. An alternative is to have such operations inflate the string from UTF8 to TwoByte, but that’s also not ideal.
* Converting SpiderMonkey’s own string algorithms to work on UTF8 would require a lot more work. This includes changing the irregexp regular expression engine we imported from V8 a few months ago (it already had code to handle Latin1 strings).
- ninkendo 7y ago> Linear-time indexing This is great for ASCII when you know there's no such thing as combining characters/etc, but I would like to remind everyone reading this that there's no such thing as "linear indexing" of user-perceived characters in Unicode. User-perceived characters need processing in order to index due to Grapheme clusters potentially using many code points together. (https://unicode.org/reports/tr29/#Grapheme_Cluster_Boundaries https://unicode.org/reports/tr29/#Grapheme_Cluster_Boundarie...) For instance, (on my machine, a dark-skinned male teacher) is a combination of these characters: - U+1F468 Man - U+1F3FE Medium-Dark Skin Tone - U+200D Zero Width Joiner - U+1F3EB School And knowing what the byte index of the start/end of that character in a string cannot be done by just multiplying an offset by some multiple of number of bytes per character.
- cygx 7y agoI would like to remind everyone reading this that there's no such thing as "linear indexing" of user-perceived characters in Unicode. There is if you dynamically assign virtual code points to grapheme clusters.
- nitrogen 7y agoIsn't ASCII indexing constant time and Unicode grapheme indexing ~linear time?
- ninkendo 7y agoCorrect, i seem to have quoted the wrong part of the GP comment... I meant to address this quote: > adding a special flag to indicate all characters in the string are ASCII, so that we can still use O(1) indexing in this case But I got confused and addressed the linear indexing part. But yeah, ASCII is constant time, but you can’t assume even UTF-32 can be constant time due to variable length grapheme clusters.
- bradleyjg 7y agoAfter almost 4000 years have we concluded that the alphabet was a bad idea after all?
- BeeOnRope 7y agoI interpreted the OP's question as "Why not use UTF-8 as the secondary encoding in these types of hybrid string classes which use a narrower representation when possible", not "Why not convert all strings to UTF-8.