5 ms·
Because it depends on string encoding to define length. Let's say you have a UTF-8 string that stores emoji, what would the length be? You have the binary lengt
by stevefan1999 4y ago
Because it depends on string encoding to define length. Let's say you have a UTF-8 string that stores emoji, what would the length be? You have the binary length (that is, up to 7 in the emoji example as it could store up to 7 bytes), or you have the canonical length (which is 1 because we have 1 visible emoji symbol).
Thus the concept of string length itself already depends on different types of string. Unless it is strictly defined like Java and Rust do, you can't be sure what to store in the prefix. And even that makes string interoperability still exceedingly hard.
That's why I have a question though: Why don't people design a "small string" to prepend just one unsigned byte into the string as a fast path to check length? This operation will thus be O(1) if you have a string length less than 255, and if it hits 255/-1, we can determine that it is a long string and fallback to do null check. Other than the stale string issue (which means the actual length does not respect the stoted length) I found no more major issues with that.
Side note: The aforementioned encoding problem gets serious with regards to database collation. Far as I know, Redis used binary collation while other DBMS will need user to determine a default collation, most likely Latin-1.
> There’s an argument that taking a substring off a C string a) can be done without allocating a copy b) can be done with pure pointer manipulation. In practice though I’m not sure how critical that case is.
You can't. What if the original data mutated? To take a substring off a C string without copy in practice, you will at least need a copy-on-write data structure. In some system however this can be done with virtual memory by issuing page fault to redirect to the same source memory. In theory you should use something like a Rope [1] or trie or even some Clojure persistent vector black magic that God knows what the hell is that about [2]
[1]: https://en.wikipedia.org/wiki/Rope_%28data_structure%29?wprov=sfla1 https://en.wikipedia.org/wiki/Rope_%28data_structure%29?wpro...
[2]: https://hypirion.com/musings/understanding-persistent-vector-pt-1 https://hypirion.com/musings/understanding-persistent-vector...
- masklinn 4y ago> Because it depends on string encoding to define length. Let's say you have a UTF-8 string that stores emoji, what would the length be? The encoded length. First because it’s necessary, second because it’s C and would not bother with anything else, third because USV length is useless, fourth because grapheme clusters have dependencies on the locale and data files, and are only useful in some cases. And their relation with code units / USVs is irregular (adding USVs can decrease GCs).
- ajb 4y agoAlthough true today, that's definitely not the original reason. Unicode was invented after C
- bawolff 4y agoBesides the historical context others have pointed out, I'm pretty sure most people looking for a string length want the number of bytes. The number of code points in a unicode string is almost never useful. At most you might want width rendered on a screen, but that has nothing to do with number of codepoints. > You have the binary length (that is, up to 7 in the emoji example as it could store up to 7 bytes), or you have the canonical length (which is 1 because we have 1 visible emoji symbol). These numbers aren't really accurate. It depends how you define a "single" emoiji, but there are some that are > 7 utf-8 bytes, and there are some that are > 1 code point. E.g. the rainbow flag is a single graphical symbol that is 14 bytes long and 4 code points > Far as I know, Redis used binary collation while other DBMS will need user to determine a default collation, most likely Latin-1. Latin1 is a charset not a collation.