11 ms·

In summary the two useful measures of unicode length are: * Number of native (preferably UTF-8) code units * Number of extended grapheme clusters The first i

In summary the two useful measures of unicode length are: * Number of native (preferably UTF-8) code units * Number of extended grapheme clusters The first is useful for basic string operations. The second is good for telling you what a user would consider a "character".

bikeshaving 6y ago
Why do people prefer UTF-8 coordinates? While for storage I think we should use UTF-8, when working with strings live it’s just so much easier to use UTF-16 because it’s predictable: 1 unit for the basic plane and 2 for everything else (the multi-character emoji and modifier stuff aside). I am probably biased because I mostly use and think about DOMStrings which are always UTF-16 but I’m not sure why people who use languages which are more flexible about string representations than JavaScript would not also appreciate this kind of regularity.
- pdpi 6y ago
  I struggle to see why you'd ever want UTF-16. If you're using a variable length encoding, might as well stick to UTF-8. If you want predictable sizes, there's UTF-32 instead. ~Also, DOM Strings are not UTF-16, they're UCS-16.~ EDIT: UCS-2, not UCS-16. Also, I'm confusing the DOM with EcmaScript, and even that hasn't been true in a while.
  - ChrisSD 6y ago
    > Also, DOM Strings are not UTF-16, they're UCS-16. Hm, according to the spec they should be interpreted as UTF-16 but this isn't enforced by the language so it can contain unpaired surrogates: From https://heycam.github.io/webidl/#idl-DOMString https://heycam.github.io/webidl/#idl-DOMString > Such sequences are commonly interpreted as UTF-16 encoded strings [RFC2781] although this is not required... Nothing in this specification requires a DOMString value to be a valid UTF-16 string. From https://262.ecma-international.org/11.0/#sec-ecmascript-language-types-string-type https://262.ecma-international.org/11.0/#sec-ecmascript-lang... > The String type is the set of all ordered sequences of zero or more 16-bit unsigned integer values (“elements”) up to a maximum length of 253 - 1 elements. The String type is generally used to represent textual data in a running ECMAScript program, in which case each element in the String is treated as a UTF-16 code unit value... Operations that do interpret String values treat each element as a single UTF-16 code unit. However, ECMAScript does not restrict the value of or relationships between these code units, so operations that further interpret String contents as sequences of Unicode code points encoded in UTF-16 must account for ill-formed subsequences.
  - bikeshaving 6y ago
    UTF-32 is a bit like giving up and saying code units equal code points right? I’m more interested in the comparison between UTF-8 and UTF-16, where UTF-8 requires 1 to 3 bytes just in the BMP, with 3 bytes for CJK characters. I’m saying that as a quick measure of the actual length of a string, UTF-16 is much more predictable and provides a nice intuitive estimation of how long text actually is, as well as providing a fair encoding for most commonly used languages. RE the UTF-16 vs UCS-2 stuff, that’s probably a distinction which has technical meaning but will collapse at some point because no one actually cares, much like the distinction between URI and URL.
    - masklinn 6y ago
      > I’m saying that as a quick measure of the actual length of a string, UTF-16 is much more predictable Meaning the software will deal much less well when it's wrong. > and provides a nice intuitive estimation of how long text actually is Not really due to combining codepoints, which make it not useful. > as well as providing a fair encoding for most commonly used languages. Which we know effectively doesn't matter: it's essentially only a gain for pure CJK text being stored at rest, because otherwise the waste on ASCII will more than compensate for the gain.
  - josephg 6y ago
    Yes UTF-16 / UCS-2 is a silly encoding and in retrospect it was a mistake. It matters because for awhile it was believed that 2 bytes would be enough to encode any Unicode character. During this time, lots of important languages appeared which figured a 2 byte fixed size encoding was better than a variable length encoding. So Java, javascript, C# and others all use UCS-2. Now they suffer the downsides of both using a variable length encodings and being memory inefficient. The string.length property in all these languages is almost totally meaningless. UTF-8 is the encoding you should generally always reach for when designing new systems, or when implementing a network protocol. Rust, Go and other newer languages all use UTF-8 internally because it’s better. Well, and in Go’s case because it’s author, Rob Pike also had a hand in inventing UTF-8. Ironically C and UNIX, which (mostly) stubbornly stuck with single byte character encodings generally works better with UTF-8 than a lot of newer languages.
- ChrisSD 6y ago
  The distinction between the basic plane and everything else is not particularly useful, imho. So I'm not sure I understand what the advantage of UTF-16 is here? It's an arbitrary division either way.
- vlmutolo 6y ago
  Performance is one reason. UTF-8 is up to twice as compact as UTF-16, which allows much better cache locality. And for Latin-like text, you’re probably frequently hitting the 2x better limit.
- samatman 6y ago
  The article explains this. To summarize: the "codepoint" is a broken metric for what a grapheme "is" in basically any context. Edge cases would be truncating with a guarantee of a valid encoding on the substring? But really you want to truncate at extended grapheme cluster boundaries. Truncating a country flag between two of the regional indicator symbols might not throw errors in your code, but no user is going to consider that a valid string. The same is true of all manner of composed characters, and there are a lot of them. So the only advantage of using the very-often-longer UTF-16 encoding is that it's an attractive nuisance! This makes it easier to write code which will do the wrong thing, constantly, but at a low enough rate that developers will put off fixing it. Unicode is variable width, and what width you need is application-specific. That's the whole point of the article! UTF-8 doesn't try to hide any of this from you.
wodenokoto 6y ago
> The first is useful for basic string operations. Reversing a string is what I would consider basic string operations, but I also expect it not to break emoji and other grapheme clusters. Nothing is easy.
- ChrisSD 6y ago
  Personally I'd consider reversing a string to be a pretty niche use case and, as you say, it's a complex operation. Especially in some languages. Tbh I can't think of a time when I've actually needed to do that.
- giva 6y ago
  I agree. Also, truncating a string won't work. The first 3 characters of "Österreich" are not "Ös" in my opinion.
- darrylb42 6y ago
  Outside of an interview have you ever needed to reverse a string?
  - tgv 6y ago
    Sorting by reversed string is not totally uncommon.
    - neolog 6y ago
      What is a use case for this?
      - carapace 6y ago
        I did this yesterday. find ... | rev | sort | rev > sorted-filelist I had several directories and I wanted to pull out the unique list of certain files (TrueType fonts) across all of them, regardless of which subdirs they were in. (I'm omitting the find CLI args for clarity; the command just finds all the *.ttf (case-insensitive) files in the dirs.) By reversing the lines before sorting them (and un-reversing them) they come out sorted and grouped by filename.
deleted 6y ago
[deleted]
dahfizz 6y ago
> Number of native (preferably UTF-8) code units > The first is useful for basic string operations Can you expand on this? I don't see why knowing the number of code units would be useful except when calculating the total size of the string to allocate memory. Basic string operations, such as converting to uppercase, would operate on codepoints, regardless of how many code units are used to encode that codepoint. Converting 'Á' to 'á', for example, is an operation on one codepoint but multiple code units.
- tedunangst 6y ago
  Neither code points nor units help upper casing ß.
- 7786655 6y ago
  >calculating the total size of the string to allocate memory In some languages this is 90% of everything you do with strings. In other languages it's still 90% of everything done to strings, but done automatically.
- josephg 6y ago
  > Can you expand on this? I don't see why knowing the number of code units would be useful except when calculating the total size of the string to allocate memory I’ve used this for collaborative editing. If you want to send a change saying “insert A at position 10”, the question is: what units should you use for “position 10”? - If you use byte offsets then you have to enforce an encoding on all machines, even when that doesn’t make sense. And you’re allowing the encoding to become corrupted by edits in invalid locations. (Which goes against the principle of making invalid state impossible to represent). - If you use grapheme clusters, the positions aren’t portable between systems or library versions. What today is position 10 in a string might tomorrow be position 9 due to new additions to the Unicode spec. The cleanest answer I’ve found is to count using Unicode codepoints. This approach is encoding-agnostic, portable, simple, well defined and stable across time and between platforms.
masklinn 6y ago
> The first is useful for basic string operations. The only thing it's useful for is sizing up storage. It does nothing for "basic string operations" unless "basic string operations" are solely 7-bit ascii manipulations.
- ChrisSD 6y ago
  It's useful for any and all operations that involve indexing a range. Yes you locate the specific indexes by using extended grapheme clusters, but you use the retrieved byte indexes to actually perform the basic operations. These indexes can also be cached so you don't have to recalculate their byte position every time (so long as the string isn't modified).
  - deleted 6y ago
    [deleted]
  - masklinn 6y ago
    > It's useful for any and all operations that involve indexing a range. That has little to do with the string length, those indices can be completely opaque for all you care. > Yes you locate the specific indexes by using extended grapheme clusters Do you? I'd expect that the indices are mostly located using some sort of pattern matching.
    - ChrisSD 6y ago
      And how does that pattern matching work?
      - masklinn 6y ago
        By traversing the string?
        ChrisSD 6y ago
        Yes. Any pattern matching has to match on extended grapheme clusters or else we're back to the old "treating a string like it's 7-bit ascii" problem.
samatman 6y ago
While I agree with this assessment, it means that these are the basic string operations: * Passing strings around * Reading and writing strings from files / sockets * Concatenation Anything else should reckon with extended grapheme clusters, whether it does so or not. Even proper upcasing is impossible without knowing, for one example, whether or not the string is in Turkish.
- josephg 6y ago
  One difficulty in using extended grapheme clusters is that the codepoints which will be merged into a cluster changes depending on the Unicode version, and sometimes the platform and library. For collaborative editing, the basic unit of measure is codepoints rather than grapheme clusters because you don’t want any ambiguity about where an insert is happening on different peers. Or for a library bump to change how historical data is interpreted.
  - d110af5ccf 6y ago
    > For collaborative editing, the basic unit of measure is codepoints I'd quibble it's not the basic unit of measure so much as how changesets are represented. The user edits based on grapheme clusters. The final edit is then encoded using codepoints, which makes sense because a changeset amounts to a collection of basic string operations (splitting, concatenating, etc). As you note, it would be undesirable for changesets to be aware of higher level string representation details.
    - samatman 6y ago
      For that matter, as long as the format is restricted to one encoding, I don't see why the unit of a changeset can't just be a byte array. I can see why it would happen to be a codepoint, this might be ergonomic for the language, but it seems to me that, like clustering codepoints together in graphemes, clustering bytes into codepoints is something the runtime takes care of, such that a changeset will be a valid example of all three.
      - 6y ago
emergie 6y ago
Contemplate 2 methods of writing a 'dz' digraph dz - \u0064\u007a, 2 basic latin block codepoints DZ - \u0044\u005a Dz - \u0044\u007a ǳ - \u01f3, lowercase, single codepoint Ǳ - \u01f1, uppercase ǲ - \u01f2, TITLECASE! What happens if you try to express dż or dź from polish orthography? You can use dż - \u0064\u017c - d followed by 'LATIN SMALL LETTER Z WITH DOT ABOVE' dż - \u0064\u007a\u0307 - d followed by z, followed by combining diacritical dot above ǳ̇ - \u01f3\u0307 - ǳ with combining diacritical dot above multiplied by uppercase and titlecase forms In polish orthography dz digraph is considered 2 letters, despite being only one sound (głoska). I'm not so sure about macedonian orthography, they might count it as one thing. Medieval ß is a letter/ligature that was created from ſʒ - that is a long s and a tailed z. In other words it is a form of 'sz' digraph. Contemporarily it is used only in german orthography. How long is ß? By some rules uppercasing ß yields SS or SZ. Should uppercasing or titlecasing operations change length of a string?
- saurik 6y ago
  I mean, if you remove the concept of a computer, and ask the question "how many letters are in this word?", you are likely going to end up in some highly-contextual conversations--taking into account all of language, culture, geography, and time--with respect to some of these examples... the concept of "uppercasing or titlecasing" has absolutely no reason to somehow have some logical basis like "the number of characters (which I will note is itself poorly-defined even on a computer) remains constant".

bikeshaving 6y ago

Why do people prefer UTF-8 coordinates? While for storage I think we should use UTF-8, when working with strings live it’s just so much easier to use UTF-16 because it’s predictable: 1 unit for the basic plane and 2 for everything else (the multi-character emoji and modifier stuff aside). I am probably biased because I mostly use and think about DOMStrings which are always UTF-16 but I’m not sure why people who use languages which are more flexible about string representations than JavaScript would not also appreciate this kind of regularity.

pdpi 6y ago
I struggle to see why you'd ever want UTF-16. If you're using a variable length encoding, might as well stick to UTF-8. If you want predictable sizes, there's UTF-32 instead. ~Also, DOM Strings are not UTF-16, they're UCS-16.~ EDIT: UCS-2, not UCS-16. Also, I'm confusing the DOM with EcmaScript, and even that hasn't been true in a while.
- ChrisSD 6y ago
  > Also, DOM Strings are not UTF-16, they're UCS-16. Hm, according to the spec they should be interpreted as UTF-16 but this isn't enforced by the language so it can contain unpaired surrogates: From https://heycam.github.io/webidl/#idl-DOMString https://heycam.github.io/webidl/#idl-DOMString > Such sequences are commonly interpreted as UTF-16 encoded strings [RFC2781] although this is not required... Nothing in this specification requires a DOMString value to be a valid UTF-16 string. From https://262.ecma-international.org/11.0/#sec-ecmascript-language-types-string-type https://262.ecma-international.org/11.0/#sec-ecmascript-lang... > The String type is the set of all ordered sequences of zero or more 16-bit unsigned integer values (“elements”) up to a maximum length of 253 - 1 elements. The String type is generally used to represent textual data in a running ECMAScript program, in which case each element in the String is treated as a UTF-16 code unit value... Operations that do interpret String values treat each element as a single UTF-16 code unit. However, ECMAScript does not restrict the value of or relationships between these code units, so operations that further interpret String contents as sequences of Unicode code points encoded in UTF-16 must account for ill-formed subsequences.
- bikeshaving 6y ago
  UTF-32 is a bit like giving up and saying code units equal code points right? I’m more interested in the comparison between UTF-8 and UTF-16, where UTF-8 requires 1 to 3 bytes just in the BMP, with 3 bytes for CJK characters. I’m saying that as a quick measure of the actual length of a string, UTF-16 is much more predictable and provides a nice intuitive estimation of how long text actually is, as well as providing a fair encoding for most commonly used languages. RE the UTF-16 vs UCS-2 stuff, that’s probably a distinction which has technical meaning but will collapse at some point because no one actually cares, much like the distinction between URI and URL.
  - masklinn 6y ago
    > I’m saying that as a quick measure of the actual length of a string, UTF-16 is much more predictable Meaning the software will deal much less well when it's wrong. > and provides a nice intuitive estimation of how long text actually is Not really due to combining codepoints, which make it not useful. > as well as providing a fair encoding for most commonly used languages. Which we know effectively doesn't matter: it's essentially only a gain for pure CJK text being stored at rest, because otherwise the waste on ASCII will more than compensate for the gain.
- josephg 6y ago
  Yes UTF-16 / UCS-2 is a silly encoding and in retrospect it was a mistake. It matters because for awhile it was believed that 2 bytes would be enough to encode any Unicode character. During this time, lots of important languages appeared which figured a 2 byte fixed size encoding was better than a variable length encoding. So Java, javascript, C# and others all use UCS-2. Now they suffer the downsides of both using a variable length encodings and being memory inefficient. The string.length property in all these languages is almost totally meaningless. UTF-8 is the encoding you should generally always reach for when designing new systems, or when implementing a network protocol. Rust, Go and other newer languages all use UTF-8 internally because it’s better. Well, and in Go’s case because it’s author, Rob Pike also had a hand in inventing UTF-8. Ironically C and UNIX, which (mostly) stubbornly stuck with single byte character encodings generally works better with UTF-8 than a lot of newer languages.
ChrisSD 6y ago
The distinction between the basic plane and everything else is not particularly useful, imho. So I'm not sure I understand what the advantage of UTF-16 is here? It's an arbitrary division either way.
vlmutolo 6y ago
Performance is one reason. UTF-8 is up to twice as compact as UTF-16, which allows much better cache locality. And for Latin-like text, you’re probably frequently hitting the 2x better limit.
samatman 6y ago
The article explains this. To summarize: the "codepoint" is a broken metric for what a grapheme "is" in basically any context. Edge cases would be truncating with a guarantee of a valid encoding on the substring? But really you want to truncate at extended grapheme cluster boundaries. Truncating a country flag between two of the regional indicator symbols might not throw errors in your code, but no user is going to consider that a valid string. The same is true of all manner of composed characters, and there are a lot of them. So the only advantage of using the very-often-longer UTF-16 encoding is that it's an attractive nuisance! This makes it easier to write code which will do the wrong thing, constantly, but at a low enough rate that developers will put off fixing it. Unicode is variable width, and what width you need is application-specific. That's the whole point of the article! UTF-8 doesn't try to hide any of this from you.

wodenokoto 6y ago

> The first is useful for basic string operations. Reversing a string is what I would consider basic string operations, but I also expect it not to break emoji and other grapheme clusters. Nothing is easy.

ChrisSD 6y ago
Personally I'd consider reversing a string to be a pretty niche use case and, as you say, it's a complex operation. Especially in some languages. Tbh I can't think of a time when I've actually needed to do that.
giva 6y ago
I agree. Also, truncating a string won't work. The first 3 characters of "Österreich" are not "Ös" in my opinion.
darrylb42 6y ago
Outside of an interview have you ever needed to reverse a string?
- tgv 6y ago
  Sorting by reversed string is not totally uncommon.
  - neolog 6y ago
    What is a use case for this?
    - carapace 6y ago
      I did this yesterday. find ... | rev | sort | rev > sorted-filelist I had several directories and I wanted to pull out the unique list of certain files (TrueType fonts) across all of them, regardless of which subdirs they were in. (I'm omitting the find CLI args for clarity; the command just finds all the *.ttf (case-insensitive) files in the dirs.) By reversing the lines before sorting them (and un-reversing them) they come out sorted and grouped by filename.

deleted 6y ago

[deleted]

dahfizz 6y ago

> Number of native (preferably UTF-8) code units > The first is useful for basic string operations Can you expand on this? I don't see why knowing the number of code units would be useful except when calculating the total size of the string to allocate memory. Basic string operations, such as converting to uppercase, would operate on codepoints, regardless of how many code units are used to encode that codepoint. Converting 'Á' to 'á', for example, is an operation on one codepoint but multiple code units.

tedunangst 6y ago
Neither code points nor units help upper casing ß.
7786655 6y ago
>calculating the total size of the string to allocate memory In some languages this is 90% of everything you do with strings. In other languages it's still 90% of everything done to strings, but done automatically.
josephg 6y ago
> Can you expand on this? I don't see why knowing the number of code units would be useful except when calculating the total size of the string to allocate memory I’ve used this for collaborative editing. If you want to send a change saying “insert A at position 10”, the question is: what units should you use for “position 10”? - If you use byte offsets then you have to enforce an encoding on all machines, even when that doesn’t make sense. And you’re allowing the encoding to become corrupted by edits in invalid locations. (Which goes against the principle of making invalid state impossible to represent). - If you use grapheme clusters, the positions aren’t portable between systems or library versions. What today is position 10 in a string might tomorrow be position 9 due to new additions to the Unicode spec. The cleanest answer I’ve found is to count using Unicode codepoints. This approach is encoding-agnostic, portable, simple, well defined and stable across time and between platforms.

masklinn 6y ago

> The first is useful for basic string operations. The only thing it's useful for is sizing up storage. It does nothing for "basic string operations" unless "basic string operations" are solely 7-bit ascii manipulations.

ChrisSD 6y ago
It's useful for any and all operations that involve indexing a range. Yes you locate the specific indexes by using extended grapheme clusters, but you use the retrieved byte indexes to actually perform the basic operations. These indexes can also be cached so you don't have to recalculate their byte position every time (so long as the string isn't modified).
- deleted 6y ago
  [deleted]
- masklinn 6y ago
  > It's useful for any and all operations that involve indexing a range. That has little to do with the string length, those indices can be completely opaque for all you care. > Yes you locate the specific indexes by using extended grapheme clusters Do you? I'd expect that the indices are mostly located using some sort of pattern matching.
  - ChrisSD 6y ago
    And how does that pattern matching work?
    - masklinn 6y ago
      By traversing the string?
      - ChrisSD 6y ago
        Yes. Any pattern matching has to match on extended grapheme clusters or else we're back to the old "treating a string like it's 7-bit ascii" problem.

samatman 6y ago

While I agree with this assessment, it means that these are the basic string operations: * Passing strings around * Reading and writing strings from files / sockets * Concatenation Anything else should reckon with extended grapheme clusters, whether it does so or not. Even proper upcasing is impossible without knowing, for one example, whether or not the string is in Turkish.

josephg 6y ago
One difficulty in using extended grapheme clusters is that the codepoints which will be merged into a cluster changes depending on the Unicode version, and sometimes the platform and library. For collaborative editing, the basic unit of measure is codepoints rather than grapheme clusters because you don’t want any ambiguity about where an insert is happening on different peers. Or for a library bump to change how historical data is interpreted.
- d110af5ccf 6y ago
  > For collaborative editing, the basic unit of measure is codepoints I'd quibble it's not the basic unit of measure so much as how changesets are represented. The user edits based on grapheme clusters. The final edit is then encoded using codepoints, which makes sense because a changeset amounts to a collection of basic string operations (splitting, concatenating, etc). As you note, it would be undesirable for changesets to be aware of higher level string representation details.
  - samatman 6y ago
    For that matter, as long as the format is restricted to one encoding, I don't see why the unit of a changeset can't just be a byte array. I can see why it would happen to be a codepoint, this might be ergonomic for the language, but it seems to me that, like clustering codepoints together in graphemes, clustering bytes into codepoints is something the runtime takes care of, such that a changeset will be a valid example of all three.
    - 6y ago

emergie 6y ago

Contemplate 2 methods of writing a 'dz' digraph dz - \u0064\u007a, 2 basic latin block codepoints DZ - \u0044\u005a Dz - \u0044\u007a ǳ - \u01f3, lowercase, single codepoint Ǳ - \u01f1, uppercase ǲ - \u01f2, TITLECASE! What happens if you try to express dż or dź from polish orthography? You can use dż - \u0064\u017c - d followed by 'LATIN SMALL LETTER Z WITH DOT ABOVE' dż - \u0064\u007a\u0307 - d followed by z, followed by combining diacritical dot above ǳ̇ - \u01f3\u0307 - ǳ with combining diacritical dot above multiplied by uppercase and titlecase forms In polish orthography dz digraph is considered 2 letters, despite being only one sound (głoska). I'm not so sure about macedonian orthography, they might count it as one thing. Medieval ß is a letter/ligature that was created from ſʒ - that is a long s and a tailed z. In other words it is a form of 'sz' digraph. Contemporarily it is used only in german orthography. How long is ß? By some rules uppercasing ß yields SS or SZ. Should uppercasing or titlecasing operations change length of a string?

saurik 6y ago
I mean, if you remove the concept of a computer, and ask the question "how many letters are in this word?", you are likely going to end up in some highly-contextual conversations--taking into account all of language, culture, geography, and time--with respect to some of these examples... the concept of "uppercasing or titlecasing" has absolutely no reason to somehow have some logical basis like "the number of characters (which I will note is itself poorly-defined even on a computer) remains constant".