5 ms·

After having read (quite interesting) the article, I still don't quite get the subtitle: > 'But It’s Better that "[emoji]".len() == 17 and Rather Useless that

• dahfizz 6y ago
  I agree. The author talks a little bit about which UTF encoding makes sense in which situation, but they never make an argument about which result from len is correct. My two cents is that string length should always be the number of Unicode codepoints in the string, regardless of encoding. If you want the byte length, I'm sure there is a sizeof equivalent for your language. When we call len() on an array, we want the number of objects in the array. When we iterate over an array, we want to deal with one object from the array at a time. We don't care how big an object is. A large object shouldn't count for more than 1 when calculating the array length. Similarly, a unicode codepoint is the fundamental object in a string. The byte-size of a codepoint does not affect the length of the string. It makes no sense to iterate over each byte in a unicode string, because a byte on its own is completely meaningless. len() should be the number of objects we can iterate over, just like in arrays.
  • MayeulC 6y ago
    But what about combining characters? https://en.wikipedia.org/wiki/Combining_Diacritical_Marks https://en.wikipedia.org/wiki/Combining_Diacritical_Marks Should the letter plus a combining character count as one (I think so), or two characters? Should you normalize before counting length? And so on.
    • dahfizz 6y ago
      Combining characters are their own unicode codepoint, so they count towards length. The beauty of this approach is that its simple and objective. If you had a list of 5 Dom Element objects, and one Dom Attr object, the length of that list is 6. Its nonsensical to say "The Attr object modifies an Element object, so its not really in the list".
      • shawnz 6y ago
        Going by bytes is also simple and objective. And also totally arbitrary, just like going by codepoints. Which is the most useful for dealing with strings in practice though? Are either interpretations useful at all?
        • dahfizz 6y ago
          Going byte by byte is useless. You can't do anything with a single byte of a unicode codepoint (unless, by luck, the codepoint is encoded in a single byte). Codepoint is the smallest useful unit of a unicode string. It is a character, and you can do all the character things with it. If you wanted to implement a toUpper() function for example, you would want to iterate over all the codepoints.
          • masklinn 6y ago
            > If you wanted to implement a toUpper() function for example, you would want to iterate over all the codepoints. Nope. In order to deal with special casings you will have to span multiple codepoints, at which point it's no more work with whatever the code units are.
      • d110af5ccf 6y ago
        > Combining characters are their own unicode codepoint, so they count towards length. This is incredibly arbitrary - it depends entirely on what "length" means for a particular usecase. From the user's perspective there might only be a single character on the screen. Any nontrivial string operation must be based around grapheme clusters, otherwise it is fundamentally broken. Codepoints are a useful encoding agnostic way to handle basic (split & concatenate) operations, but the method by which those offsets are determined needs to be grapheme cluster aware. Raw byte offsets are encoding specific and only really useful for allocating the underlying storage.
    • rurban 6y ago
      You don't count the length. Length specifies the size of the internal encoding. You count the width. And there are Unicode rules how you count the width. Which do change every year.
  • ChrisSD 6y ago
    What is a string an array of? If you ask the user, it's an array of characters (aka extended grapheme clusters in Unicode speak). If you ask the machine it's an array of integers (how many bytes make up an integer depends on the encoding used). Nothing really considers them an array of code points. Code points are only useful as intermediary values when converting between encodings or interpreting an encoded string as grapheme clusters.
    • deleted 6y ago
      [deleted]
    • dahfizz 6y ago
      > If you ask the machine it's an array of integers Not sure what you mean by this. A string is an array of bytes, in the way that literally every array is an array of bytes, but its not "implemented" with integers. Its a UTC-encoded array of bytes. And what is the information that is encoded in those bytes? Codepoints. That's what UTF does, it lets us store unicode codepoints as bytes. There is a reasonable argument that the machine, or at least the developer, considers a string as an array of codepoints.
      • ChrisSD 6y ago
        UTF-8 is an array of bytes (8 bit integers). UTF-16 is an array of 16 bit integers. UTF-32 is an array of 32 bit integers. The machine doesn't know anything about code points. If you want to index into the array you'll need to know the integer offset.
        • dahfizz 6y ago
          > The machine doesn't know anything about code points. If you want to index into the array you'll need to know the integer offset. The machine doesn't know anything about Colors either. But if I defined a Color object, I would be able to put Color objects into an array and count how many Color objects I had. You're being needlessly reductive. > UTF-8 is an array of bytes (8 bit integers) UTF-8 encodes a codepoint with 1-4 single-byte code units. The reason UTF-8 exists is to provide a way for machines and developers to interact with unicode codepoints. Is a huffman code an array of bits? Or is it a list of symbols encoded using bits?