5 ms·

I agree, "length" is an ambiguous function name. It should probably not exist and instead you have functions with units in the name: .sizeBytes, .widthCharacte

I agree, "length" is an ambiguous function name. It should probably not exist and instead you have functions with units in the name: .sizeBytes, .widthCharacters, .widthResAdjPixels, and so on. Back when the world was ASCII you could get away with just .length because the numbers would always be the same, but with Unicode and all of the other complications of the modern world it isn't sufficient.

sroussey 3y ago
Seeing a .length on something makes me think it’s an iterable without looking more deeply. In JS, for(i in emoji) will iterate twice, but for(i of emoji) will iterate once. ;)
- josephg 3y ago
  Yeah; I've recently noticed that almost every time I use string.length in javascript, its wrong and going to break something as soon as emoji appears. In my code, I always want to deal with either the number of codepoints or the number of UTF8 bytes. String.length gives you neither, but unfortunately it looks correct until you test with non-ASCII strings.
  - sroussey 3y ago
    At least the “of” loop returns what is selectable via a cursor in the browser. I think…
    - josephg 3y ago
      Yeah; its really confusing but javascript - for legacy reasons - treats strings as "arrays of UCS2 items". But javascript also implements iterator on strings which iterate through strings in unicode codepoints. Thats why "of" loops work differently from "in" loops. (for-of in javascript uses Symbol.iterator). That also means you can pull a string apart into an array of unicode codepoints using [...somestring].
ant6n 3y ago
length is not ambiguous at all. Its the number of elements in the array. A string in python3 is an array of unicode code points, so the length of a string is the number of unicode code points. If you want the number of bytes, you need to encode the string in a unicode format (utf8, utf16 or utf32) to get a bytes object, which is an array of bytes. Then you can get the length of that. Remember, one of the big accomplishments (breaking changes) of python 3 is that all strings are Unicode, not byte arrays. If you want to view a dtring as bytes, you need to convert the string to bytes. But note the number of bytes depends on the ancoding u use (utf8, …).
- sirsinsalot 3y ago
  Exactly this. People conflate unicode with encoding quite a bit. I think it was plan9 and early Go that used "runes" as a unit, where one or more runes formed a character and an array of runes could be encoded into bytes using a given encoding. The in memory size of a rune was just an implementation detail, and while it could be important for the programmer that the size of a rune was 2 bytes, this didn't mean the length of an array of 2 runes was 4. I always liked the rune unit, and while my memory is hazy I think it was just code points. I think part of the issue is programmers and apis mixing bit units for in memory representation of a conceptual value mapping (unicode), conceptual characters, stored size when encoded and so on ... without firming up those abstractions with interfaces. It gets lossy.
  - eesmith 3y ago
    FWIW, CPython uses one of several Unicode string implementation representations, depending on the code points involved: >>> import sys >>> s = 'A' * 1000 >>> len(s) 1000 >>> sys.getsizeof(s) 1049 >>> s = '\N{SNOWMAN WITHOUT SNOW}' * 1000 >>> len(s) 1000 >>> sys.getsizeof(s) 2074 >>> s = '\N{MUSICAL SYMBOL G CLEF}' * 1000 >>> len(s) 1000 >>> sys.getsizeof(s) 4076 See https://peps.python.org/pep-0393/ https://peps.python.org/pep-0393/ . Mentioned in the linked-to article with "CPython since 3.3 makes the same idea three-level with code point semantics".
- coldtea 3y ago
  >length is not ambiguous at all. Its the number of elements in the array That's because you defined it first as "the number of elements in the array". It is ambiguous however because that's not how people understand it when it comes to strings, and there are several counter-intuitive ways they expect it to behave. Not to mention there might not be any "array". A string (whatever the encoding / representation) is a chunk of memory, not an array. That you can often use a method to traverse it doesn't mean it's in an array.

sroussey 3y ago

Seeing a .length on something makes me think it’s an iterable without looking more deeply. In JS, for(i in emoji) will iterate twice, but for(i of emoji) will iterate once. ;)

josephg 3y ago
Yeah; I've recently noticed that almost every time I use string.length in javascript, its wrong and going to break something as soon as emoji appears. In my code, I always want to deal with either the number of codepoints or the number of UTF8 bytes. String.length gives you neither, but unfortunately it looks correct until you test with non-ASCII strings.
- sroussey 3y ago
  At least the “of” loop returns what is selectable via a cursor in the browser. I think…
  - josephg 3y ago
    Yeah; its really confusing but javascript - for legacy reasons - treats strings as "arrays of UCS2 items". But javascript also implements iterator on strings which iterate through strings in unicode codepoints. Thats why "of" loops work differently from "in" loops. (for-of in javascript uses Symbol.iterator). That also means you can pull a string apart into an array of unicode codepoints using [...somestring].

ant6n 3y ago

length is not ambiguous at all. Its the number of elements in the array. A string in python3 is an array of unicode code points, so the length of a string is the number of unicode code points. If you want the number of bytes, you need to encode the string in a unicode format (utf8, utf16 or utf32) to get a bytes object, which is an array of bytes. Then you can get the length of that. Remember, one of the big accomplishments (breaking changes) of python 3 is that all strings are Unicode, not byte arrays. If you want to view a dtring as bytes, you need to convert the string to bytes. But note the number of bytes depends on the ancoding u use (utf8, …).

sirsinsalot 3y ago
Exactly this. People conflate unicode with encoding quite a bit. I think it was plan9 and early Go that used "runes" as a unit, where one or more runes formed a character and an array of runes could be encoded into bytes using a given encoding. The in memory size of a rune was just an implementation detail, and while it could be important for the programmer that the size of a rune was 2 bytes, this didn't mean the length of an array of 2 runes was 4. I always liked the rune unit, and while my memory is hazy I think it was just code points. I think part of the issue is programmers and apis mixing bit units for in memory representation of a conceptual value mapping (unicode), conceptual characters, stored size when encoded and so on ... without firming up those abstractions with interfaces. It gets lossy.
- eesmith 3y ago
  FWIW, CPython uses one of several Unicode string implementation representations, depending on the code points involved: >>> import sys >>> s = 'A' * 1000 >>> len(s) 1000 >>> sys.getsizeof(s) 1049 >>> s = '\N{SNOWMAN WITHOUT SNOW}' * 1000 >>> len(s) 1000 >>> sys.getsizeof(s) 2074 >>> s = '\N{MUSICAL SYMBOL G CLEF}' * 1000 >>> len(s) 1000 >>> sys.getsizeof(s) 4076 See https://peps.python.org/pep-0393/ https://peps.python.org/pep-0393/ . Mentioned in the linked-to article with "CPython since 3.3 makes the same idea three-level with code point semantics".
coldtea 3y ago
>length is not ambiguous at all. Its the number of elements in the array That's because you defined it first as "the number of elements in the array". It is ambiguous however because that's not how people understand it when it comes to strings, and there are several counter-intuitive ways they expect it to behave. Not to mention there might not be any "array". A string (whatever the encoding / representation) is a chunk of memory, not an array. That you can often use a method to traverse it doesn't mean it's in an array.