11 ms·

> Python 3’s approach is unambiguously the worst one, though. Did I miss the part where he explains this take? It's made up of 5 valid unicode code units. For

> Python 3’s approach is unambiguously the worst one, though. Did I miss the part where he explains this take? It's made up of 5 valid unicode code units. For a language where you're not supposed to need to know the byte size semantics, the correct length should be 5. What am I missing? The close second being 17, because length in bytes. Is another fine way to represent this data, e.g. what a successful write of some sort would look like. Network or file. I guess I'm basing this all on the idea that it's almost always a mistake to confuse how a program manages some data, vs how a drawing lib might. Your language shouldn't concern it self with how many glyphs it needs to draw... until you actually try to draw them.

benatkin 3y ago
I think he meant to bring out defensiveness with that quip. He never says that it's a big deal, just that it's the worst way to get the length of a string containing emoji, presumably of the mainstream languages.
- tehsauce 3y ago
  Why is it the worst way though?
  - deleted 3y ago
    [deleted]
  - imron 3y ago
    > Note about Python 3 added on 2019-09-09: Originally this article claimed that Python 3 guaranteed UTF-32 validity. This was in error. Python 3 guarantees that the units of the string stay within the Unicode code point range but does not guarantee the absence of surrogates. It not only allows unpaired surrogates, which might be explained by wishing to be compatible with the value space of potentially-invalid UTF-16, but Python 3 allows materializing even surrogate pairs, which is a truly bizarre design. The previous conclusions stand with the added conclusion that Python 3 is even more messed up than I thought!
  - markmark 3y ago
    Perhaps the part about it needing lookups of the unicode database and being dependent on the version of the database used?
    - lmm 3y ago
      That's not true though. It just counts the number of code units, that's not version dependent. It's certainly no worse than counting the number of UTF-16 points (I'd argue it's better since it's less arbitrary - whether something is a unicode scalar is a design decision, whether something is in the BMP or not is mostly an accident of implementation).
  - masklinn 3y ago
    Because it’s never actually useful. You can’t use that information to know how much actual space it takes (in storage) as nobody sane stores UTF-32, you can’t use it to know much much logical space it takes (aka the user’s interpretation), you can’t use it to know how much visual space it takes (not that you can ever get that), and you can’t use it to segment or process the text. A length in codepoints gives you nothing that’s really actionable, at least not that you’d need outside of a context where you could easily obtain it otherwise.
    - L3viathan 3y ago
      It is useful: When iterating over a string in Python (which I hope you agree _is_ useful?), you get that many parts.
      - chrismorgan 3y ago
        > When iterating over a string in Python (which I hope you agree _is_ useful?) Not often. There’s almost nothing useful you can correctly do with a sequence of code points.
      - masklinn 3y ago
        > It is useful: When iterating over a string in Python (which I hope you agree _is_ useful?), you get that many parts. That’s… not useful? I can’t say I remember ever caring knowing how many items I would be getting during an iteration[0]. If I want to set an iteration limit I can just… do that, using `islice` or some such. [0] in python anyway, in lower level language there can be a utility in order to pre-allocate an output collection
      - deleted 3y ago
        [deleted]
    - boxed 3y ago
      But 100% of all those complaints also apply to JS, except that UTF-16 is just much more stupid than UTF-32?
    - globular-toast 3y ago
arcticbull 3y ago
It is wrong that "{emoji}".length == 7 -- but it's wrong because there's no such thing as the 'length' of a string out of context. A string should be viewed as an opaque data type with views into it depending on what you're trying to do. You can have its length in the context of storage/retrieval/transmission (UTF-8 byte count), its length in the context of parsing (code points), its length in the context of editing (grapheme clusters) or length in the context of display (a bounding box in points when used in conjunction with a specific font and paragraph style attributes). Claiming to provide an out-of-context length is strictly wrong because there's no such thing. This is where people get confused. The attribute shouldn't be 'length' it should be something like 'countOfCodePoints' or exposed via a `CodePoints` type view. It's particularly bad because so often (esp. for western programmers) 'countOfCodePoints' == 'countOfBytesInUTF8' == 'countOfGraphemeClusters' == """length""" so it's hella easy to accidentally write buggy software. Especially for people who don't know the above about unicode, which let's face it, most people don't. Not until they have to explain to their designer why they can't limit a label to '10 characters.' ("What do you mean there's no such thing as a character, and what am I trying to do?"). This is basically the tl;dr of the article but it's also my personal opinion. All of this isn't about 'wrong' so much as 'imprecise and overloaded terminology making it easy to write buggy software through poor abstractions.' If python explained which length you were getting, then this article wouldn't exist.
- jandrese 3y ago
  I agree, "length" is an ambiguous function name. It should probably not exist and instead you have functions with units in the name: .sizeBytes, .widthCharacters, .widthResAdjPixels, and so on. Back when the world was ASCII you could get away with just .length because the numbers would always be the same, but with Unicode and all of the other complications of the modern world it isn't sufficient.
  - sroussey 3y ago
    Seeing a .length on something makes me think it’s an iterable without looking more deeply. In JS, for(i in emoji) will iterate twice, but for(i of emoji) will iterate once. ;)
BeefWellington 3y ago
5 makes perfect sense to me; the author's complaints seem kinda silly. An area this makes sense is, what do you expect to get if you do something like: emoji = " " print(emoji[:3]) Should this throw an error because there's only one displayed "character"? Should it return only a partial codepoint by returning only the byte data for the first 3 bytes? Modern strings are complex objects that have evolved a bit past char[] or byte[].
- masklinn 3y ago
  > Should this throw an error because there's only one displayed "character"? Why should it not? You’re literally breaking the content. Though in reality, indexing strings is a broken operation. That you’re using it at all is the core issue. > Modern strings are complex objects that have evolved a bit past char[] or byte[]. And yet that’s exactly what you’re advocating, just with 21 bit chars.
  - BeefWellington 3y ago
    > Why should it not? You’re literally breaking the content. Strings are just an array of unicode codepoints rather than "characters", so all I'm doing is asking for the first three of those codepoints. > Though in reality, indexing strings is a broken operation. That you’re using it at all is the core issue. Substring is a broken operation? What's the justification for that idea?
    - masklinn 3y ago
      > Strings are just an array of unicode codepoints rather than "characters", so all I'm doing is asking for the first three of those codepoints. "Ice trays are just a pile of molecules rather than "cubes", so all I'm doing is separating those molecules", he states as he activates the igniter. > Substring is a broken operation? What's the justification for that idea? You take a thing and you mangle beyond recognition without regards for its purpose or meaning. That's like considering the jaws of life a normal part of opening a door to take a piss at work.
  - 3y ago
emodendroket 3y ago
> I'm basing this all on the idea that it's almost always a mistake to confuse how a program manages some data, vs how a drawing lib might. Your language shouldn't concern it self with how many glyphs it needs to draw... until you actually try to draw them. Well, why not? There are a lot of things that people would want to call string.length for — drawing little equals signs under text in a terminal, for a frivolous example — where that’s the whole reason they’re making the call. Off the top of my head I’m not really sure how you solve that with variable-width characters if there’s no way to separate out or count them.
- coldtea 3y ago
  >Well, why not? Separation of concerns?
  - emodendroket 3y ago
    I’m not sure what separation of concerns is served by not providing a method to show how many glyphs are in a given string.
    - coldtea 3y ago
      Who said there shouldn't be one? The point is there should be more than one, and that not all are a language/string library-level concern. The context is "I guess I'm basing this all on the idea that it's almost always a mistake to confuse how a program manages some data, vs how a drawing lib might. Your language shouldn't concern it self with how many glyphs it needs to draw... until you actually try to draw them." This means that shouldn't be some generic "length" method, but appropriate separate-concerns methods (plural), some of which (e.g. regarding character width in pixels when rendered) even belong to a drawing lib and not the language at all. The parent's point is that length (bytes), characters (count), and glyphs (size, shape) are different concerns. The latter would concern a drawing lib or a renderer, but not be a core string method (which should concern itself with the abstract notion of characters and the concrete notion of bytes).
chrismorgan 3y ago
> For a language where you're not supposed to need to know the byte size semantics, the correct length should be 5. What am I missing? In the words of the article: “The choice of UTF-32 (or Python 3-style code point sequences) arises from wanting the wrong thing.” “Not needing to know the byte size semantics” seems reasonable, but it simply isn’t a useful goal. The things it makes easier or faster (knowing how many code points there are, and O(1) indexing by code point) are things you shouldn’t be doing—and when you have to interact with the rest of the world, you now have a more expensive encoding step that is always needed, rather than just sometimes if you’d chosen UTF-8 or even UTF-16.
- pierrebai 3y ago
  ... but maybe it simplifies and speed-up the internal processing? I haven't looked at Python 3 C implementation of strings, but that is a guess. Also, IIRC, Python 3 has the ability to keep different internal representations of strings and uses the most compact one. If all character are 7-bit ASCII, it uses bytes representation. That's what I remember from Python dev discussions long ago. But the overall tone of the article is still bashful. Caring about the internal representation of strings and bashing UTF-32 feels lame and angry. (Especially if I'm right about the multi-rep nature of Python 3: their choice is good for most text, and they could add a UTF-8 internal rep in the future, although that would probably break enough code that expect the UTF-32 value for len() that it is not worth it.)
  - chrismorgan 3y ago
    No. Python internally would be made much faster by working on pure UTF-8. Absolutely nothing internal to the language uses the operations that code point semantics speeds up. Since you mention the varying internal representation of strings: that’s PEP 393 <https://peps.python.org/pep-0393/ https://peps.python.org/pep-0393/>, which landed in CPython 3.3, and it generally made things slower by introducing a lot of branching and reallocating and such, though it does speed up some cases due to having to touch less memory, and some methods due to being able to quickly rule out possibilities (e.g. str.isascii can immediately return False for a canonical UCS-2 or UCS-4 string, since if they were ASCII they’d have been of the Latin-1 kind). PEP 393 was done because people were complaining about how much memory their UCS-4 encoding had been using. Note also how PEP 393 retains code point semantics: Latin-1 (Unicode values 0–255), UCS-2 or UCS-4; all fixed-width encodings of code point sequences. PEP 393 does also allow a string to cache UTF-8 representation (see PyCompactUnicodeObject.{utf8, utf8_length}), choosing “UTF-8 as the recommended way of exposing strings to C code”, but I gather this isn’t used very much. (Related: PyPy 7.1 shifted to using UTF-8 exclusively internally, and according to https://www.pypy.org/posts/2019/03/pypy-v71-released-now-uses-utf-8-451324088028792912.html https://www.pypy.org/posts/2019/03/pypy-v71-released-now-use... got a “nice speed bump” out of it.)
globular-toast 3y ago
> The close second being 17, because length in bytes. Is another fine way to represent this data, e.g. what a successful write of some sort would look like. Network or file. Almost. 17 is the number of bytes it occupies in memory. But you don't generally dump memory directly to disk or network. It happens to make sense (and it's convenient) for utf8 strings. But it's better to be explicit about that. Python is better. If you care about bytes, say you care about bytes: len(bytes(" ", "utf8")) == 17 len(bytes(" ", "utf-16-be")) == 14 len(bytes(" ", "utf-32-be")) == 20
lowdownfork 3y ago
As far as I can tell, you're only missing two things: 1. It's five "Unicode scalars," that's the name for the top-level logical unit. The term "code points" technically refers to a lower-level concept, one that varies across encodings, just not as much as the number of bytes. I didn't know that, and it's the helpful thing I learned from this article. UPDATE: And it's also not true, sorry. "code units" are the lower-level concept from the article, "code points" are a more expansive category at the same level: https://www.unicode.org/versions/Unicode10.0.0/ch03.pdf#G7404 https://www.unicode.org/versions/Unicode10.0.0/ch03.pdf#G740... 2. The author takes it as an unstated assumption that top-level logical structure is useless because any specific usage either ignores all structure or has a point at which low-level structure comes into play. (That assumption is false: Top-level structure is useful for keeping track of what you are doing and as a sort of "common currency" for translating between different low level representations. For example, see the very first table in the article.)

benatkin 3y ago

I think he meant to bring out defensiveness with that quip. He never says that it's a big deal, just that it's the worst way to get the length of a string containing emoji, presumably of the mainstream languages.

tehsauce 3y ago
Why is it the worst way though?
- deleted 3y ago
  [deleted]
- imron 3y ago
  > Note about Python 3 added on 2019-09-09: Originally this article claimed that Python 3 guaranteed UTF-32 validity. This was in error. Python 3 guarantees that the units of the string stay within the Unicode code point range but does not guarantee the absence of surrogates. It not only allows unpaired surrogates, which might be explained by wishing to be compatible with the value space of potentially-invalid UTF-16, but Python 3 allows materializing even surrogate pairs, which is a truly bizarre design. The previous conclusions stand with the added conclusion that Python 3 is even more messed up than I thought!
- markmark 3y ago
  Perhaps the part about it needing lookups of the unicode database and being dependent on the version of the database used?
  - lmm 3y ago
    That's not true though. It just counts the number of code units, that's not version dependent. It's certainly no worse than counting the number of UTF-16 points (I'd argue it's better since it's less arbitrary - whether something is a unicode scalar is a design decision, whether something is in the BMP or not is mostly an accident of implementation).
- masklinn 3y ago
  Because it’s never actually useful. You can’t use that information to know how much actual space it takes (in storage) as nobody sane stores UTF-32, you can’t use it to know much much logical space it takes (aka the user’s interpretation), you can’t use it to know how much visual space it takes (not that you can ever get that), and you can’t use it to segment or process the text. A length in codepoints gives you nothing that’s really actionable, at least not that you’d need outside of a context where you could easily obtain it otherwise.
  - L3viathan 3y ago
    It is useful: When iterating over a string in Python (which I hope you agree _is_ useful?), you get that many parts.
    - chrismorgan 3y ago
      > When iterating over a string in Python (which I hope you agree _is_ useful?) Not often. There’s almost nothing useful you can correctly do with a sequence of code points.
    - masklinn 3y ago
      > It is useful: When iterating over a string in Python (which I hope you agree _is_ useful?), you get that many parts. That’s… not useful? I can’t say I remember ever caring knowing how many items I would be getting during an iteration[0]. If I want to set an iteration limit I can just… do that, using `islice` or some such. [0] in python anyway, in lower level language there can be a utility in order to pre-allocate an output collection
    - deleted 3y ago
      [deleted]
  - boxed 3y ago
    But 100% of all those complaints also apply to JS, except that UTF-16 is just much more stupid than UTF-32?
  - globular-toast 3y ago

arcticbull 3y ago

It is wrong that "{emoji}".length == 7 -- but it's wrong because there's no such thing as the 'length' of a string out of context. A string should be viewed as an opaque data type with views into it depending on what you're trying to do. You can have its length in the context of storage/retrieval/transmission (UTF-8 byte count), its length in the context of parsing (code points), its length in the context of editing (grapheme clusters) or length in the context of display (a bounding box in points when used in conjunction with a specific font and paragraph style attributes). Claiming to provide an out-of-context length is strictly wrong because there's no such thing. This is where people get confused. The attribute shouldn't be 'length' it should be something like 'countOfCodePoints' or exposed via a `CodePoints` type view. It's particularly bad because so often (esp. for western programmers) 'countOfCodePoints' == 'countOfBytesInUTF8' == 'countOfGraphemeClusters' == """length""" so it's hella easy to accidentally write buggy software. Especially for people who don't know the above about unicode, which let's face it, most people don't. Not until they have to explain to their designer why they can't limit a label to '10 characters.' ("What do you mean there's no such thing as a character, and what am I trying to do?"). This is basically the tl;dr of the article but it's also my personal opinion. All of this isn't about 'wrong' so much as 'imprecise and overloaded terminology making it easy to write buggy software through poor abstractions.' If python explained which length you were getting, then this article wouldn't exist.

jandrese 3y ago
I agree, "length" is an ambiguous function name. It should probably not exist and instead you have functions with units in the name: .sizeBytes, .widthCharacters, .widthResAdjPixels, and so on. Back when the world was ASCII you could get away with just .length because the numbers would always be the same, but with Unicode and all of the other complications of the modern world it isn't sufficient.
- sroussey 3y ago
  Seeing a .length on something makes me think it’s an iterable without looking more deeply. In JS, for(i in emoji) will iterate twice, but for(i of emoji) will iterate once. ;)

BeefWellington 3y ago

5 makes perfect sense to me; the author's complaints seem kinda silly. An area this makes sense is, what do you expect to get if you do something like: emoji = " " print(emoji[:3]) Should this throw an error because there's only one displayed "character"? Should it return only a partial codepoint by returning only the byte data for the first 3 bytes? Modern strings are complex objects that have evolved a bit past char[] or byte[].

masklinn 3y ago
> Should this throw an error because there's only one displayed "character"? Why should it not? You’re literally breaking the content. Though in reality, indexing strings is a broken operation. That you’re using it at all is the core issue. > Modern strings are complex objects that have evolved a bit past char[] or byte[]. And yet that’s exactly what you’re advocating, just with 21 bit chars.
- BeefWellington 3y ago
  > Why should it not? You’re literally breaking the content. Strings are just an array of unicode codepoints rather than "characters", so all I'm doing is asking for the first three of those codepoints. > Though in reality, indexing strings is a broken operation. That you’re using it at all is the core issue. Substring is a broken operation? What's the justification for that idea?
  - masklinn 3y ago
    > Strings are just an array of unicode codepoints rather than "characters", so all I'm doing is asking for the first three of those codepoints. "Ice trays are just a pile of molecules rather than "cubes", so all I'm doing is separating those molecules", he states as he activates the igniter. > Substring is a broken operation? What's the justification for that idea? You take a thing and you mangle beyond recognition without regards for its purpose or meaning. That's like considering the jaws of life a normal part of opening a door to take a piss at work.
- 3y ago

emodendroket 3y ago

> I'm basing this all on the idea that it's almost always a mistake to confuse how a program manages some data, vs how a drawing lib might. Your language shouldn't concern it self with how many glyphs it needs to draw... until you actually try to draw them. Well, why not? There are a lot of things that people would want to call string.length for — drawing little equals signs under text in a terminal, for a frivolous example — where that’s the whole reason they’re making the call. Off the top of my head I’m not really sure how you solve that with variable-width characters if there’s no way to separate out or count them.

coldtea 3y ago
>Well, why not? Separation of concerns?
- emodendroket 3y ago
  I’m not sure what separation of concerns is served by not providing a method to show how many glyphs are in a given string.
  - coldtea 3y ago
    Who said there shouldn't be one? The point is there should be more than one, and that not all are a language/string library-level concern. The context is "I guess I'm basing this all on the idea that it's almost always a mistake to confuse how a program manages some data, vs how a drawing lib might. Your language shouldn't concern it self with how many glyphs it needs to draw... until you actually try to draw them." This means that shouldn't be some generic "length" method, but appropriate separate-concerns methods (plural), some of which (e.g. regarding character width in pixels when rendered) even belong to a drawing lib and not the language at all. The parent's point is that length (bytes), characters (count), and glyphs (size, shape) are different concerns. The latter would concern a drawing lib or a renderer, but not be a core string method (which should concern itself with the abstract notion of characters and the concrete notion of bytes).

chrismorgan 3y ago

> For a language where you're not supposed to need to know the byte size semantics, the correct length should be 5. What am I missing? In the words of the article: “The choice of UTF-32 (or Python 3-style code point sequences) arises from wanting the wrong thing.” “Not needing to know the byte size semantics” seems reasonable, but it simply isn’t a useful goal. The things it makes easier or faster (knowing how many code points there are, and O(1) indexing by code point) are things you shouldn’t be doing—and when you have to interact with the rest of the world, you now have a more expensive encoding step that is always needed, rather than just sometimes if you’d chosen UTF-8 or even UTF-16.

pierrebai 3y ago
... but maybe it simplifies and speed-up the internal processing? I haven't looked at Python 3 C implementation of strings, but that is a guess. Also, IIRC, Python 3 has the ability to keep different internal representations of strings and uses the most compact one. If all character are 7-bit ASCII, it uses bytes representation. That's what I remember from Python dev discussions long ago. But the overall tone of the article is still bashful. Caring about the internal representation of strings and bashing UTF-32 feels lame and angry. (Especially if I'm right about the multi-rep nature of Python 3: their choice is good for most text, and they could add a UTF-8 internal rep in the future, although that would probably break enough code that expect the UTF-32 value for len() that it is not worth it.)
- chrismorgan 3y ago
  No. Python internally would be made much faster by working on pure UTF-8. Absolutely nothing internal to the language uses the operations that code point semantics speeds up. Since you mention the varying internal representation of strings: that’s PEP 393 <https://peps.python.org/pep-0393/ https://peps.python.org/pep-0393/>, which landed in CPython 3.3, and it generally made things slower by introducing a lot of branching and reallocating and such, though it does speed up some cases due to having to touch less memory, and some methods due to being able to quickly rule out possibilities (e.g. str.isascii can immediately return False for a canonical UCS-2 or UCS-4 string, since if they were ASCII they’d have been of the Latin-1 kind). PEP 393 was done because people were complaining about how much memory their UCS-4 encoding had been using. Note also how PEP 393 retains code point semantics: Latin-1 (Unicode values 0–255), UCS-2 or UCS-4; all fixed-width encodings of code point sequences. PEP 393 does also allow a string to cache UTF-8 representation (see PyCompactUnicodeObject.{utf8, utf8_length}), choosing “UTF-8 as the recommended way of exposing strings to C code”, but I gather this isn’t used very much. (Related: PyPy 7.1 shifted to using UTF-8 exclusively internally, and according to https://www.pypy.org/posts/2019/03/pypy-v71-released-now-uses-utf-8-451324088028792912.html https://www.pypy.org/posts/2019/03/pypy-v71-released-now-use... got a “nice speed bump” out of it.)

globular-toast 3y ago

> The close second being 17, because length in bytes. Is another fine way to represent this data, e.g. what a successful write of some sort would look like. Network or file. Almost. 17 is the number of bytes it occupies in memory. But you don't generally dump memory directly to disk or network. It happens to make sense (and it's convenient) for utf8 strings. But it's better to be explicit about that. Python is better. If you care about bytes, say you care about bytes: len(bytes(" ", "utf8")) == 17 len(bytes(" ", "utf-16-be")) == 14 len(bytes(" ", "utf-32-be")) == 20

lowdownfork 3y ago

As far as I can tell, you're only missing two things: 1. It's five "Unicode scalars," that's the name for the top-level logical unit. The term "code points" technically refers to a lower-level concept, one that varies across encodings, just not as much as the number of bytes. I didn't know that, and it's the helpful thing I learned from this article. UPDATE: And it's also not true, sorry. "code units" are the lower-level concept from the article, "code points" are a more expansive category at the same level: https://www.unicode.org/versions/Unicode10.0.0/ch03.pdf#G7404 https://www.unicode.org/versions/Unicode10.0.0/ch03.pdf#G740... 2. The author takes it as an unstated assumption that top-level logical structure is useless because any specific usage either ignores all structure or has a point at which low-level structure comes into play. (That assumption is false: Top-level structure is useful for keeping track of what you are doing and as a sort of "common currency" for translating between different low level representations. For example, see the very first table in the article.)