6 ms·
It is wrong that "{emoji}".length == 7 -- but it's wrong because there's no such thing as the 'length' of a string out of context. A string should be viewed as
by arcticbull 3y ago
It is wrong that "{emoji}".length == 7 -- but it's wrong because there's no such thing as the 'length' of a string out of context.
A string should be viewed as an opaque data type with views into it depending on what you're trying to do. You can have its length in the context of storage/retrieval/transmission (UTF-8 byte count), its length in the context of parsing (code points), its length in the context of editing (grapheme clusters) or length in the context of display (a bounding box in points when used in conjunction with a specific font and paragraph style attributes).
Claiming to provide an out-of-context length is strictly wrong because there's no such thing. This is where people get confused.
The attribute shouldn't be 'length' it should be something like 'countOfCodePoints' or exposed via a `CodePoints` type view.
It's particularly bad because so often (esp. for western programmers) 'countOfCodePoints' == 'countOfBytesInUTF8' == 'countOfGraphemeClusters' == """length""" so it's hella easy to accidentally write buggy software. Especially for people who don't know the above about unicode, which let's face it, most people don't. Not until they have to explain to their designer why they can't limit a label to '10 characters.' ("What do you mean there's no such thing as a character, and what am I trying to do?").
This is basically the tl;dr of the article but it's also my personal opinion.
All of this isn't about 'wrong' so much as 'imprecise and overloaded terminology making it easy to write buggy software through poor abstractions.'
If python explained which length you were getting, then this article wouldn't exist.
- jandrese 3y agoI agree, "length" is an ambiguous function name. It should probably not exist and instead you have functions with units in the name: .sizeBytes, .widthCharacters, .widthResAdjPixels, and so on. Back when the world was ASCII you could get away with just .length because the numbers would always be the same, but with Unicode and all of the other complications of the modern world it isn't sufficient.
- sroussey 3y agoSeeing a .length on something makes me think it’s an iterable without looking more deeply. In JS, for(i in emoji) will iterate twice, but for(i of emoji) will iterate once. ;)
- josephg 3y agoYeah; I've recently noticed that almost every time I use string.length in javascript, its wrong and going to break something as soon as emoji appears. In my code, I always want to deal with either the number of codepoints or the number of UTF8 bytes. String.length gives you neither, but unfortunately it looks correct until you test with non-ASCII strings.
- sroussey 3y agoAt least the “of” loop returns what is selectable via a cursor in the browser. I think…
- josephg 3y agoYeah; its really confusing but javascript - for legacy reasons - treats strings as "arrays of UCS2 items". But javascript also implements iterator on strings which iterate through strings in unicode codepoints. Thats why "of" loops work differently from "in" loops. (for-of in javascript uses Symbol.iterator). That also means you can pull a string apart into an array of unicode codepoints using [...somestring].
- ant6n 3y agolength is not ambiguous at all. Its the number of elements in the array. A string in python3 is an array of unicode code points, so the length of a string is the number of unicode code points. If you want the number of bytes, you need to encode the string in a unicode format (utf8, utf16 or utf32) to get a bytes object, which is an array of bytes. Then you can get the length of that. Remember, one of the big accomplishments (breaking changes) of python 3 is that all strings are Unicode, not byte arrays. If you want to view a dtring as bytes, you need to convert the string to bytes. But note the number of bytes depends on the ancoding u use (utf8, …).
- sirsinsalot 3y agoExactly this. People conflate unicode with encoding quite a bit. I think it was plan9 and early Go that used "runes" as a unit, where one or more runes formed a character and an array of runes could be encoded into bytes using a given encoding. The in memory size of a rune was just an implementation detail, and while it could be important for the programmer that the size of a rune was 2 bytes, this didn't mean the length of an array of 2 runes was 4. I always liked the rune unit, and while my memory is hazy I think it was just code points. I think part of the issue is programmers and apis mixing bit units for in memory representation of a conceptual value mapping (unicode), conceptual characters, stored size when encoded and so on ... without firming up those abstractions with interfaces. It gets lossy.
- eesmith 3y agoFWIW, CPython uses one of several Unicode string implementation representations, depending on the code points involved: >>> import sys >>> s = 'A' * 1000 >>> len(s) 1000 >>> sys.getsizeof(s) 1049 >>> s = '\N{SNOWMAN WITHOUT SNOW}' * 1000 >>> len(s) 1000 >>> sys.getsizeof(s) 2074 >>> s = '\N{MUSICAL SYMBOL G CLEF}' * 1000 >>> len(s) 1000 >>> sys.getsizeof(s) 4076 See https://peps.python.org/pep-0393/ https://peps.python.org/pep-0393/ . Mentioned in the linked-to article with "CPython since 3.3 makes the same idea three-level with code point semantics".
- coldtea 3y ago>length is not ambiguous at all. Its the number of elements in the array That's because you defined it first as "the number of elements in the array". It is ambiguous however because that's not how people understand it when it comes to strings, and there are several counter-intuitive ways they expect it to behave. Not to mention there might not be any "array". A string (whatever the encoding / representation) is a chunk of memory, not an array. That you can often use a method to traverse it doesn't mean it's in an array.
- prng2021 3y ago> A string should be viewed as an opaque data type with views into it depending on what you're trying to do. > The attribute shouldn't be 'length' it should be something like 'countOfCodePoints' or exposed via a `CodePoints` type view. That’s a great way to explain it and something I’ll keep with me. Thanks
- eviks 3y agoWhat do you mean there is no such thing as a character when grapheme cluster is exactly that? This is also the out-of-context , and people get confused because instead of this human context attribute they've been forced to use all the other alternatives that require more knowledge
- deleted 3y ago[deleted]
- arcticbull 3y agoCharacters in context are printable or non-printable/formatting marks right? I agree they probably meant grapheme clusters, but grapheme clusters can vary dramatically in width so the point of the conversation was to explain why a bounding box was a better approximation of their goals.
- cameldrv 3y agoThey do very in width, but with a proportional font that’s true even with ASCII text. What grapheme clusters tells you is how many times you have to press the arrow key/backspace to get to the beginning of the string.
- tsimionescu 3y agoOnly if the text editor made some bad assumptions. You're forgetting about non-printable characters, such as the LTR mark. These are not part of grapheme clusters (or are their own grapheme cluster), but the cursor shouldn't probably stop at them.
- arcticbull 3y agoYou know it's been a long time since this conversation but I think, reflecting, it has to do with grapheme clusters not being particularly consistent across operating systems and over time. The article even has an example where one Unicode spec encodes the same 5 USVs and either 1 or 2 graphemes.
- 3y ago
- charcircuit 3y agoCounting graphene clusters is a hard problem because it depends on the font that is being used. It only exists at render time in the context of a specific client. If the user can freely change a font it is impossible to send a string of 3 graphene clusters because you won't know if it actually will show up as 3 to client or a different number.
- oaiey 3y agoI wanted to brainfart about that length in the typical assumed usage should be 1 ignoring the inner encoding of Unicode of emoji ... But your comment was spot on and showed me my own assumption would fall into exactly this view scheme. Thank you. Have my upvote.
- rightbyte 3y ago> for western programmers And by western you mean american, right? You can't even use ASCII in the UK --- '£'.
- mjburgess 3y ago£ is still just a byte
- edent 3y agoAnd yet it is reasonably common to see "£" when the UTF-8 is misinterpreted.
- johannes1234321 3y agoWhen using latin-1/latin-15/iso-8859-1/iso-8859-15/cp1252 that statement is true. With utf-8 it is two bytes (c2 a3), if a software uses utf-16, ucs-2, etc. it may be more.
- josefx 3y agoNot in any modern encoding and certainly not in ASCII either. Having the highest order bit set makes that kind of problematic.
- tuukkah 3y agoNope, says UTF-8.
- rightbyte 3y ago'u32_pound & 0xff == u32_pound' happens to be true, ye. It doesn't make it a byte. You need the leading 0s.
- mjburgess 3y agoMy mistake
- arcticbull 3y agoYes I was using western as synonymous with ASCII users, and it’s not. My bad.
- pajko 3y ago> Not until they have to explain to their designer why they can't limit a label to '10 characters.' Or in a single font. It's impossible to render any mixed combination of simplified Chinese, traditional Chinese and Japanese with a single font (Korean might be also involved, but not sure about that). Even in Unicode, characters might share the the same space which don't have anything common in their looks, nor in their meaning. That applies to the shared CJK space as well. Btw. Japanese has halfwidth and fullwidth characters.
- thaumasiotes 3y ago> It's impossible to render any mixed combination of simplified Chinese, traditional Chinese and Japanese with a single font (Korean might be also involved, but not sure about that) Well, that could be phrased better. Many such mixed combinations would encounter no problems. There is "Han Unification" in Unicode, in which certain graphical forms are declared equivalent and the intent is that they display as Japanese characters if you print them in a Japanese font, but as Chinese characters if you print them in a Chinese font. 直 is a good example of how that looks; try viewing it in different fonts. But nobody likes unification and explicit fixed forms are constantly being defined so that it's possible to talk about them. Imagine if I wanted to write "in Old English, the word for dog was hund"... except that your font automatically replaced the sequence hund with a special ligature that looks exactly like dog. So we have separate unicode points for ⻘ (modern, CJK RADICAL BLUE) and ⾭ (old, KANGXI RADICAL BLUE), and for ⿓ (traditional Chinese, KANGXI RADICAL DRAGON), ⻰ (simplified Chinese, CJK RADICAL C-SIMPLIFIED DRAGON), and ⻯ (Japanese, CJK RADICAL J-SIMPLIFIED DRAGON). Interestingly, the dragon characters are all considered different according to the original "Han Unified" specification, where they are CJK UNIFIED IDEOGRAPH 9F8D, CJK UNIFIED IDEOGRAPH 9F99, and CJK UNIFIED IDEOGRAPH 7ADC. In contrast, there is only the one "unified" form of 直, CJK UNIFIED IDEOGRAPH 76F4, but you can refer to its Chinese form explicitly with CJK COMPATIBILITY IDEOGRAPH FAA8 and to its Japanese form with CJK COMPATIBILITY IDEOGRAPH 2F940. (My browser font fails to render either of those.) It was never possible to rely entirely on the font to handle dealing with simplified vs traditional characters for you, for the obvious reason that their mapping is not one-to-one. In simplified Chinese, 后 means "after"† or "behind" and it also means "empress". In traditional Chinese, "after" and "behind" would be 後. And "empress" would be... 后. This means there can be no way for a traditional Chinese font to determine what it should display if you write 后. Ultratraditional Korean hanja participate in the same variation of forms that we see between Chinese and Japanese. But it isn't normal to write Korean in hanja outside of very specific contexts. Hangul are radically different and belong to a separate part of unicode entirely. † "After" in time. "After" in sequence is 下, "below".
- zarzavat 3y agoThere are three notions of length that make sense: 1. UTF-8 byte length 2. Code point count 3. Extended grapheme cluster count #3 makes sense for users but it doesn’t make sense for programs which often need to work at the code point level. I expect programming language string length to obey the law: len(a ++ b) = len(a) + len(b) For example, if I concatenate a two strings, one containing an “e” and one containing a combining acute accent, then I expect the length to be longer than a string containing a precomposed ‘é’ character. It’s in fact useful if strings that look the same but have different code points have different lengths, because it tells you that they’re not the same (and maybe you forgot to normalize something etc). Code point length is the most useful for people who are actually writing string algorithms based upon Unicode. UTF-8 length is useful for people who are treating strings as opaque byte sequences, but in that case they should be using a bytes/buffer object and not a string object, except in very low-level languages that don’t want to pay an encoding/decoding cost. Extended grapheme cluster count is useful for people who are constructing certain kinds of user interfaces, where the number of characters is limited for a policy rather than memory or width reason. i.e. when length limits are imposed by human policy, grapheme cluster count is the way to go. Length limits for memory reasons should rather be in UTF-8 bytes. If you need a limit for visual width reasons then you need to go measure the string in pixels, otherwise I’m going to put a U+FDFD in there and ruin your day. UTF-16 length can GTFO.
- wtetzner 3y ago> Code point length is the most useful for people who are actually writing string algorithms based upon Unicode. What algorithms would you be writing against code points?
- insanitybit 3y agoI suspect primarily substring, where if you index by bytes you'll mangle the string, but if you index by codepoints everything works out.
- deleted 3y ago[deleted]
- Izkata 3y ago> It is wrong that "{emoji}".length == 7 I get what you're intending, but it's kinda funny because that string actually is 7
- seagreen 3y agoFantastic comment, thank you. One question I have is which of these are fixed across Unicode standards. (a) Byte count for storage and (b) code point count are definitely fixed, Unicode provides an explicit guarantee of the latter: https://www.unicode.org/policies/stability_policy.html https://www.unicode.org/policies/stability_policy.html (c) Visual width will change depending on the system. But what about (d) grapheme count? If I make a microblogging site which limits post length to 144 graphemes, can my database invariants break when I upgrade my version of Unicode?
- elcaro 3y agoThis is exactly what Raku does. Neither strings nor arrays have a `.length` method because it's arbitrary. [0] > "\c[FACE PALM]".chars 1 [1] > "\c[FACE PALM]".codes 1 [2] > "\c[FACE PALM]".encode.bytes 4 [3] > "\c[FACE PALM]".encode utf8:0x<F0 9F A4 A6> [4] > "\c[FACE PALM]".encode('utf-16') utf16:0x<D83E DD26> FWIW, to get the "length" of an array, the method is `.elems`.
- GoblinSlayer 3y ago>It's particularly bad because so often (esp. for western programmers) 'countOfCodePoints' == 'countOfBytesInUTF8' == 'countOfGraphemeClusters' == """length""" so it's hella easy to accidentally write buggy software. Then programmers will pick a random view and assume its length equals the number of characters and bytes. Also the grapheme view will introduce an OS-dependent bug.