25 ms·

It’s not wrong that "🤦🏼‍♂️".length == 7 (2019)

280 points by triyambakam 3y ago

lifthrasiir 3y ago
HN discards emojis in the title. The original emoji was https://emojipedia.org/man-facepalming-medium-light-skin-tone/ https://emojipedia.org/man-facepalming-medium-light-skin-ton... which consists of 5 Unicode code points. Also please make sure to read the first heading after the title, which summarizes the whole point of this essay.
- dabluecaboose 3y ago
  Was definitely confused on the meaning of that one haha. Thanks for the info!
- deleted 3y ago
  [deleted]
- bobajeff 3y ago
  That makes more sense. I'm sure there could be some language out there where a empty strings length could be 7.
  - dabluecaboose 3y ago
    I have to admit, I gave that thought a bit too much time before I clicked the link and discovered what the title should have said
    - MiddleEndian 3y ago
      Based on the title, I assumed it was gonna be something about smart quotes, and I thought, "of course, another reason to hate smart quotes!"
  - jakear 3y ago
    Not sure if HN will handle it, but this empty string's length is 7: ' ' ( '\u200b'.repeat(7) ) Edit: HN does not, in fact, handle it.
    - dingledork69 3y ago
      HN does, in fact, handle it, and chose to discard it.
      - bombcar 3y ago
        You need to use the latest and greatest: Awww dang they updated the filter, booo
        TeMPOraL 3y ago
        You can remove the word "they" from your comment and it'll still work for this site.
      - jakear 3y ago
        In the CIA since of "we need to handle these democratically elected leaders", perhaps. But if I was selling you a drop-in comment widget and boasted "it handles all of Unicode", but really I was just running s/[^ -~]+/ /g, wouldn't you feel a bit let down?
- ethbr0 3y ago
  Today I learned, that multi-glyph emojis are a thing. Zero width joiner emoji sequences https://emojipedia.org/emoji-zwj-sequence/ https://emojipedia.org/emoji-zwj-sequence/
- dang 3y ago
  Ok, we've remojied the title above. Edit: there are always exceptions - https://news.ycombinator.com/item?id=34460417 https://news.ycombinator.com/item?id=34460417
  - skykooler 3y ago
    Interestingly, Firefox on Wayland renders the emoji correctly in the tab title, but the window title renders it as two rectangles and the male symbol. I assume this must be some difference between system fonts vs Firefox's fonts.
  - avgcorrection 3y ago
    dang is definitely wrong about what “plain text” means in that thread.
    - dang 3y ago
      Happy to wrong but when saying something like that, you should explain why and how, so we can all learn something!
      - avgcorrection 3y ago
        Plain text is often contrasted with markup or rich text. UTF-8 text is neither markup nor rich text.
sillysaurusx 3y ago
Measuring the length of text is really hard. Font fallback is hard. All of these things, you take for granted till you write your own game engine. Apparently the thing to use is a library with a very strange name, which does glyph placement. I’ll go look for it. EDIT: harfbuzz https://harfbuzz.github.io/why-do-i-need-a-shaping-engine.html https://harfbuzz.github.io/why-do-i-need-a-shaping-engine.ht...
- missblit 3y ago
  Measuring the length of text is easy. Now measuring the length of text before actually printing it / running the shaping engine, now that's hard.
- grt_thr 3y ago
  [dead]
grayhatter 3y ago
> Python 3’s approach is unambiguously the worst one, though. Did I miss the part where he explains this take? It's made up of 5 valid unicode code units. For a language where you're not supposed to need to know the byte size semantics, the correct length should be 5. What am I missing? The close second being 17, because length in bytes. Is another fine way to represent this data, e.g. what a successful write of some sort would look like. Network or file. I guess I'm basing this all on the idea that it's almost always a mistake to confuse how a program manages some data, vs how a drawing lib might. Your language shouldn't concern it self with how many glyphs it needs to draw... until you actually try to draw them.
- benatkin 3y ago
  I think he meant to bring out defensiveness with that quip. He never says that it's a big deal, just that it's the worst way to get the length of a string containing emoji, presumably of the mainstream languages.
  - tehsauce 3y ago
    Why is it the worst way though?
    - deleted 3y ago
      [deleted]
    - imron 3y ago
      > Note about Python 3 added on 2019-09-09: Originally this article claimed that Python 3 guaranteed UTF-32 validity. This was in error. Python 3 guarantees that the units of the string stay within the Unicode code point range but does not guarantee the absence of surrogates. It not only allows unpaired surrogates, which might be explained by wishing to be compatible with the value space of potentially-invalid UTF-16, but Python 3 allows materializing even surrogate pairs, which is a truly bizarre design. The previous conclusions stand with the added conclusion that Python 3 is even more messed up than I thought!
    - markmark 3y ago
paddw 3y ago
I think this is a really a naming convention issue. Len() is ambiguous, you really want either num_chars() or utfxx_len(). Of course, the issue of what counts as a character is confusing in its own right...
- arp242 3y ago
  In Python len() on a bytes type gives you the number of bytes, and len() on a str type gives you the number of codepoints. I think that makes sense, as strings are only intended to deal with text, and you should never have to worry about byte indexing at all.
  - lmm 3y ago
    The argument is that indexing by codepoint is even less useful than indexing by byte.
    - chasontherobot 3y ago
      As someone who has done both, I'd say that argument is wrong. It is much more convenient to index by code point. Indexing by bytes is almost always what you don't want to do, and leads to a lot of errors.
      - lmm 3y ago
        What were the use cases where you found it useful to index by code point (and therefore not by grapheme cluster)?
        arp242 3y ago
        In many cases it's not very useful, but there are clearly cases where it is, e.g. if you want to normalize text, compose/change emojis, stuff like that. A codepoint is the "smallest useful addressable unit" when dealing with Unicode text, so it makes sense that's the default. It's also comparatively expensive to address grapheme clusters.
geoffpado 3y ago
(2019), for what it's worth.
- benatkin 3y ago
  In 2023 we should be able to see that Python's isn't the worst, or rather is tied for the worst, and that they're all a little bit tricky but fine.
hgs3 3y ago
Python 3's approach is the most correct: Unicode defines text as a sequence of code points. UTF-whatever is an implementation detail.
- adgjlsfhk1 3y ago
  Unicode defines text as a number of different types of things. They are sequences of codepoints, sequences of graphemes, sequences of graphime clusters. Furthermore, codepoints are different depending on how you normalize them. Accented characters can be written two different ways and have a different number of codepoints depending on how you write them (and if normalization is used)
  - Spivak 3y ago
    Grapheme are a made up human thing that, while useful, is locale dependent. Most people when they talk about grapheme clusters mean the default "locale-independent" graphemes but it's not the only one (in Hungarian for example 'ly' is a single letter). Having the same string be two different lengths in two countries is… let's go with surprising. The common denominator where everyone computes the same number is code points.
    - adgjlsfhk1 3y ago
      Except they won't (if they are doing normalization). à and à have different numbers of codepoints (the first is 0x00E0, the second is 0xc3 0xa0).
- pjscott 3y ago
  Treating Unicode strings as a sequence of code points is a completely valid thing to do, but is usually not what you actually care about when dealing with text. Really, are code points any less of an implementation detail?
  - croes 3y ago
    I think parent means most correct of the three given examples from Rust, JS and Python. Especially because the article says that Python's take is the worst.
ars 3y ago
Am I wrong for assuming the .length should return a length in bytes? If you want to use 32bit units, then multiply your output by 4. If you want to do Unicode string manipulation and length counting, then use specific functions for that - but the base internal .length function should just output bytes.
- Arnavion 3y ago
  >then multiply your output by 4. That is not how UTF-32 works. >but the base internal .length function should just output bytes. Do you think the length of an `int64_t[3]` array should be 3 or 24?
  - ars 3y ago
    > Do you think the length of an `int64_t[3]` array should be 3 or 24? There should be functions to do both: sizeof(int64_t[3]) * sizeof(int64_t) for example to get bytes. In this example, the base function should do bytes, and there should be a unicode function to count it in other ways. I could be sizing to fit in a database, or send over the wire, or I might want visible space on the screen, or I might want to know how to move the cursor. Each of those types of length should be supported.
    - tialaramex 3y ago
      > There should be functions to do both: sizeof(int64_t[3]) * sizeof(int64_t) for example to get bytes. That's not what that does, the answer to the code you wrote is 192 because you're multiplying the same size factor twice.
      - ars 3y ago
        No I'm not, read it again. The first one has size 3, because there are 3 elements in the array. The second one has size 8 because an int64 is 8 bytes.
        tialaramex 3y ago
deleted 3y ago
[deleted]
2h 3y ago
this one one of those things that people point to when comparing languages, but in reality rarely matters. with Go, you just get the number of bytes, which the the correct default thing to do: https://godocs.io/builtin#len https://godocs.io/builtin#len if the language default was anything other than this, THAT WOULD BE WRONG and unexpected. I would prefer the default to be the dumb, fast thing. then if I want the slow, fancy thing, I can import some first or third party package.
- ladberg 3y ago
  I think to some extent it depends on the language. In the article they talk about Swift's implementation, which by default does the slow, fancy thing (but makes it easy to do the dumb, fast thing). String manipulation in Swift is almost certainly going to be used for a GUI for end users of many possible languages / locales, so it makes sense to spend the extra cycles to get the fancy version by default. If it isn't the default then you'll end up with half the apps on the App Store displaying broken text on line breaks, ellipses, wrapping, etc. on their hand-rolled UI stack.
- eviks 3y ago
  Why would you want dumb??? (and it's not expected that a character's length is>1 unless you've been conditioned to excpect it)
  - Mawr 3y ago
    Because whenever you want to store or transmit a string only the byte count matters (the size of the string). All the fancy unicode stuff on top of bytes is for the display layers to handle. The default should be grounded to the reality of the programmer.
    - eviks 3y ago
      Human interaction is a more grounded reality for programmers vs. the dumb land of pure bytes, so even at that conceptual level the default should be smart And bytes is the only thing that matter for a specific type of string, conveniently named, sequence of bytes
dahfizz 3y ago
Maybe not wrong, but it's the worst option. 5 is the number of code points, and 17 is the number of bytes. Both are reasonable answers. 7 is the number of code units for utf-16. Seems like the least useful option.
- Dylan16807 3y ago
  That means 7 is also a measure of bytes, just slightly more awkward. So it's roughly on par with 17. For 5, the idea is that while you might want to iterate code points, the total number of code points is less useful than either grapheme count or byte count. I think that argument makes sense.
  - lmm 3y ago
    > That means 7 is also a measure of bytes, just slightly more awkward. It's not a real measure of bytes though. It's the count of bytes in an encoding scheme that is (probably) neither what you use to communicate with the outside world nor what your language runtime uses. (And certainly it's no better than 5, since that's also a measure of bytes in a particular encoding).
    - MobiusHorizons 3y ago
      The JavaScript language forces utf16 (whether or not v8 uses that representation under the hood). For instance if you want to substring the indexes you pass are for utf16 codepoints
      - lmm 3y ago
        Sure, but arguing that that's a good reason for length to count utf16 is purely circular.
    - Dylan16807 3y ago
      Lots of systems use UTF-16 internally and externally. Counting bytes in UTF-16 is, on average, almost as useful as counting bytes in UTF-8. I don't think just about anything communicates in UTF-32. 5 is basically just a codepoint count, and as such I don't think its usefulness rating should be between the byte counts.
bitwize 3y ago
Once again, strings are not simple sequences of characters. It's also useless to "index" into a string without specifying what you're indexing for the same reason.
- astrea 3y ago
  This is like when you first dive into date-time and slowly unravel the maddening complexity of it all.
xeeeeeeeeeeenu 3y ago
I'm not a fan of "everything you know about X is wrong" articles. Very often they try to present some little tidbit of knowledge as a revelation and mislead the reader in the process. In this case, the tidbit is: "grapheme clusters exist and they are useful". The misleading part is that the article draws a false equivalence between what the author calls "UTF-32 code units" and UTF-16 code units. UTF-32 code units are Unicode code points. This is a general Unicode concept that exists in all Unicode encodings. UTF-16 code units, on the other hand, are an implementation detail of UTF-16. It is wrong to present them as equally arbitrary concepts.
- masklinn 3y ago
  > UTF-32 code units are Unicode code points. They’re not. UTF-32 code units have a 1:1 mapping to USVs, surrogates are not valid. > It is wrong to present them as equally arbitrary concepts. Is it? It’s not like they’re any more useful. Arguably less so, UTF-16 is at least a somewhat common storage medium.
  - xeeeeeeeeeeenu 3y ago
    >They’re not. UTF-32 code units have a 1:1 mapping to USVs, surrogates are not valid. This is true, although very pedantic and irrelevant to the point of my comment. The distinction only matters when you're dealing with ill-formed strings. BTW, Python strings can store surrogates. >Is it? It’s not like they’re any more useful. Arguably less so, UTF-16 is at least a somewhat common storage medium. If you aren't directly dealing with UTF-16, UTF-16 code units aren't useful at all. Code points/USVs, OTOH, are the building blocks of Unicode strings and various Unicode algorithms operate on them. They're low-level, but not useless.
- chrismorgan 3y ago
  > UTF-16 code units, on the other hand, are an implementation detail of UTF-16. Would that it were only so. Instead, UTF-16 ruined Unicode for everyone with the abomination that is surrogates, and almost nothing that deals with UTF-16 actually asserts well-formedness, and ill-formed UTF-16 cannot be represented in UTF-8 or UTF-32. UTF-32 and UTF-8 code units are truly implementation details of their encodings, as other encodings don’t need to know about them in any way. UTF-32’s code units are a trivial mapping between scalar values and 32-bit values (not four-byte values, given the big- and little-endian variants), but that still causes UTF-32 code units to be semantically distinct from Unicode scalar values. U+12345 is a Unicode scalar value and doesn’t have any “size”: it’s an abstract value. 0x00012345 is a UTF-32 code unit, a 32-bit value. If you’re talking about encoding of Unicode scalar values, you talk about code units. Even when talking about UTF-32, the code unit/scalar value semantic distinction is worth maintaining.
duxup 3y ago
Is there a situation where I am going to try to get the length of an emoji and I care about the outcome?
- justinator 3y ago
  There's most definitely a situation where you want the length of string that contains an emoji character and perhaps dumb things happen if you get that wrong.
- nosefurhairdo 3y ago
  Enforcing a character limit on user input.
  - hermitdev 3y ago
    Now I want to know: how much does this emoji count towards password strength?
jlebar 3y ago
I was 100% prepared to believe that the length of empty string in js is 7. Then upon opening the post I was 100% ready to believe that js has three different string length functions that all handle Unicode differently.
- baq 3y ago
  and the gist of the problem is that it doesn't.
secret-noun 3y ago
Related: I wrote a little web app that lets you see the codepoints for text like this https://unicode-x-ray.com/?t=%F0%9F%A4%A6%F0%9F%8F%BC%E2%80%8D%E2%99%82%EF%B8%8F https://unicode-x-ray.com/?t=%F0%9F%A4%A6%F0%9F%8F%BC%E2%80%... (sorry if link looks scary, that's just the URL encoding of this emoji)
- farhanhubble 3y ago
  I worked on Unicode parsers for several years and the single most indispensable tool was https://r12a.github.io/app-conversion/ https://r12a.github.io/app-conversion/
eviks 3y ago
Very good and informative article, though still not convincing that the nudge to make the shortest "len" command use the human readable size of grapheme clusters like in Swift isn't the best design approach, all the non-intuitive sizes should be special
- pavlov 3y ago
  The article shows that the Swift approach produces different values for length depending on operating system and text library versions. Is that really intuitive?
  - eviks 3y ago
    Showing unintuitive results in a small number of cases always beats showing unintuitive results for all the cases. The Swift approach can also asymptotically reach perfection. The other approach will be broken forever. These are categorical improvements in design
    - pavlov 3y ago
      The Swift approach can't reach perfection in isolation because data from the future can always break it. That's why in the article you see Swift running on Ubuntu 14.04 returning len==2 while the same code on Ubuntu 18.04 returns len==1 for the same emoji string. IMO that's a big philosophical question here: do we accept that "string length" means something you can't compute for arbitrary strings unless your code is receiving annual updates containing the latest Unicode interpretation instructions?
      - stephencanon 3y ago
        Swift includes its own Unicode data tables with the standard library since last year, so it’s now tied to the stdlib version rather than some other library that may or may not be updated on the system.
      - eviks 3y ago
        Your example shows an improvement, which proves my point (also don't drop the word asymptotically, nothing can ever be perfect, that's not the issue, being closer to perfect is a positive) And you can compute it, you can pin a Unicode version and ship it in the language if those platform differences are unbearable (so, you can actually isolate it and simply ignore the future :)) The bigger philosophical question: how much longer do we accept that "string length" does not measure the most intuitive measure of string length and call a byte a char?
omoikane 3y ago
Ruby gives you the choice to iterate over all types, via `each_byte`, `each_char`, `each_codepoint`, or `each_grapheme_cluster`. https://ruby-doc.org/3.2.2/String.html#class-String-label-Methods+for+Iterating https://ruby-doc.org/3.2.2/String.html#class-String-label-Me...
- h0l0cube 3y ago
  Elixir also makes this distinction: https://hexdocs.pm/elixir/1.12/String.html#module-code-points-and-grapheme-cluster https://hexdocs.pm/elixir/1.12/String.html#module-code-point...
- nerdponx 3y ago
  Python would benefit from this significantly IMO.
- sundarurfriend 3y ago
  In Julia, iterating over a string by default behaves like `each_char`. `codeunits(str)` lets you access the underlying code units, which is bytes for the default UTF-8 encoding. (External packages implement UTF-16 and others, and there `codeunits` could return non-bytes, for eg. 16-bit values for UTF-16.) The Unicode stdlib provides `graphemes(str)`, the equivalent of `each_grapheme_cluster`.
chefandy 3y ago
It's 1 in elixir which measures graphemes by default. iex(3)> String.length(" ") 1 Edit: looks like HN doesn't support that emoji in code blocks, at least.
- jnsaff2 3y ago
  it's also 5 if you like: > length(' ') Which is: [129318, 127996, 8205, 9794, 65039]
justinator 3y ago
perl -e 'use utf8; print length(""). "\n";' 1
- justinator 3y ago
  Should be the emoji char between the double q's in, 'length("")' but whatcha gonna do.
ceeam 3y ago
These emoticons should never have been a part of Unicode in the first place. Second big mistake of that org after the Unihan fiasco.
- themerone 3y ago
  Emoji's are a fun way to illustrate a problem that happens with languages too.
- dchest 3y ago
  If not for emoji, lots of software wouldn't care about correctly processing strings in other languages, so it's good that we have them.
  - hnfong 3y ago
    This. I've been quite happy that popular emojis were introduced in supplementary planes, because my language has quite a few common words (eg. 𨋢 [lift/escalator]) that ended up on plane 2. Proper software support for those characters used to be terrible, but things got much better after emojis became popular. So, thanks and sorry everyone :)
- Findecanor 3y ago
  I agree that they introduced unnecessary complexity for text encoding, and for font-rendering (which are expected to support multi-coloured emoticons now). I once started writing on a text editor, and then fell deep into Unicode handling. I have now spent more work on the Unicode parts than on anything else in the program. I think that the industry could have instead adopted the old web-forum convention of colon-word-encoding, originating from ASCII art. Example: ":facepalm:". When the sequence is not supported as an emoji, it degrades gracefully into text that can be understood by anyone reading it instead of into a sequence of empty squares or diamonds with question marks in them. Text also provides a more efficient input method than having to browse for an icon in a list.
  - kps 3y ago
    Naturally Unicode did the equivalent of colon-encoding too, for flags.
deleted 3y ago
[deleted]
neallindsay 3y ago
Until reading this I had never heard of UTF-32. It doesn't seem like a good way to encode strings.
- nerdponx 3y ago
  It's useful if you want array-like semantics (e.g. O(1) lookup) on Unicode text strings, because you have a fixed size for every codepoint, unliked UTF-8. Python for example uses it internally.
  - Joker_vD 3y ago
    And it compresses just as well as UTF-8 for transfer/storage purposes.
  - chrismorgan 3y ago
    Except code point indexing simply isn’t useful. In the words of the article: “The choice of UTF-32 (or Python 3-style code point sequences) arises from wanting the wrong thing.”
    - nerdponx 3y ago
      I think that's the issue here. People disagree on how useful or not useful it is. It's maybe not ideal, but I don't think it's anywhere near so bad as to be entirely not useful. Strings-are-sequences-of-bytes is worse in my opinion. Python literally used to have that. It was worse.
      - chrismorgan 3y ago
        The problem with what Python used to have is that the encoding wasn’t fixed. I’ll agree with you that strings-are-sequences-of-bytes is bad. That’s painful compiler-flag, codepage, &c. territory. But what’s not bad is strings-are-sequences-of-code-units. That’s what Rust has, for example. Rust strings aren’t sequences of bytes, but of UTF-8 code units, and the two are semantically very different.
foxes 3y ago
All these abominations are because of non strict typing String = List ( Char ) Chars don’t have a length, like a number doesn’t have a length - unless you talk about number of bits. If you are working with strings stick with strings. The string of a single character should be “1”. Just enforce proper typing. Anything else is not consistent.
- chasontherobot 3y ago
  Python has strong typing which seems to be what you mean here rather than strict typing. A "character" is not a well defined term in Unicode, rather the "base" that does not vary across implementations is code points, which is what Python measures when you get the length of a string.
- jillesvangurp 3y ago
  No, it's caused by cost. For example Java has a char type. It's a 16 bit numeric value because Java uses UTF-16 internally for encoding strings. Java Strings are basically immutable char arrays with some fluff around them. If you ask for the String length, it returns the length of the underlying array. Nice and simple and unsurprising. And relatively cheap. Most more recent languages use 8 bit bytes and UTF-8 instead because that is emerged as the most common character encoding. But UTF-16 was a reasonable choice a quarter century ago and the practical difference doesn't matter that much and changing it would be disruptive. If you put unicode characters consisting of multiple data points into a String, it necessarily increases the amount of chars. There's no way around that. Because there is no such thing as a UnicodeChar type in Java. You can't actually assign multi data point unicode characters to a char. Essentially all the workarounds for a 'correct' unicode character count in a String would either end up using a different and probably way more expensive data structure (e.g. a list of a list of chars or bytes where each list is a unicode character) or implementing some expensive logic for counting characters that is O(n) instead O(1). Most languages ranging from extremely strictly typed to weakly typed don't do that for cost reasons. The tradeoff is simply not worth the price it takes. This stackoverflow post provides a few suggestions for how you could count 'correctly'. https://stackoverflow.com/questions/15947992/java-unicode-string-length https://stackoverflow.com/questions/15947992/java-unicode-st... that illustrates the point nicely.
Jach 3y ago
Unsurprising that (at least some implementation of) Swift does the least wrong thing in returning 1. I think it's also one of the few languages that will return a count of 1 for the madness that is country flag emojis https://docs.swift.org/swift-book/documentation/the-swift-programming-language/stringsandcharacters#Extended-Grapheme-Clusters https://docs.swift.org/swift-book/documentation/the-swift-pr...
- ant6n 3y ago
  “Least wrong” sounds very silly. Its like programmers are discovering theres a difference between bytes, unicode code points and grapheme clusters and are unsure about how their favorite programming language represents strings, and then decide there should be some behavior that doesnt follow from the documentation. The “length of an emoji” depends on the data type used to represent it. Its that simple and that correct.
- baq 3y ago
  'least wrong length method' for strings is not providing one in the first place.
deleted 3y ago
[deleted]
quaintdev 3y ago
How the hell you inserted an emoji in title? Afaik we cant use emoji on this website
- Arnavion 3y ago
  https://news.ycombinator.com/item?id=36160029 https://news.ycombinator.com/item?id=36160029
- deleted 3y ago
  [deleted]
farhanhubble 3y ago
Why should length in programming be devoid of units? Why can't we have length be (8, UTF32_CODEPOINTS) <class UTF32_CODEPOINT_SZ(8)>?
- mrkeen 3y ago
  We can and do - they're called types. However it's always going to be up to the library developers as to whether two things are the same type.
frou_dh 3y ago
I encountered some real world unicode/emoji breakdown recently. I set my surname in a webapp to an emoji country flag because I needed a way to communicate where I was. Elsewhere in the app, it showed surnames as just their initial, e.g. "John S". There, mine showed as a featureless black flag rather than the flag I set. Presumably because that is the first codepoint of several that make up the flag.
- tialaramex 3y ago
  > There, mine showed as a featureless black flag rather than the flag I set. Presumably because that is the first codepoint of several that make up the flag. The country flags are each made of two Unicode code points, which Unicode calls Regional Indicator Symbols. There are twenty six, one for each of the Latin capital letters A through Z. These are used to encode a flag by writing the ISO two letter country code from ISO-3166-1 e.g. F + R is France, you get a French flag. Given your black flag experience, and the fact this is an English language forum, I'd guess maybe you wanted a flag for some entity that isn't a UN member state or some sort of recognised similar entity (e.g. the European flag EU symbolising the continent of Europe) and thus doesn't have an ISO two letter code, such as California or Wales. Those are built from a waving black flag plus their long ISO-3166-2 region code
namaria 3y ago
I have read somewhere that you should learn 2 or 3 programming languages from the get go. If you learn one, you run the risk of letting it's shape dictate how you mentally model computation. At some point someone who learned a dynamically typed programming language first is bound to find out why data types matter.
- Firmwarrior 3y ago
  I started with JavaScript and went to c/logic gates/assembly, and it wasn't too bad I had to learn what pointers were, but it was OK, haha. That said, maybe people who don't have "The Knack" would be better off learning a slightly harder language first..
  - namaria 3y ago
    I definitely don't have talent for logic and quantitative thinking. It takes a long time and many iterations for even simple concepts in mathematics to sink in for me. I benefited greatly from learning first Scheme and also making sense of C and OS internals before trying to grok interpreted languages. I'm currently trying to get some proficiency in Go and it's been great fun!
- preommr 3y ago
  > should learn 2 or 3 programming languages from the get go. So... Java/C#/Kotlin?
- omoikane 3y ago
  I had a "programming languages" class that did that, where we did assignments in Python (scripting), OCaml (functional), and Prolog (logic). This is because most other classes used compiled imperative languages such as C++ and Java.
alkonaut 3y ago
Really the correct way to design string APIs would be to not have an ambiguous "length" at all, but to always require specifying whether you want UTF8-bytes, memory bytes, code points, graphemes, whatever. However such an API would be pretty cumbersome because for all non-edge cases (read: a western language and a reasonable encoding that language - which when looking at world demographics is a very narrow way of saying non-edge case) we just want to ignore all that fancy stuff and assume it's latin-1/ascii and use "Length" and get on with it, usually accepting that it doesn't work for many scripts or emoji. So almost every api I have encountered has both the dangerous or ambiguous "length" and any number of the more specific counts. Good? No. But good enough, I guess. A much worse related API that exists every where is that for parsing and formatting numbers to and from text. How that's done "depends" but most languages I have seen - unfortunately - offers a "default way". In the worst examples - looking at you .NET - this default uses the system env and assumes formatting and parsing numbers should us the OS locale. Horrible horrible idea when used in conjunction with automatic type conversions. WriteLine($"The size is {3.5}"); shouldn't print "3.5" in the US and "3,5" somewhere else.
- hardware2win 3y ago
  >Horrible horrible idea when used in conjunction with automatic type conversions. WriteLine($"The size is {3.5}"); shouldn't print "3.5" in the US and "3,5" somewhere else. Actually why?
  - alkonaut 3y ago
    Because it’s only (maybe) a good design if it’s to be read by a human but that’s not a very general case. Instead people unknowingly make for example some exporter for a text format and write code that writes "X={x_coord}" and it passes all the unit tests and all the acceptance tests and then it breaks once it hits a French or scandinavian machine. A great example how bad it is would be that the C# compiler repo for a very long time had tests that failed for everyone with non-US formatting.
- ezfe 3y ago
  Swift defaults to grapheme clusters, and allows you to request other lengths on-demand.
kolibril13 3y ago
In python, `print("-".join(f"{ord(c):x}" for c in emoji))` will give `'1f926-1f3fc-200d-2642-fe0f'`. That can be useful to request emojis, see https://github.com/hfg-gmuend/openmoji/blob/ad588c8fb4b028d71c5743347ef5dc6968b97579/FAQ.md?plain=1#L141-L157 https://github.com/hfg-gmuend/openmoji/blob/ad588c8fb4b028d7...
PhilipRoman 3y ago
I cannot think of a single common case where grapheme cluster count is important. If you want to print them aligned to a terminal - guess what, double width characters exist, so the only reliable way is to print them first, measure the cursor movement using escape sequences, calculate length and erase the originally printed data. Even for limiting input field sizes byte count is much better, as otherwise you are opening up yourself for unicode denial of service. I think the game Minecraft has such an exploit where you can fit in absurd amounts of utf-8 data (to the point of data corruption in multiplayer games) since it's limited by visual length. My personal favorite dealing with UTF-8: pretend it's ascii and assume everything above 128 is an alphabetic character. It just works. For 99% of use cases it doesn't matter if the content is emojis, families of emojis, or ancient sumerian scripts. You can parse JSON and most other formats this way without caring about code points at all. The trend of unicodizing everything was a mistake, just treat strings as bytes and parse them as utf-8 only when you really need it (like when building a text editor or a browser engine from scratch).
- zkldi 3y ago
  this doesn't "just work" at all? utf-8 is a variable width encoding and if you treat it like ascii your software just isn't going to work outside of the ascii range.
  - PhilipRoman 3y ago
    But it does - the genius of utf-8 is that it was deliberately designed to be backwards compatible (it even preserves the ascii sorting order). You can run C programs written before utf-8 was invented with utf-8 inputs (unlike with the abomination that is utf-16). If a code point is outside the ascii range (0-127 inclusive), then it's utf-8 encoding is also guaranteed to not contain any ascii bytes. So as long as you treat anything inside 128-255 as "some unknown character", the utf-8 code points will be preserved and eventually displayed when the byte sequence is parsed as utf-8 by your terminal/browser/whatever.
- vlmutolo 3y ago
  I think tweets are restricted by number of extended grapheme clusters. And probably also by number of bytes.
Obscurity4340 3y ago
Can anyone comment as to whether there are any problems associated with using emojiis to enhance the entropy of passwords? For passwords you only need to autofill but never actually type, I feel like it would be an easy way to augment passwords but I don't know whether it would directly translate in every situation.
tylergetsay 3y ago
Interesting; emojis also make SMS really weird. according to twilio including a emoji will change the character limit from 160 to 70
- dpiers 3y ago
  That’s because SMS has a 1120 bit message capacity and uses one of two encoding mechanisms. GSM-7 uses 7 bits per character and a correspondingly limited character set that allows for 1120/7 = 160 characters. UCS-2/UTF-16 uses 16 bits per character for a total of 1120/16 = 70 characters per message.
- locallost 3y ago
  It's not weird because it's not a 160 limit in the sense of characters. Non-English language people are I guess more aware of this -- using a character not in the standard English a-z often counts as two characters.
julik 3y ago
So many of these conversations could be easier if there would not be `length()` functions but `length_in_<whats_exactly>()` functions instead.
- speleding 3y ago
  In ruby you have " ".codepoints.size == 5 and " ".bytes.size == 17 (It also has `length` which equals codepoints.size)
  - Freaky 3y ago
    .each_codepoint.size is more efficient than .codepoints.size, as it creates a sized Enumerator that avoids needing to build an intermediate Array. For strings with only single-byte characters it reduces to returning the already-stored stored byte length. Same goes for .each_byte.size, but for that you have the faster .bytesize method that avoids the intermediate Enumerator.
  - runarberg 3y ago
    JavaScript is a weird one. To count UTF-16 bytes you write: " ".length For unicode character count you write: [..." "].length And for grapheme count (or language aware word/sentence count) you write: [...new Intl.Segmenter('en-US', { granularity: "grapheme" }).segment(" ")].length For word/sentence count you swap out the granularity option.
    - chrismorgan 3y ago
      > [..." "].length Mind you, this is inefficient due to unnecessarily constructing an array. Here’s a more efficient version, though the difference will normally be fairly slight: function codePointLength(str) { let len = 0; for (const c of str) { len++; } return len; } Kinda sad there are no equivalents to the Array methods that work on iterators. Array.prototype.reduce.call(str[Symbol.iterator](), (a, _) => a + 1, 0) doesn’t work since those methods only work on array-like types (meaning those with a length property and indexed by number—and yes, all these Array methods are explicitly defined that way deliberately so you can use them on other array-like types), not iterators. > [...new Intl.Segmenter('en-US', { granularity: "grapheme" }).segment(" ")].length Caution: Intl.Segmenter may not be available, so be sure to have a fallback if you want to use it. Chromium shipped it 2½ years ago, Safari 2 years ago, and Firefox hasn’t shipped it yet. (No idea why and I haven’t looked. It’s not always the case: I know of other Intl things that Firefox has shipped first.)
asimpletune 3y ago
What does swift or rust return?
twodave 3y ago
Obligatory: https://www.joelonsoftware.com/2003/10/08/the-absolute-minimum-every-software-developer-absolutely-positively-must-know-about-unicode-and-character-sets-no-excuses/ https://www.joelonsoftware.com/2003/10/08/the-absolute-minim...
- chrismorgan 3y ago
  I was never much impressed with that article (too much irrelevant story for an “absolute minimum”), and by now it’s very dated. A lot of what it’s talking about (most notably code pages) is now completely irrelevant to the vast majority of developers, who might never encounter or need to worry about them in their entire careers.
hateful 3y ago
The wife and I have a Google sheet that we use for our shared calendar - and we put an emoji before each "event" and in top row of each day I show the Emoji for that days entries. But I need to do: > =LEFT(F280,2) & LEFT(F281,2) & LEFT(F282,2) & LEFT(F283,2) Since the Emojis are actually 2 bytes.
- jameshart 3y ago
  And as this article indicates, many - and indeed an increasing number - of emojis are many more than 2 bytes. Flags, skin tones, groups, professions…
dveeden2 3y ago
MySQL has two different length functions... mysql> WITH chars AS (SELECT ' ' c) -> SELECT LENGTH(c), CHAR_LENGTH(c) FROM chars; +-----------+----------------+ | LENGTH(c) | CHAR_LENGTH(c) | +-----------+----------------+ | 17 | 5 | +-----------+----------------+ 1 row in set (0.01 sec) Note that the doesn't seem to render in preformatted text on HN. This should be easier to reproduce: mysql> WITH chars AS (SELECT 0xF09FA4A6F09F8FBCE2808DE29982EFB88F c) -> SELECT CONVERT(c USING utf8mb4), LENGTH(c), CHAR_LENGTH(c) FROM chars; +--------------------------+-----------+----------------+ | CONVERT(c USING utf8mb4) | LENGTH(c) | CHAR_LENGTH(c) | +--------------------------+-----------+----------------+ | | 17 | 17 | +--------------------------+-----------+----------------+ 1 row in set (0.00 sec)
rzwitserloot 3y ago
Java loaded full unicode code point semantics into its standard `java.lang.String` class. These _are not guaranteed_ to have `O(1)` performance characteristics, because the underlying storage format is dynamically either a UTF-16-esque variant (with surrogate pairs for characters that don't fit in 16 bit), or a single-byte-per-char format if the string does not contain any non-ASCII. This has the advantage of being very very slightly more obvious, given that both methods exist and are documented: void main() { String x = "(that emoji here)"; System.out.println("Chars: " + x.length()); System.out.println("Codepoints: " + x.codePointCount(0, x.length())); System.out.println("As stream of chars (= UTF16-esque with surrogate pairs):"); x.chars().forEach(System.out::println); System.out.println("As a stream of codepoints:"); x.codePoints().forEach(System.out::println); } This ends up printing: Chars: 7 Codepoints: 5 As stream of chars (= UTF16-esque with surrogate pairs): 55358 56614 55356 57340 8205 9794 65039 As a stream of codepoints: 129318 127996 8205 9794 65039 NB: Apparently many hackernews readers know java but don't use it all that often day-to-day. The provided java snippet is vanilla valid and can be executed with `java ThatFile.java` (no need to compile it first), though it does use preview features. The fact that the codepoint counter is a very awkward `codePointCount` call has the dubious benefit of highlighting this method loops through and therefore would be quite slow on very large strings.
- ElectricalUnion 3y ago
  > The provided java snippet is vanilla valid Did not you still need the `java --source 11 ${filename_without_java_extension_because_JEP_330}` to use it? And you still need a wrapper class with a static method main in it.
  - JZerf 3y ago
    I was a little puzzled by this compared to what I was used to with Java in the past. It looks like the grandparent's code relies on JEP 445 ( https://openjdk.org/jeps/445 https://openjdk.org/jeps/445 ) which is a preview feature as was mentioned but it also apparently requires the very latest Java 21 which hasn't even been officially released yet.
  - 3y ago
JohnFen 3y ago
It's things like this that make me hate unicode so much.
iandanforth 3y ago
Life imitates Hitchhikers Guide to the Galaxy. Sure the answer is 7 but do you know what the question is?
joshspankit 3y ago
My RSS reader showed the title as “It’s not wrong that “ “.length == 7” and I had to click on it
11235813213455 3y ago
[..." "].length === 5 // in JS
rurban 3y ago
Naming, folks. Length = 5 Size: depends on the encoding Width = 1
eternityforest 3y ago
1 seems like the only acceptable answer. For all intents and purposes, a user will count it as one character. Truncating the string without including the whole cluster would change the meaning of it, and is not an operation anyone would do as a general purpose thing any more than someone would want to randomly replace the last character with random letters. It looks like one character. I'd rather APIs let us continue pretending it is one character.

280 points by triyambakam 3y ago

HN discards emojis in the title. The original emoji was https://emojipedia.org/man-facepalming-medium-light-skin-tone/ https://emojipedia.org/man-facepalming-medium-light-skin-ton... which consists of 5 Unicode code points. Also please make sure to read the first heading after the title, which summarizes the whole point of this essay.

dabluecaboose 3y ago
Was definitely confused on the meaning of that one haha. Thanks for the info!
deleted 3y ago
[deleted]
bobajeff 3y ago
That makes more sense. I'm sure there could be some language out there where a empty strings length could be 7.
- dabluecaboose 3y ago
  I have to admit, I gave that thought a bit too much time before I clicked the link and discovered what the title should have said
  - MiddleEndian 3y ago
    Based on the title, I assumed it was gonna be something about smart quotes, and I thought, "of course, another reason to hate smart quotes!"
- jakear 3y ago
  Not sure if HN will handle it, but this empty string's length is 7: ' ' ( '\u200b'.repeat(7) ) Edit: HN does not, in fact, handle it.
  - dingledork69 3y ago
    HN does, in fact, handle it, and chose to discard it.
    - bombcar 3y ago
      You need to use the latest and greatest: Awww dang they updated the filter, booo
      - TeMPOraL 3y ago
        You can remove the word "they" from your comment and it'll still work for this site.
    - jakear 3y ago
      In the CIA since of "we need to handle these democratically elected leaders", perhaps. But if I was selling you a drop-in comment widget and boasted "it handles all of Unicode", but really I was just running s/[^ -~]+/ /g, wouldn't you feel a bit let down?
ethbr0 3y ago
Today I learned, that multi-glyph emojis are a thing. Zero width joiner emoji sequences https://emojipedia.org/emoji-zwj-sequence/ https://emojipedia.org/emoji-zwj-sequence/
dang 3y ago
Ok, we've remojied the title above. Edit: there are always exceptions - https://news.ycombinator.com/item?id=34460417 https://news.ycombinator.com/item?id=34460417
- skykooler 3y ago
  Interestingly, Firefox on Wayland renders the emoji correctly in the tab title, but the window title renders it as two rectangles and the male symbol. I assume this must be some difference between system fonts vs Firefox's fonts.
- avgcorrection 3y ago
  dang is definitely wrong about what “plain text” means in that thread.
  - dang 3y ago
    Happy to wrong but when saying something like that, you should explain why and how, so we can all learn something!
    - avgcorrection 3y ago
      Plain text is often contrasted with markup or rich text. UTF-8 text is neither markup nor rich text.

sillysaurusx 3y ago

Measuring the length of text is really hard. Font fallback is hard. All of these things, you take for granted till you write your own game engine. Apparently the thing to use is a library with a very strange name, which does glyph placement. I’ll go look for it. EDIT: harfbuzz https://harfbuzz.github.io/why-do-i-need-a-shaping-engine.html https://harfbuzz.github.io/why-do-i-need-a-shaping-engine.ht...

missblit 3y ago
Measuring the length of text is easy. Now measuring the length of text before actually printing it / running the shaping engine, now that's hard.
grt_thr 3y ago
[dead]

grayhatter 3y ago

> Python 3’s approach is unambiguously the worst one, though. Did I miss the part where he explains this take? It's made up of 5 valid unicode code units. For a language where you're not supposed to need to know the byte size semantics, the correct length should be 5. What am I missing? The close second being 17, because length in bytes. Is another fine way to represent this data, e.g. what a successful write of some sort would look like. Network or file. I guess I'm basing this all on the idea that it's almost always a mistake to confuse how a program manages some data, vs how a drawing lib might. Your language shouldn't concern it self with how many glyphs it needs to draw... until you actually try to draw them.

benatkin 3y ago
I think he meant to bring out defensiveness with that quip. He never says that it's a big deal, just that it's the worst way to get the length of a string containing emoji, presumably of the mainstream languages.
- tehsauce 3y ago
  Why is it the worst way though?
  - deleted 3y ago
    [deleted]
  - imron 3y ago
    > Note about Python 3 added on 2019-09-09: Originally this article claimed that Python 3 guaranteed UTF-32 validity. This was in error. Python 3 guarantees that the units of the string stay within the Unicode code point range but does not guarantee the absence of surrogates. It not only allows unpaired surrogates, which might be explained by wishing to be compatible with the value space of potentially-invalid UTF-16, but Python 3 allows materializing even surrogate pairs, which is a truly bizarre design. The previous conclusions stand with the added conclusion that Python 3 is even more messed up than I thought!
  - markmark 3y ago

paddw 3y ago

I think this is a really a naming convention issue. Len() is ambiguous, you really want either num_chars() or utfxx_len(). Of course, the issue of what counts as a character is confusing in its own right...

arp242 3y ago
In Python len() on a bytes type gives you the number of bytes, and len() on a str type gives you the number of codepoints. I think that makes sense, as strings are only intended to deal with text, and you should never have to worry about byte indexing at all.
- lmm 3y ago
  The argument is that indexing by codepoint is even less useful than indexing by byte.
  - chasontherobot 3y ago
    As someone who has done both, I'd say that argument is wrong. It is much more convenient to index by code point. Indexing by bytes is almost always what you don't want to do, and leads to a lot of errors.
    - lmm 3y ago
      What were the use cases where you found it useful to index by code point (and therefore not by grapheme cluster)?
      - arp242 3y ago
        In many cases it's not very useful, but there are clearly cases where it is, e.g. if you want to normalize text, compose/change emojis, stuff like that. A codepoint is the "smallest useful addressable unit" when dealing with Unicode text, so it makes sense that's the default. It's also comparatively expensive to address grapheme clusters.

geoffpado 3y ago

(2019), for what it's worth.

benatkin 3y ago
In 2023 we should be able to see that Python's isn't the worst, or rather is tied for the worst, and that they're all a little bit tricky but fine.

hgs3 3y ago

Python 3's approach is the most correct: Unicode defines text as a sequence of code points. UTF-whatever is an implementation detail.

adgjlsfhk1 3y ago
Unicode defines text as a number of different types of things. They are sequences of codepoints, sequences of graphemes, sequences of graphime clusters. Furthermore, codepoints are different depending on how you normalize them. Accented characters can be written two different ways and have a different number of codepoints depending on how you write them (and if normalization is used)
- Spivak 3y ago
  Grapheme are a made up human thing that, while useful, is locale dependent. Most people when they talk about grapheme clusters mean the default "locale-independent" graphemes but it's not the only one (in Hungarian for example 'ly' is a single letter). Having the same string be two different lengths in two countries is… let's go with surprising. The common denominator where everyone computes the same number is code points.
  - adgjlsfhk1 3y ago
    Except they won't (if they are doing normalization). à and à have different numbers of codepoints (the first is 0x00E0, the second is 0xc3 0xa0).
pjscott 3y ago
Treating Unicode strings as a sequence of code points is a completely valid thing to do, but is usually not what you actually care about when dealing with text. Really, are code points any less of an implementation detail?
- croes 3y ago
  I think parent means most correct of the three given examples from Rust, JS and Python. Especially because the article says that Python's take is the worst.

ars 3y ago

Am I wrong for assuming the .length should return a length in bytes? If you want to use 32bit units, then multiply your output by 4. If you want to do Unicode string manipulation and length counting, then use specific functions for that - but the base internal .length function should just output bytes.

Arnavion 3y ago
>then multiply your output by 4. That is not how UTF-32 works. >but the base internal .length function should just output bytes. Do you think the length of an `int64_t[3]` array should be 3 or 24?
- ars 3y ago
  > Do you think the length of an `int64_t[3]` array should be 3 or 24? There should be functions to do both: sizeof(int64_t[3]) * sizeof(int64_t) for example to get bytes. In this example, the base function should do bytes, and there should be a unicode function to count it in other ways. I could be sizing to fit in a database, or send over the wire, or I might want visible space on the screen, or I might want to know how to move the cursor. Each of those types of length should be supported.
  - tialaramex 3y ago
    > There should be functions to do both: sizeof(int64_t[3]) * sizeof(int64_t) for example to get bytes. That's not what that does, the answer to the code you wrote is 192 because you're multiplying the same size factor twice.
    - ars 3y ago
      No I'm not, read it again. The first one has size 3, because there are 3 elements in the array. The second one has size 8 because an int64 is 8 bytes.
      - tialaramex 3y ago

deleted 3y ago

[deleted]

2h 3y ago

this one one of those things that people point to when comparing languages, but in reality rarely matters. with Go, you just get the number of bytes, which the the correct default thing to do: https://godocs.io/builtin#len https://godocs.io/builtin#len if the language default was anything other than this, THAT WOULD BE WRONG and unexpected. I would prefer the default to be the dumb, fast thing. then if I want the slow, fancy thing, I can import some first or third party package.

ladberg 3y ago
I think to some extent it depends on the language. In the article they talk about Swift's implementation, which by default does the slow, fancy thing (but makes it easy to do the dumb, fast thing). String manipulation in Swift is almost certainly going to be used for a GUI for end users of many possible languages / locales, so it makes sense to spend the extra cycles to get the fancy version by default. If it isn't the default then you'll end up with half the apps on the App Store displaying broken text on line breaks, ellipses, wrapping, etc. on their hand-rolled UI stack.
eviks 3y ago
Why would you want dumb??? (and it's not expected that a character's length is>1 unless you've been conditioned to excpect it)
- Mawr 3y ago
  Because whenever you want to store or transmit a string only the byte count matters (the size of the string). All the fancy unicode stuff on top of bytes is for the display layers to handle. The default should be grounded to the reality of the programmer.
  - eviks 3y ago
    Human interaction is a more grounded reality for programmers vs. the dumb land of pure bytes, so even at that conceptual level the default should be smart And bytes is the only thing that matter for a specific type of string, conveniently named, sequence of bytes

dahfizz 3y ago

Maybe not wrong, but it's the worst option. 5 is the number of code points, and 17 is the number of bytes. Both are reasonable answers. 7 is the number of code units for utf-16. Seems like the least useful option.

Dylan16807 3y ago
That means 7 is also a measure of bytes, just slightly more awkward. So it's roughly on par with 17. For 5, the idea is that while you might want to iterate code points, the total number of code points is less useful than either grapheme count or byte count. I think that argument makes sense.
- lmm 3y ago
  > That means 7 is also a measure of bytes, just slightly more awkward. It's not a real measure of bytes though. It's the count of bytes in an encoding scheme that is (probably) neither what you use to communicate with the outside world nor what your language runtime uses. (And certainly it's no better than 5, since that's also a measure of bytes in a particular encoding).
  - MobiusHorizons 3y ago
    The JavaScript language forces utf16 (whether or not v8 uses that representation under the hood). For instance if you want to substring the indexes you pass are for utf16 codepoints
    - lmm 3y ago
      Sure, but arguing that that's a good reason for length to count utf16 is purely circular.
  - Dylan16807 3y ago
    Lots of systems use UTF-16 internally and externally. Counting bytes in UTF-16 is, on average, almost as useful as counting bytes in UTF-8. I don't think just about anything communicates in UTF-32. 5 is basically just a codepoint count, and as such I don't think its usefulness rating should be between the byte counts.

bitwize 3y ago

Once again, strings are not simple sequences of characters. It's also useless to "index" into a string without specifying what you're indexing for the same reason.

astrea 3y ago
This is like when you first dive into date-time and slowly unravel the maddening complexity of it all.

xeeeeeeeeeeenu 3y ago

I'm not a fan of "everything you know about X is wrong" articles. Very often they try to present some little tidbit of knowledge as a revelation and mislead the reader in the process. In this case, the tidbit is: "grapheme clusters exist and they are useful". The misleading part is that the article draws a false equivalence between what the author calls "UTF-32 code units" and UTF-16 code units. UTF-32 code units are Unicode code points. This is a general Unicode concept that exists in all Unicode encodings. UTF-16 code units, on the other hand, are an implementation detail of UTF-16. It is wrong to present them as equally arbitrary concepts.

masklinn 3y ago
> UTF-32 code units are Unicode code points. They’re not. UTF-32 code units have a 1:1 mapping to USVs, surrogates are not valid. > It is wrong to present them as equally arbitrary concepts. Is it? It’s not like they’re any more useful. Arguably less so, UTF-16 is at least a somewhat common storage medium.
- xeeeeeeeeeeenu 3y ago
  >They’re not. UTF-32 code units have a 1:1 mapping to USVs, surrogates are not valid. This is true, although very pedantic and irrelevant to the point of my comment. The distinction only matters when you're dealing with ill-formed strings. BTW, Python strings can store surrogates. >Is it? It’s not like they’re any more useful. Arguably less so, UTF-16 is at least a somewhat common storage medium. If you aren't directly dealing with UTF-16, UTF-16 code units aren't useful at all. Code points/USVs, OTOH, are the building blocks of Unicode strings and various Unicode algorithms operate on them. They're low-level, but not useless.
chrismorgan 3y ago
> UTF-16 code units, on the other hand, are an implementation detail of UTF-16. Would that it were only so. Instead, UTF-16 ruined Unicode for everyone with the abomination that is surrogates, and almost nothing that deals with UTF-16 actually asserts well-formedness, and ill-formed UTF-16 cannot be represented in UTF-8 or UTF-32. UTF-32 and UTF-8 code units are truly implementation details of their encodings, as other encodings don’t need to know about them in any way. UTF-32’s code units are a trivial mapping between scalar values and 32-bit values (not four-byte values, given the big- and little-endian variants), but that still causes UTF-32 code units to be semantically distinct from Unicode scalar values. U+12345 is a Unicode scalar value and doesn’t have any “size”: it’s an abstract value. 0x00012345 is a UTF-32 code unit, a 32-bit value. If you’re talking about encoding of Unicode scalar values, you talk about code units. Even when talking about UTF-32, the code unit/scalar value semantic distinction is worth maintaining.

duxup 3y ago

Is there a situation where I am going to try to get the length of an emoji and I care about the outcome?

justinator 3y ago
There's most definitely a situation where you want the length of string that contains an emoji character and perhaps dumb things happen if you get that wrong.
nosefurhairdo 3y ago
Enforcing a character limit on user input.
- hermitdev 3y ago
  Now I want to know: how much does this emoji count towards password strength?

jlebar 3y ago

I was 100% prepared to believe that the length of empty string in js is 7. Then upon opening the post I was 100% ready to believe that js has three different string length functions that all handle Unicode differently.

baq 3y ago
and the gist of the problem is that it doesn't.

secret-noun 3y ago

Related: I wrote a little web app that lets you see the codepoints for text like this https://unicode-x-ray.com/?t=%F0%9F%A4%A6%F0%9F%8F%BC%E2%80%8D%E2%99%82%EF%B8%8F https://unicode-x-ray.com/?t=%F0%9F%A4%A6%F0%9F%8F%BC%E2%80%... (sorry if link looks scary, that's just the URL encoding of this emoji)

farhanhubble 3y ago
I worked on Unicode parsers for several years and the single most indispensable tool was https://r12a.github.io/app-conversion/ https://r12a.github.io/app-conversion/

eviks 3y ago

Very good and informative article, though still not convincing that the nudge to make the shortest "len" command use the human readable size of grapheme clusters like in Swift isn't the best design approach, all the non-intuitive sizes should be special

pavlov 3y ago
The article shows that the Swift approach produces different values for length depending on operating system and text library versions. Is that really intuitive?
- eviks 3y ago
  Showing unintuitive results in a small number of cases always beats showing unintuitive results for all the cases. The Swift approach can also asymptotically reach perfection. The other approach will be broken forever. These are categorical improvements in design
  - pavlov 3y ago
    The Swift approach can't reach perfection in isolation because data from the future can always break it. That's why in the article you see Swift running on Ubuntu 14.04 returning len==2 while the same code on Ubuntu 18.04 returns len==1 for the same emoji string. IMO that's a big philosophical question here: do we accept that "string length" means something you can't compute for arbitrary strings unless your code is receiving annual updates containing the latest Unicode interpretation instructions?
    - stephencanon 3y ago
      Swift includes its own Unicode data tables with the standard library since last year, so it’s now tied to the stdlib version rather than some other library that may or may not be updated on the system.
    - eviks 3y ago
      Your example shows an improvement, which proves my point (also don't drop the word asymptotically, nothing can ever be perfect, that's not the issue, being closer to perfect is a positive) And you can compute it, you can pin a Unicode version and ship it in the language if those platform differences are unbearable (so, you can actually isolate it and simply ignore the future :)) The bigger philosophical question: how much longer do we accept that "string length" does not measure the most intuitive measure of string length and call a byte a char?

omoikane 3y ago

Ruby gives you the choice to iterate over all types, via `each_byte`, `each_char`, `each_codepoint`, or `each_grapheme_cluster`. https://ruby-doc.org/3.2.2/String.html#class-String-label-Methods+for+Iterating https://ruby-doc.org/3.2.2/String.html#class-String-label-Me...

h0l0cube 3y ago
Elixir also makes this distinction: https://hexdocs.pm/elixir/1.12/String.html#module-code-points-and-grapheme-cluster https://hexdocs.pm/elixir/1.12/String.html#module-code-point...
nerdponx 3y ago
Python would benefit from this significantly IMO.
sundarurfriend 3y ago
In Julia, iterating over a string by default behaves like `each_char`. `codeunits(str)` lets you access the underlying code units, which is bytes for the default UTF-8 encoding. (External packages implement UTF-16 and others, and there `codeunits` could return non-bytes, for eg. 16-bit values for UTF-16.) The Unicode stdlib provides `graphemes(str)`, the equivalent of `each_grapheme_cluster`.

chefandy 3y ago

It's 1 in elixir which measures graphemes by default. iex(3)> String.length(" ") 1 Edit: looks like HN doesn't support that emoji in code blocks, at least.

jnsaff2 3y ago
it's also 5 if you like: > length(' ') Which is: [129318, 127996, 8205, 9794, 65039]

justinator 3y ago

perl -e 'use utf8; print length(""). "\n";' 1

justinator 3y ago
Should be the emoji char between the double q's in, 'length("")' but whatcha gonna do.

ceeam 3y ago

These emoticons should never have been a part of Unicode in the first place. Second big mistake of that org after the Unihan fiasco.

themerone 3y ago
Emoji's are a fun way to illustrate a problem that happens with languages too.
dchest 3y ago
If not for emoji, lots of software wouldn't care about correctly processing strings in other languages, so it's good that we have them.
- hnfong 3y ago
  This. I've been quite happy that popular emojis were introduced in supplementary planes, because my language has quite a few common words (eg. 𨋢 [lift/escalator]) that ended up on plane 2. Proper software support for those characters used to be terrible, but things got much better after emojis became popular. So, thanks and sorry everyone :)
Findecanor 3y ago
I agree that they introduced unnecessary complexity for text encoding, and for font-rendering (which are expected to support multi-coloured emoticons now). I once started writing on a text editor, and then fell deep into Unicode handling. I have now spent more work on the Unicode parts than on anything else in the program. I think that the industry could have instead adopted the old web-forum convention of colon-word-encoding, originating from ASCII art. Example: ":facepalm:". When the sequence is not supported as an emoji, it degrades gracefully into text that can be understood by anyone reading it instead of into a sequence of empty squares or diamonds with question marks in them. Text also provides a more efficient input method than having to browse for an icon in a list.
- kps 3y ago
  Naturally Unicode did the equivalent of colon-encoding too, for flags.

deleted 3y ago

[deleted]

neallindsay 3y ago

Until reading this I had never heard of UTF-32. It doesn't seem like a good way to encode strings.

nerdponx 3y ago
It's useful if you want array-like semantics (e.g. O(1) lookup) on Unicode text strings, because you have a fixed size for every codepoint, unliked UTF-8. Python for example uses it internally.
- Joker_vD 3y ago
  And it compresses just as well as UTF-8 for transfer/storage purposes.
- chrismorgan 3y ago
  Except code point indexing simply isn’t useful. In the words of the article: “The choice of UTF-32 (or Python 3-style code point sequences) arises from wanting the wrong thing.”
  - nerdponx 3y ago
    I think that's the issue here. People disagree on how useful or not useful it is. It's maybe not ideal, but I don't think it's anywhere near so bad as to be entirely not useful. Strings-are-sequences-of-bytes is worse in my opinion. Python literally used to have that. It was worse.
    - chrismorgan 3y ago
      The problem with what Python used to have is that the encoding wasn’t fixed. I’ll agree with you that strings-are-sequences-of-bytes is bad. That’s painful compiler-flag, codepage, &c. territory. But what’s not bad is strings-are-sequences-of-code-units. That’s what Rust has, for example. Rust strings aren’t sequences of bytes, but of UTF-8 code units, and the two are semantically very different.

foxes 3y ago

All these abominations are because of non strict typing String = List ( Char ) Chars don’t have a length, like a number doesn’t have a length - unless you talk about number of bits. If you are working with strings stick with strings. The string of a single character should be “1”. Just enforce proper typing. Anything else is not consistent.

chasontherobot 3y ago
Python has strong typing which seems to be what you mean here rather than strict typing. A "character" is not a well defined term in Unicode, rather the "base" that does not vary across implementations is code points, which is what Python measures when you get the length of a string.
jillesvangurp 3y ago
No, it's caused by cost. For example Java has a char type. It's a 16 bit numeric value because Java uses UTF-16 internally for encoding strings. Java Strings are basically immutable char arrays with some fluff around them. If you ask for the String length, it returns the length of the underlying array. Nice and simple and unsurprising. And relatively cheap. Most more recent languages use 8 bit bytes and UTF-8 instead because that is emerged as the most common character encoding. But UTF-16 was a reasonable choice a quarter century ago and the practical difference doesn't matter that much and changing it would be disruptive. If you put unicode characters consisting of multiple data points into a String, it necessarily increases the amount of chars. There's no way around that. Because there is no such thing as a UnicodeChar type in Java. You can't actually assign multi data point unicode characters to a char. Essentially all the workarounds for a 'correct' unicode character count in a String would either end up using a different and probably way more expensive data structure (e.g. a list of a list of chars or bytes where each list is a unicode character) or implementing some expensive logic for counting characters that is O(n) instead O(1). Most languages ranging from extremely strictly typed to weakly typed don't do that for cost reasons. The tradeoff is simply not worth the price it takes. This stackoverflow post provides a few suggestions for how you could count 'correctly'. https://stackoverflow.com/questions/15947992/java-unicode-string-length https://stackoverflow.com/questions/15947992/java-unicode-st... that illustrates the point nicely.

Jach 3y ago

Unsurprising that (at least some implementation of) Swift does the least wrong thing in returning 1. I think it's also one of the few languages that will return a count of 1 for the madness that is country flag emojis https://docs.swift.org/swift-book/documentation/the-swift-programming-language/stringsandcharacters#Extended-Grapheme-Clusters https://docs.swift.org/swift-book/documentation/the-swift-pr...

ant6n 3y ago
“Least wrong” sounds very silly. Its like programmers are discovering theres a difference between bytes, unicode code points and grapheme clusters and are unsure about how their favorite programming language represents strings, and then decide there should be some behavior that doesnt follow from the documentation. The “length of an emoji” depends on the data type used to represent it. Its that simple and that correct.
baq 3y ago
'least wrong length method' for strings is not providing one in the first place.

deleted 3y ago

[deleted]

quaintdev 3y ago

How the hell you inserted an emoji in title? Afaik we cant use emoji on this website

Arnavion 3y ago
https://news.ycombinator.com/item?id=36160029 https://news.ycombinator.com/item?id=36160029
deleted 3y ago
[deleted]

farhanhubble 3y ago

Why should length in programming be devoid of units? Why can't we have length be (8, UTF32_CODEPOINTS) <class UTF32_CODEPOINT_SZ(8)>?

mrkeen 3y ago
We can and do - they're called types. However it's always going to be up to the library developers as to whether two things are the same type.

frou_dh 3y ago

I encountered some real world unicode/emoji breakdown recently. I set my surname in a webapp to an emoji country flag because I needed a way to communicate where I was. Elsewhere in the app, it showed surnames as just their initial, e.g. "John S". There, mine showed as a featureless black flag rather than the flag I set. Presumably because that is the first codepoint of several that make up the flag.

tialaramex 3y ago
> There, mine showed as a featureless black flag rather than the flag I set. Presumably because that is the first codepoint of several that make up the flag. The country flags are each made of two Unicode code points, which Unicode calls Regional Indicator Symbols. There are twenty six, one for each of the Latin capital letters A through Z. These are used to encode a flag by writing the ISO two letter country code from ISO-3166-1 e.g. F + R is France, you get a French flag. Given your black flag experience, and the fact this is an English language forum, I'd guess maybe you wanted a flag for some entity that isn't a UN member state or some sort of recognised similar entity (e.g. the European flag EU symbolising the continent of Europe) and thus doesn't have an ISO two letter code, such as California or Wales. Those are built from a waving black flag plus their long ISO-3166-2 region code

namaria 3y ago

I have read somewhere that you should learn 2 or 3 programming languages from the get go. If you learn one, you run the risk of letting it's shape dictate how you mentally model computation. At some point someone who learned a dynamically typed programming language first is bound to find out why data types matter.

Firmwarrior 3y ago
I started with JavaScript and went to c/logic gates/assembly, and it wasn't too bad I had to learn what pointers were, but it was OK, haha. That said, maybe people who don't have "The Knack" would be better off learning a slightly harder language first..
- namaria 3y ago
  I definitely don't have talent for logic and quantitative thinking. It takes a long time and many iterations for even simple concepts in mathematics to sink in for me. I benefited greatly from learning first Scheme and also making sense of C and OS internals before trying to grok interpreted languages. I'm currently trying to get some proficiency in Go and it's been great fun!
preommr 3y ago
> should learn 2 or 3 programming languages from the get go. So... Java/C#/Kotlin?
omoikane 3y ago
I had a "programming languages" class that did that, where we did assignments in Python (scripting), OCaml (functional), and Prolog (logic). This is because most other classes used compiled imperative languages such as C++ and Java.

alkonaut 3y ago

Really the correct way to design string APIs would be to not have an ambiguous "length" at all, but to always require specifying whether you want UTF8-bytes, memory bytes, code points, graphemes, whatever. However such an API would be pretty cumbersome because for all non-edge cases (read: a western language and a reasonable encoding that language - which when looking at world demographics is a very narrow way of saying non-edge case) we just want to ignore all that fancy stuff and assume it's latin-1/ascii and use "Length" and get on with it, usually accepting that it doesn't work for many scripts or emoji. So almost every api I have encountered has both the dangerous or ambiguous "length" and any number of the more specific counts. Good? No. But good enough, I guess. A much worse related API that exists every where is that for parsing and formatting numbers to and from text. How that's done "depends" but most languages I have seen - unfortunately - offers a "default way". In the worst examples - looking at you .NET - this default uses the system env and assumes formatting and parsing numbers should us the OS locale. Horrible horrible idea when used in conjunction with automatic type conversions. WriteLine($"The size is {3.5}"); shouldn't print "3.5" in the US and "3,5" somewhere else.

hardware2win 3y ago
>Horrible horrible idea when used in conjunction with automatic type conversions. WriteLine($"The size is {3.5}"); shouldn't print "3.5" in the US and "3,5" somewhere else. Actually why?
- alkonaut 3y ago
  Because it’s only (maybe) a good design if it’s to be read by a human but that’s not a very general case. Instead people unknowingly make for example some exporter for a text format and write code that writes "X={x_coord}" and it passes all the unit tests and all the acceptance tests and then it breaks once it hits a French or scandinavian machine. A great example how bad it is would be that the C# compiler repo for a very long time had tests that failed for everyone with non-US formatting.
ezfe 3y ago
Swift defaults to grapheme clusters, and allows you to request other lengths on-demand.

kolibril13 3y ago

In python, `print("-".join(f"{ord(c):x}" for c in emoji))` will give `'1f926-1f3fc-200d-2642-fe0f'`. That can be useful to request emojis, see https://github.com/hfg-gmuend/openmoji/blob/ad588c8fb4b028d71c5743347ef5dc6968b97579/FAQ.md?plain=1#L141-L157 https://github.com/hfg-gmuend/openmoji/blob/ad588c8fb4b028d7...

PhilipRoman 3y ago

I cannot think of a single common case where grapheme cluster count is important. If you want to print them aligned to a terminal - guess what, double width characters exist, so the only reliable way is to print them first, measure the cursor movement using escape sequences, calculate length and erase the originally printed data. Even for limiting input field sizes byte count is much better, as otherwise you are opening up yourself for unicode denial of service. I think the game Minecraft has such an exploit where you can fit in absurd amounts of utf-8 data (to the point of data corruption in multiplayer games) since it's limited by visual length. My personal favorite dealing with UTF-8: pretend it's ascii and assume everything above 128 is an alphabetic character. It just works. For 99% of use cases it doesn't matter if the content is emojis, families of emojis, or ancient sumerian scripts. You can parse JSON and most other formats this way without caring about code points at all. The trend of unicodizing everything was a mistake, just treat strings as bytes and parse them as utf-8 only when you really need it (like when building a text editor or a browser engine from scratch).

zkldi 3y ago
this doesn't "just work" at all? utf-8 is a variable width encoding and if you treat it like ascii your software just isn't going to work outside of the ascii range.
- PhilipRoman 3y ago
  But it does - the genius of utf-8 is that it was deliberately designed to be backwards compatible (it even preserves the ascii sorting order). You can run C programs written before utf-8 was invented with utf-8 inputs (unlike with the abomination that is utf-16). If a code point is outside the ascii range (0-127 inclusive), then it's utf-8 encoding is also guaranteed to not contain any ascii bytes. So as long as you treat anything inside 128-255 as "some unknown character", the utf-8 code points will be preserved and eventually displayed when the byte sequence is parsed as utf-8 by your terminal/browser/whatever.
vlmutolo 3y ago
I think tweets are restricted by number of extended grapheme clusters. And probably also by number of bytes.

Obscurity4340 3y ago

Can anyone comment as to whether there are any problems associated with using emojiis to enhance the entropy of passwords? For passwords you only need to autofill but never actually type, I feel like it would be an easy way to augment passwords but I don't know whether it would directly translate in every situation.

tylergetsay 3y ago

Interesting; emojis also make SMS really weird. according to twilio including a emoji will change the character limit from 160 to 70

dpiers 3y ago
That’s because SMS has a 1120 bit message capacity and uses one of two encoding mechanisms. GSM-7 uses 7 bits per character and a correspondingly limited character set that allows for 1120/7 = 160 characters. UCS-2/UTF-16 uses 16 bits per character for a total of 1120/16 = 70 characters per message.
locallost 3y ago
It's not weird because it's not a 160 limit in the sense of characters. Non-English language people are I guess more aware of this -- using a character not in the standard English a-z often counts as two characters.

julik 3y ago

So many of these conversations could be easier if there would not be `length()` functions but `length_in_<whats_exactly>()` functions instead.

speleding 3y ago
In ruby you have " ".codepoints.size == 5 and " ".bytes.size == 17 (It also has `length` which equals codepoints.size)
- Freaky 3y ago
  .each_codepoint.size is more efficient than .codepoints.size, as it creates a sized Enumerator that avoids needing to build an intermediate Array. For strings with only single-byte characters it reduces to returning the already-stored stored byte length. Same goes for .each_byte.size, but for that you have the faster .bytesize method that avoids the intermediate Enumerator.
- runarberg 3y ago
  JavaScript is a weird one. To count UTF-16 bytes you write: " ".length For unicode character count you write: [..." "].length And for grapheme count (or language aware word/sentence count) you write: [...new Intl.Segmenter('en-US', { granularity: "grapheme" }).segment(" ")].length For word/sentence count you swap out the granularity option.
  - chrismorgan 3y ago
    > [..." "].length Mind you, this is inefficient due to unnecessarily constructing an array. Here’s a more efficient version, though the difference will normally be fairly slight: function codePointLength(str) { let len = 0; for (const c of str) { len++; } return len; } Kinda sad there are no equivalents to the Array methods that work on iterators. Array.prototype.reduce.call(str[Symbol.iterator](), (a, _) => a + 1, 0) doesn’t work since those methods only work on array-like types (meaning those with a length property and indexed by number—and yes, all these Array methods are explicitly defined that way deliberately so you can use them on other array-like types), not iterators. > [...new Intl.Segmenter('en-US', { granularity: "grapheme" }).segment(" ")].length Caution: Intl.Segmenter may not be available, so be sure to have a fallback if you want to use it. Chromium shipped it 2½ years ago, Safari 2 years ago, and Firefox hasn’t shipped it yet. (No idea why and I haven’t looked. It’s not always the case: I know of other Intl things that Firefox has shipped first.)

asimpletune 3y ago

What does swift or rust return?

twodave 3y ago

Obligatory: https://www.joelonsoftware.com/2003/10/08/the-absolute-minimum-every-software-developer-absolutely-positively-must-know-about-unicode-and-character-sets-no-excuses/ https://www.joelonsoftware.com/2003/10/08/the-absolute-minim...

chrismorgan 3y ago
I was never much impressed with that article (too much irrelevant story for an “absolute minimum”), and by now it’s very dated. A lot of what it’s talking about (most notably code pages) is now completely irrelevant to the vast majority of developers, who might never encounter or need to worry about them in their entire careers.

hateful 3y ago

The wife and I have a Google sheet that we use for our shared calendar - and we put an emoji before each "event" and in top row of each day I show the Emoji for that days entries. But I need to do: > =LEFT(F280,2) & LEFT(F281,2) & LEFT(F282,2) & LEFT(F283,2) Since the Emojis are actually 2 bytes.

jameshart 3y ago
And as this article indicates, many - and indeed an increasing number - of emojis are many more than 2 bytes. Flags, skin tones, groups, professions…

dveeden2 3y ago

MySQL has two different length functions... mysql> WITH chars AS (SELECT ' ' c) -> SELECT LENGTH(c), CHAR_LENGTH(c) FROM chars; +-----------+----------------+ | LENGTH(c) | CHAR_LENGTH(c) | +-----------+----------------+ | 17 | 5 | +-----------+----------------+ 1 row in set (0.01 sec) Note that the doesn't seem to render in preformatted text on HN. This should be easier to reproduce: mysql> WITH chars AS (SELECT 0xF09FA4A6F09F8FBCE2808DE29982EFB88F c) -> SELECT CONVERT(c USING utf8mb4), LENGTH(c), CHAR_LENGTH(c) FROM chars; +--------------------------+-----------+----------------+ | CONVERT(c USING utf8mb4) | LENGTH(c) | CHAR_LENGTH(c) | +--------------------------+-----------+----------------+ | | 17 | 17 | +--------------------------+-----------+----------------+ 1 row in set (0.00 sec)

rzwitserloot 3y ago

Java loaded full unicode code point semantics into its standard `java.lang.String` class. These _are not guaranteed_ to have `O(1)` performance characteristics, because the underlying storage format is dynamically either a UTF-16-esque variant (with surrogate pairs for characters that don't fit in 16 bit), or a single-byte-per-char format if the string does not contain any non-ASCII. This has the advantage of being very very slightly more obvious, given that both methods exist and are documented: void main() { String x = "(that emoji here)"; System.out.println("Chars: " + x.length()); System.out.println("Codepoints: " + x.codePointCount(0, x.length())); System.out.println("As stream of chars (= UTF16-esque with surrogate pairs):"); x.chars().forEach(System.out::println); System.out.println("As a stream of codepoints:"); x.codePoints().forEach(System.out::println); } This ends up printing: Chars: 7 Codepoints: 5 As stream of chars (= UTF16-esque with surrogate pairs): 55358 56614 55356 57340 8205 9794 65039 As a stream of codepoints: 129318 127996 8205 9794 65039 NB: Apparently many hackernews readers know java but don't use it all that often day-to-day. The provided java snippet is vanilla valid and can be executed with `java ThatFile.java` (no need to compile it first), though it does use preview features. The fact that the codepoint counter is a very awkward `codePointCount` call has the dubious benefit of highlighting this method loops through and therefore would be quite slow on very large strings.

ElectricalUnion 3y ago
> The provided java snippet is vanilla valid Did not you still need the `java --source 11 ${filename_without_java_extension_because_JEP_330}` to use it? And you still need a wrapper class with a static method main in it.
- JZerf 3y ago
  I was a little puzzled by this compared to what I was used to with Java in the past. It looks like the grandparent's code relies on JEP 445 ( https://openjdk.org/jeps/445 https://openjdk.org/jeps/445 ) which is a preview feature as was mentioned but it also apparently requires the very latest Java 21 which hasn't even been officially released yet.
- 3y ago

JohnFen 3y ago

It's things like this that make me hate unicode so much.

iandanforth 3y ago

Life imitates Hitchhikers Guide to the Galaxy. Sure the answer is 7 but do you know what the question is?

joshspankit 3y ago

My RSS reader showed the title as “It’s not wrong that “ “.length == 7” and I had to click on it

11235813213455 3y ago

[..." "].length === 5 // in JS

rurban 3y ago

Naming, folks. Length = 5 Size: depends on the encoding Width = 1

eternityforest 3y ago

1 seems like the only acceptable answer. For all intents and purposes, a user will count it as one character. Truncating the string without including the whole cluster would change the meaning of it, and is not an operation anyone would do as a general purpose thing any more than someone would want to randomly replace the last character with random letters. It looks like one character. I'd rather APIs let us continue pretending it is one character.