10 ms·

String length functions for single emoji characters evaluate to greater than 1

116 points by kevincox 6y ago

Techyrack 6y ago
#cryptocurrency #bitcoin #crypto
dooglius 6y ago
I think this title needs an exemption from HN's no-emoji rule
- kyberias 6y ago
  Why is there such a rule?
  - colejohnson66 6y ago
    Emojis aren’t professional (depending on who you ask) and can be overused (look at a few popular GitHub projects’ READMEs - emojis on every bullet point of the feature list, for example)
    - tester34 6y ago
      I agree that they aren't the most professional thing to use, but I'm not sure why I agree Maybe it's some kind of bias?
    - fhifjfhjjjk 6y ago
      emojis aren't "professional" is your reasoning? is it 1987? should women also wear their skirts below the knee?
      - colejohnson66 6y ago
        I didn’t say that they weren’t. I’m simply saying that some do think that. Whether I do or don’t is irrelevant as I’m only speculating on the reason why.
    - Kaze404 6y ago
      What a weird reasoning. In most contexts I've seen, "professionalism" doesn't really do anything except strip away all the human factor that goes into every interaction. Personally, I don't care whether the person I'm talking to is being "professional" or not. What I care is that they're respectful and can properly communicate their thoughts. With that mindset, emojis (and emoticon) can actually add context to interactions, considering how much context is lost when communicating over text. A simple smiley face at the end of a sentence can go a long way in my experience :)
  - at_a_remove 6y ago
    Also, looking at the article, they are quite complex! It looks like handling emojis in a proper manner has a large investment and the payoff is somewhat small.
    - wongarsu 6y ago
      Emoji don't introduce anything that isn't used for various other languages. Emojis are just the most visible breakage for American users when you screw up your unicode handling. HN however handles all of unicode just fine, it just chose to specifically exclude emojis (and a bunch of other symbols)
      - MayeulC 6y ago
        I think emoji is wonderful for widespread unicode support. It combats ossification. It is also a nice carrot to incite users to install updates. However, I don't like the rabbit hole it started to go into with gendered and colored emojis. There's never going to be enough. I wish we had stuck to color-neutral and gender-neutral, like original smileys :) I find it also conveys too much meaning. I am generally not interested in knowing a person's gender or ethnic group when discussing over text... but I digress.
        samatman 6y ago
        I've said this before, but not here: Eventually, Unicode will allow you to combine flag modifiers with human emojis. So you can have a black South Korean man facepalming. This will trigger a war which ends industrial civilization, and is the leading candidate for the Great Filter.
    - Someone 6y ago
      For HN I would think almost all the complexity is in rendering. That’s a job for your browser. What’s left is things like the max length for a title (not too problematic to count that in code points or bytes, I think) The big risk, I think, is in users (mis)using rendering direction, multiple levels of accents, zero-width characters and weird code points to mess up the resulting page. Some Unicode symbols and emojis typically look a lot larger and heavier than ‘normal’ characters, switching writing direction may change visual indentation of a comment, etc. Worse, Unicode rendering being hard, text rendering bugs that crash browsers or even entire devices are discovered every now and then. If somebody entered one in a HN message, that could crash millions of devices.
  - TeMPOraL 6y ago
    Surprisingly many people try to abuse Unicode when posting submissions and comments. This includes not just emojis, but also other symbols, or glyphs that look like bold, underlined or otherwise stylized letters.
    - criddell 6y ago
      𝕴 𝖉𝖔𝖓'𝖙 𝖙𝖍𝖎𝖓𝖐 𝖙𝖍𝖊𝖞'𝖛𝖊 𝖇𝖆𝖓𝖓𝖊𝖉 𝖎𝖙 𝖊𝖓𝖙𝖎𝖗𝖊𝖑𝖞.
      - masklinn 6y ago
        HN's character stripping is completely arbitrary. You can play mahjong, domino or cards (🀕, 🁓, 🃅) but not chess, you can show some random glyphs like ⏱ or ⤪ but not others, you can use box-drawing characters (┐) but not block elements, you can use byzantine musical notation (𝈙) but only western musical symbols (𝅗𝅥) and the notes are banned, you can Z̸̠̽a̷͍̟̱͔͛͘̚ĺ̸͎̌̄̌g̷͓͈̗̓͌̏̉o̴̢̺̹̕ just fine.
    - masklinn 6y ago
      > Surprisingly many people try to abuse Unicode when posting submissions and comments. ˙ǝsnqɐ ƃuıpıoʌɐ ʇɐ qoɾ pooƃ ɐ ɥɔns sǝop sɹǝʇɔɐɹɐɥɔ ɟo ɥɔunq ɐ ƃuıuuɐq ʎʃıɹɐɹʇıqɹɐ ǝsnɐɔǝ𐐒
  - pessimizer 6y ago
    This is a forum sponsored by opinionated people who find them annoying. I find them annoying, too, so I think it's a good convention. Somehow ":)" became an industry, and as industries go, then eliminated ":)" (which gets replaced with a smiling yellow head in most places.) It isn't the only arbitrary convention enforced here, and the sum of those conventions are what attracts the audience.
  - saagarjha 6y ago
    Emoji draw attention to themselves in a way that plain text does not. I assume this is why Hacker News does not allow bolding text and strongly discourages use of ALL CAPS.
- colejohnson66 6y ago
  I’m curious: is the no-emoji rule a rule that happens to block emoji or a hardcoded rule? What I mean is: emojis (in UTF-16) have to use surrogate pairs because all their code points are in the U+1xxxx plane. Is the software just disallowing any characters needing two code points to encode (which would include emoji)? Or is it specifically singling out the emoji blocks?
  - kps 6y ago
    It seems to have changed recently. I recall a thread about plastics a few months ago where the plastic type symbols (♳ through ♹, i.e. U+2763 through U+2769) disappeared, but I see them now. Edits: ⭕ that post was https://news.ycombinator.com/item?id=25237688 https://news.ycombinator.com/item?id=25237688 ⭕ The rules seem pretty arbitrary. Recycling symbol U+2672 ♲ is allowed but recycled paper symbol U+267C is not. Chess kings ♔ are allowed but checkers kings aren't. ⭕ is allowed (for now). ⭕ I think the right thing to do would be to strip anything with emoji presentation http://unicode.org/reports/tr51/#Presentation_Style http://unicode.org/reports/tr51/#Presentation_Style
    - saagarjha 6y ago
      It’s not that simple, as many OSes render characters that aren’t strictly look emoji “as emoji” and there is no standard way to check for this.
  - wongarsu 6y ago
    It's a very deliberate and precise filter. I can write in old persian: "𐎠𐎡𐎢" (codepoints 0x103xx) or CJK "𠀀𠀁𠀂" (codepoints 0x2000x), but can't write "" (emoji with codepoint 0x1F605)
  - saagarjha 6y ago
    The latter.
colejohnson66 6y ago
I was wondering what the title meant. Turns out HN’s emoji stripper screwed with the title. It’s asking why a skin toned (not yellow) facepalm emoji’s length is 7 when the user perceives it as a single character. Tangent: Emojis are an interesting topic in regards to programming. They challenged the “rule” held by programmers that every character is a single codepoint (of 8 or 16 bits). So, str[1234] would get me the 1235th character, but it’s actually the 1235th byte. UTF-8 threw a wrench in that, but many programmers went along ignoring reality. Sadly, preexisting languages such as Arabic weren’t warning enough in regards to line breaking. As in: an Arabic “character”[a] can change its width depending on if there’s a “character” before or after it (which gives it its cursive-like look). So, a naive line breaking routine could cause bugs if it tried to break in the middle of a word. Tom Scott has a nice video on it that was filmed when the “effective power” crash was going around.[0] [0]: https://youtu.be/hJLMSllzoLA https://youtu.be/hJLMSllzoLA [a]: Arabic script isn’t technically an alphabet like Latin characters are. It’s either an abugida or abjad (depending on who you ask). See Wikipedia: https://en.wikipedia.org/wiki/Arabic_script https://en.wikipedia.org/wiki/Arabic_script
- MengerSponge 6y ago
  Interesting. Also, many of us use fonts with ligatures, which render as a single character (for example: tt, ti, ff, Th, ffi) Of course, we're taught to parse that as multiple discrete letters from an early age, so we don't get confused :)
sanedigital 6y ago
This may be the best tech writeup I've ever seen. Super in-depth, easily readable. Really well done.
- donaldihunter 6y ago
  With clickbaity misrepresentations of correctness, unfortunately.
SamBam 6y ago
After having read (quite interesting) the article, I still don't quite get the subtitle: > 'But It’s Better that "[emoji]".len() == 17 and Rather Useless that len("[emoji]") == 5' It sounds like it's just whether you're counting UTF-8/-16/-32 units. Does the article explain why one is worse and one is "rather useless?"
- dahfizz 6y ago
  I agree. The author talks a little bit about which UTF encoding makes sense in which situation, but they never make an argument about which result from len is correct. My two cents is that string length should always be the number of Unicode codepoints in the string, regardless of encoding. If you want the byte length, I'm sure there is a sizeof equivalent for your language. When we call len() on an array, we want the number of objects in the array. When we iterate over an array, we want to deal with one object from the array at a time. We don't care how big an object is. A large object shouldn't count for more than 1 when calculating the array length. Similarly, a unicode codepoint is the fundamental object in a string. The byte-size of a codepoint does not affect the length of the string. It makes no sense to iterate over each byte in a unicode string, because a byte on its own is completely meaningless. len() should be the number of objects we can iterate over, just like in arrays.
  - MayeulC 6y ago
    But what about combining characters? https://en.wikipedia.org/wiki/Combining_Diacritical_Marks https://en.wikipedia.org/wiki/Combining_Diacritical_Marks Should the letter plus a combining character count as one (I think so), or two characters? Should you normalize before counting length? And so on.
    - dahfizz 6y ago
      Combining characters are their own unicode codepoint, so they count towards length. The beauty of this approach is that its simple and objective. If you had a list of 5 Dom Element objects, and one Dom Attr object, the length of that list is 6. Its nonsensical to say "The Attr object modifies an Element object, so its not really in the list".
beaconstudios 6y ago
perhaps the issue is that there's a canonical "length" at all. It would make more sense to me to have different types of length depending on which measure you're after, like Swift apparently has but without the canonical `.count`. Because when there's multiple interpretations of a thing's length, when you ask for "length" you're leaving the developers to resolve the ambiguity and I'm of the firm belief that developers shouldn't consider themselves psychic.
- anaerobicover 6y ago
  The main reason, I think, that Swift strings have `count` is that they conform to the `Collection` protocol. Swift's stdlib has a pervasive "generic programming" philosophy, enabled by various protocol heirarchies. So, given that the property is required to be present, some semantic or the other had to be chosen. I am sure there were debates when they were writing `String` about which one was proper.
  - beaconstudios 6y ago
    that's very cringe of them. > So, given that the property is required to be present, some semantic or the other had to be chosen. Sounds like an invented solution to an invented problem. The programmer's speciality.
    - samatman 6y ago
      On the contrary it is based! This conforms exactly to our intuition about what a "collection" is. Some (exact) number of items which share some meaningful property in common such that we can consider them "the same" for purposes of enumeration and iteration. In the real world, we also have to decide what our collection is a collection of! Let's say we a pallet of candy bars, each of which is in a display box. If we want to ask "how many X" are on the pallet, we have to decide whether we're asking about the candy bars or the boxes. Clearly we should be able to answer questions about both; just as clearly, operations on the pallet should work with boxes. Because we don't want to open them, and even if we do want to open them, we have to open them, we can't just ignore their existence! I assert that the extended grapheme cluster is the "box" around bytes in a string. Even if you do care about the contents (very often you do not!) you have to know where the boundaries are to do it! Because a Fitzpatric skin tone modifier all on its own has different semantics from one found within an emoji modifier sequence. So it makes perfect sense for Swift to provide one blessed way to iterate over strings, and provide other options for when you're interested in some other level of aggregation. Which is what Swift does.
ChrisSD 6y ago
In summary the two useful measures of unicode length are: * Number of native (preferably UTF-8) code units * Number of extended grapheme clusters The first is useful for basic string operations. The second is good for telling you what a user would consider a "character".
- bikeshaving 6y ago
  Why do people prefer UTF-8 coordinates? While for storage I think we should use UTF-8, when working with strings live it’s just so much easier to use UTF-16 because it’s predictable: 1 unit for the basic plane and 2 for everything else (the multi-character emoji and modifier stuff aside). I am probably biased because I mostly use and think about DOMStrings which are always UTF-16 but I’m not sure why people who use languages which are more flexible about string representations than JavaScript would not also appreciate this kind of regularity.
  - pdpi 6y ago
    I struggle to see why you'd ever want UTF-16. If you're using a variable length encoding, might as well stick to UTF-8. If you want predictable sizes, there's UTF-32 instead. ~Also, DOM Strings are not UTF-16, they're UCS-16.~ EDIT: UCS-2, not UCS-16. Also, I'm confusing the DOM with EcmaScript, and even that hasn't been true in a while.
    - ChrisSD 6y ago
      > Also, DOM Strings are not UTF-16, they're UCS-16. Hm, according to the spec they should be interpreted as UTF-16 but this isn't enforced by the language so it can contain unpaired surrogates: From https://heycam.github.io/webidl/#idl-DOMString https://heycam.github.io/webidl/#idl-DOMString > Such sequences are commonly interpreted as UTF-16 encoded strings [RFC2781] although this is not required... Nothing in this specification requires a DOMString value to be a valid UTF-16 string. From https://262.ecma-international.org/11.0/#sec-ecmascript-language-types-string-type https://262.ecma-international.org/11.0/#sec-ecmascript-lang... > The String type is the set of all ordered sequences of zero or more 16-bit unsigned integer values (“elements”) up to a maximum length of 253 - 1 elements. The String type is generally used to represent textual data in a running ECMAScript program, in which case each element in the String is treated as a UTF-16 code unit value... Operations that do interpret String values treat each element as a single UTF-16 code unit. However, ECMAScript does not restrict the value of or relationships between these code units, so operations that further interpret String contents as sequences of Unicode code points encoded in UTF-16 must account for ill-formed subsequences.
tzs 6y ago
5 in Perl, 17 in PHP.
- Rizz 6y ago
  17 in php with strlen, which is defined as counting bytes. 1 when you use grapheme_strlen. mb_strlen and iconv_strlen return 5, and 5 is rather useless as the article says.
throwawayffffas 6y ago
Python 3's approach is clearly the best. Because it focuses on the problem at hand, unicode codepoints. A string in python is a sequence of unicode codepoints, it's length should be the number of codepoints in the sequence, it has nothing to do with bytes. To draw an absurd parallel "[emoji]".len() == 17 is equivalent to [1,2].len() == 8 (2 32 bit integers) In my opinion the most useful result in the case the article describes is 5. There should of course be a way to get 1 (the number of extended graphemes), but it should not be the strings "length".
- nerdponx 6y ago
  Don't Swift and Go support iterating over graphemes? Edit: yes, Swift is mentioned at the bottom of the article. It'd be great to have a function for that in other scripting languages like Python, Ruby, etc. There was an interesting sub-thread here on HN a while ago about how a string-of-codepoints is just as bad as a string-of-bytes (subject to truncation, incorrect iteration, incorrect length calculation) and that we should just have string-of-bytes if we can't have string-of-graphemes. I don't agree, but some people felt very strongly about it.
- banthar 6y ago
  Defining string as a sequence of unicode codepoints is the mistake. Nobody ever cares about unicode codepoints. You either want the number of bytes or the width of the string on screen. UTF-32 codepoints waste space and give you neither.
  - stkdump 6y ago
    The width on the screen is in pixels. Yes, I find monospace fonts increasingly pointless.
- anaerobicover 6y ago
  But what do you do when you're processing a string with codepoints that compose into one user-visible glyph? >>> len("🇨🇦") 2
  - 6y ago
waterside81 6y ago
I like how Go handles this case by providing the utf8.DecodeRune/utf8.DecodeRuneInString functions which return each individual "rune" as well as the size in code points. Coming from python2, for me it was the first time I saw a language handle unicode so gracefully.
eximius 6y ago
Gah, I wish I was done with my crate so I could point to lovely formatted documentaion... But I believe this can be handled explicitly and well and I'm trying to do that in my fuzzy string matching library based on fuzzywuzzy. https://github.com/logannc/fuzzywuzzy-rs/pull/26/files https://github.com/logannc/fuzzywuzzy-rs/pull/26/files
patrickas 6y ago
Raku seems to be more correct (DWIM) in this regard than all the examples given in the post... my \emoji = "\c[FACE PALM]\c[EMOJI MODIFIER FITZPATRICK TYPE-3]\c[ZERO WIDTH JOINER]\c[MALE SIGN]\c[VARIATION SELECTOR-16]"; #one character say emoji.chars; # 1 #Five code points say emoji.codes; # 5 #If I want to know how many bytes that takes up in various encodings... say emoji.encode('UTF8').bytes; # 17 bytes say emoji.encode('UTF16').bytes; # 14 bytes Edit: Updated to use the names of each code point since HN cannot display the emoji
- duskwuff 6y ago
  You can represent it as a sequence of escapes. If Raku handles this the same way as Perl5, it should be: $a = "\N{FACE PALM}\N{EMOJI MODIFIER FITZPATRICK TYPE-3}\N{ZERO WIDTH JOINER}\N{MALE SIGN}\N{VARIATION SELECTOR-16}";
  - csharptwdec19 6y ago
    now do it in YAML
- cjm42 6y ago
  And if you try to say emoji.length, you'll get an error: No such method 'length' for invocant of type 'Str'. Did you mean any of these: 'codes', 'chars'? Because as the article points out, the "length" of a string is an ambiguous concept these days.
mannykannot 6y ago
In my curmudgeonly way, I suggest zero is the correct value, to reflect the information content of almost all emoji use.
- Kaze404 6y ago
  With that metric the length of your comment should be zero as well. Probably mine also :)
- elliekelly 6y ago
  Counterpoint: Appending U+1F609, U+1F61C, or U+1F643 to the end of your comment would have added immense value in communicating your point because it would have lightened the otherwise harsh tone of your words :)
  - mannykannot 6y ago
    I almost did, but I was concerned that doing so might be flirting with dissembling, were the inherent irony not be recognized! (BTW: Is ‘!’ the original emoji?)
hateful 6y ago
If you do a formula in Google Sheets and it contains an emoji, this comes into play. For example, if the first character is an emoji and you want to reference it you need to do =LEFT(A1, 2) - and not =LEFT(A1, 1)
maxnoe 6y ago
https://pypi.org/project/grapheme/ https://pypi.org/project/grapheme/
j1elo 6y ago
If you're working with an image, you might have an Image class, that has a Image.width and a Image.height in pixels, regardless of how these pixels are laid out in memory (depends on encoding, colorspace, etc). Most if not all methods operate on these pixels, e.g. cropping, scaling, color filtering, etc. Then, there might be a Image.memory property that provides acces to the underlying, actual bytes. I don't understand why the same is not the obvious solution for strings. len("🇨🇦") should be 1, because we as humans read one emoji as one character, regardless of the memory layout behind. Most if not all methods operate on characters (reversing, concatenating, indexing). And then, if you need access to the low level data, the String.memory would contain the actual bytes... which would be different depending on the actual text encoding.
- dnautics 6y ago
  The number of bytes necessary is incredibly important for security reasons. It's arguably better to make the number of bytes be the primary value and have a secondary lookup be the number of glyphs. To be fair, some systems distinguish between size and length (with size expected to be O(1) and length allowed to be up to O(n)). For those systems proceed as parent.
bob1029 6y ago
We need to deal with mountains of user input from mobile devices and the worst we have run into is "smart" quotes from iOS devices hosing older business systems. Our end users are pretty good about not typing pirate flags into customer name fields. I still haven't run into a situation where I need to count the logical number of glyphs in a string. Any system that we need to push a string into will be limiting things on the basis of byte counts, so a .Length check is still exactly what we need. Does this cause trouble for anyone else?
- ravi-delia 6y ago
  I have used glyph counting a handful of times, mostly for width computing before I learned there were better ways. I'm 100% sure my logic was just waiting to fail on any input that didn't use the Latin alphabet.
rurban 6y ago
Rusts graphemes.count sounds nice. In C land it is called ucwidth (libunistring). Length is too arbitrary. Byte? Unicode points? Utf-8 length?
Lightbody 6y ago
Relevant related writeup that was just on HN the other day (I think... or I saw it somewhere else): https://tonsky.me/blog/emoji/ https://tonsky.me/blog/emoji/ Edit: confirmed, it was on HN just half a day prior. Probably why this article arrived too. Just surprised nobody reference this other one in this thread :)

116 points by kevincox 6y ago

Techyrack 6y ago

#cryptocurrency #bitcoin #crypto

dooglius 6y ago

I think this title needs an exemption from HN's no-emoji rule

kyberias 6y ago
Why is there such a rule?
- colejohnson66 6y ago
  Emojis aren’t professional (depending on who you ask) and can be overused (look at a few popular GitHub projects’ READMEs - emojis on every bullet point of the feature list, for example)
  - tester34 6y ago
    I agree that they aren't the most professional thing to use, but I'm not sure why I agree Maybe it's some kind of bias?
  - fhifjfhjjjk 6y ago
    emojis aren't "professional" is your reasoning? is it 1987? should women also wear their skirts below the knee?
    - colejohnson66 6y ago
      I didn’t say that they weren’t. I’m simply saying that some do think that. Whether I do or don’t is irrelevant as I’m only speculating on the reason why.
  - Kaze404 6y ago
    What a weird reasoning. In most contexts I've seen, "professionalism" doesn't really do anything except strip away all the human factor that goes into every interaction. Personally, I don't care whether the person I'm talking to is being "professional" or not. What I care is that they're respectful and can properly communicate their thoughts. With that mindset, emojis (and emoticon) can actually add context to interactions, considering how much context is lost when communicating over text. A simple smiley face at the end of a sentence can go a long way in my experience :)
- at_a_remove 6y ago
  Also, looking at the article, they are quite complex! It looks like handling emojis in a proper manner has a large investment and the payoff is somewhat small.
  - wongarsu 6y ago
    Emoji don't introduce anything that isn't used for various other languages. Emojis are just the most visible breakage for American users when you screw up your unicode handling. HN however handles all of unicode just fine, it just chose to specifically exclude emojis (and a bunch of other symbols)
    - MayeulC 6y ago
      I think emoji is wonderful for widespread unicode support. It combats ossification. It is also a nice carrot to incite users to install updates. However, I don't like the rabbit hole it started to go into with gendered and colored emojis. There's never going to be enough. I wish we had stuck to color-neutral and gender-neutral, like original smileys :) I find it also conveys too much meaning. I am generally not interested in knowing a person's gender or ethnic group when discussing over text... but I digress.
      - samatman 6y ago
        I've said this before, but not here: Eventually, Unicode will allow you to combine flag modifiers with human emojis. So you can have a black South Korean man facepalming. This will trigger a war which ends industrial civilization, and is the leading candidate for the Great Filter.
  - Someone 6y ago
    For HN I would think almost all the complexity is in rendering. That’s a job for your browser. What’s left is things like the max length for a title (not too problematic to count that in code points or bytes, I think) The big risk, I think, is in users (mis)using rendering direction, multiple levels of accents, zero-width characters and weird code points to mess up the resulting page. Some Unicode symbols and emojis typically look a lot larger and heavier than ‘normal’ characters, switching writing direction may change visual indentation of a comment, etc. Worse, Unicode rendering being hard, text rendering bugs that crash browsers or even entire devices are discovered every now and then. If somebody entered one in a HN message, that could crash millions of devices.
- TeMPOraL 6y ago
  Surprisingly many people try to abuse Unicode when posting submissions and comments. This includes not just emojis, but also other symbols, or glyphs that look like bold, underlined or otherwise stylized letters.
  - criddell 6y ago
    𝕴 𝖉𝖔𝖓'𝖙 𝖙𝖍𝖎𝖓𝖐 𝖙𝖍𝖊𝖞'𝖛𝖊 𝖇𝖆𝖓𝖓𝖊𝖉 𝖎𝖙 𝖊𝖓𝖙𝖎𝖗𝖊𝖑𝖞.
    - masklinn 6y ago
      HN's character stripping is completely arbitrary. You can play mahjong, domino or cards (🀕, 🁓, 🃅) but not chess, you can show some random glyphs like ⏱ or ⤪ but not others, you can use box-drawing characters (┐) but not block elements, you can use byzantine musical notation (𝈙) but only western musical symbols (𝅗𝅥) and the notes are banned, you can Z̸̠̽a̷͍̟̱͔͛͘̚ĺ̸͎̌̄̌g̷͓͈̗̓͌̏̉o̴̢̺̹̕ just fine.
  - masklinn 6y ago
    > Surprisingly many people try to abuse Unicode when posting submissions and comments. ˙ǝsnqɐ ƃuıpıoʌɐ ʇɐ qoɾ pooƃ ɐ ɥɔns sǝop sɹǝʇɔɐɹɐɥɔ ɟo ɥɔunq ɐ ƃuıuuɐq ʎʃıɹɐɹʇıqɹɐ ǝsnɐɔǝ𐐒
- pessimizer 6y ago
  This is a forum sponsored by opinionated people who find them annoying. I find them annoying, too, so I think it's a good convention. Somehow ":)" became an industry, and as industries go, then eliminated ":)" (which gets replaced with a smiling yellow head in most places.) It isn't the only arbitrary convention enforced here, and the sum of those conventions are what attracts the audience.
- saagarjha 6y ago
  Emoji draw attention to themselves in a way that plain text does not. I assume this is why Hacker News does not allow bolding text and strongly discourages use of ALL CAPS.
colejohnson66 6y ago
I’m curious: is the no-emoji rule a rule that happens to block emoji or a hardcoded rule? What I mean is: emojis (in UTF-16) have to use surrogate pairs because all their code points are in the U+1xxxx plane. Is the software just disallowing any characters needing two code points to encode (which would include emoji)? Or is it specifically singling out the emoji blocks?
- kps 6y ago
  It seems to have changed recently. I recall a thread about plastics a few months ago where the plastic type symbols (♳ through ♹, i.e. U+2763 through U+2769) disappeared, but I see them now. Edits: ⭕ that post was https://news.ycombinator.com/item?id=25237688 https://news.ycombinator.com/item?id=25237688 ⭕ The rules seem pretty arbitrary. Recycling symbol U+2672 ♲ is allowed but recycled paper symbol U+267C is not. Chess kings ♔ are allowed but checkers kings aren't. ⭕ is allowed (for now). ⭕ I think the right thing to do would be to strip anything with emoji presentation http://unicode.org/reports/tr51/#Presentation_Style http://unicode.org/reports/tr51/#Presentation_Style
  - saagarjha 6y ago
    It’s not that simple, as many OSes render characters that aren’t strictly look emoji “as emoji” and there is no standard way to check for this.
- wongarsu 6y ago
  It's a very deliberate and precise filter. I can write in old persian: "𐎠𐎡𐎢" (codepoints 0x103xx) or CJK "𠀀𠀁𠀂" (codepoints 0x2000x), but can't write "" (emoji with codepoint 0x1F605)
- saagarjha 6y ago
  The latter.

colejohnson66 6y ago

I was wondering what the title meant. Turns out HN’s emoji stripper screwed with the title. It’s asking why a skin toned (not yellow) facepalm emoji’s length is 7 when the user perceives it as a single character. Tangent: Emojis are an interesting topic in regards to programming. They challenged the “rule” held by programmers that every character is a single codepoint (of 8 or 16 bits). So, str[1234] would get me the 1235th character, but it’s actually the 1235th byte. UTF-8 threw a wrench in that, but many programmers went along ignoring reality. Sadly, preexisting languages such as Arabic weren’t warning enough in regards to line breaking. As in: an Arabic “character”[a] can change its width depending on if there’s a “character” before or after it (which gives it its cursive-like look). So, a naive line breaking routine could cause bugs if it tried to break in the middle of a word. Tom Scott has a nice video on it that was filmed when the “effective power” crash was going around.[0] [0]: https://youtu.be/hJLMSllzoLA https://youtu.be/hJLMSllzoLA [a]: Arabic script isn’t technically an alphabet like Latin characters are. It’s either an abugida or abjad (depending on who you ask). See Wikipedia: https://en.wikipedia.org/wiki/Arabic_script https://en.wikipedia.org/wiki/Arabic_script

MengerSponge 6y ago
Interesting. Also, many of us use fonts with ligatures, which render as a single character (for example: tt, ti, ff, Th, ffi) Of course, we're taught to parse that as multiple discrete letters from an early age, so we don't get confused :)

sanedigital 6y ago

This may be the best tech writeup I've ever seen. Super in-depth, easily readable. Really well done.

donaldihunter 6y ago
With clickbaity misrepresentations of correctness, unfortunately.

SamBam 6y ago

After having read (quite interesting) the article, I still don't quite get the subtitle: > 'But It’s Better that "[emoji]".len() == 17 and Rather Useless that len("[emoji]") == 5' It sounds like it's just whether you're counting UTF-8/-16/-32 units. Does the article explain why one is worse and one is "rather useless?"

dahfizz 6y ago
I agree. The author talks a little bit about which UTF encoding makes sense in which situation, but they never make an argument about which result from len is correct. My two cents is that string length should always be the number of Unicode codepoints in the string, regardless of encoding. If you want the byte length, I'm sure there is a sizeof equivalent for your language. When we call len() on an array, we want the number of objects in the array. When we iterate over an array, we want to deal with one object from the array at a time. We don't care how big an object is. A large object shouldn't count for more than 1 when calculating the array length. Similarly, a unicode codepoint is the fundamental object in a string. The byte-size of a codepoint does not affect the length of the string. It makes no sense to iterate over each byte in a unicode string, because a byte on its own is completely meaningless. len() should be the number of objects we can iterate over, just like in arrays.
- MayeulC 6y ago
  But what about combining characters? https://en.wikipedia.org/wiki/Combining_Diacritical_Marks https://en.wikipedia.org/wiki/Combining_Diacritical_Marks Should the letter plus a combining character count as one (I think so), or two characters? Should you normalize before counting length? And so on.
  - dahfizz 6y ago
    Combining characters are their own unicode codepoint, so they count towards length. The beauty of this approach is that its simple and objective. If you had a list of 5 Dom Element objects, and one Dom Attr object, the length of that list is 6. Its nonsensical to say "The Attr object modifies an Element object, so its not really in the list".

beaconstudios 6y ago

perhaps the issue is that there's a canonical "length" at all. It would make more sense to me to have different types of length depending on which measure you're after, like Swift apparently has but without the canonical `.count`. Because when there's multiple interpretations of a thing's length, when you ask for "length" you're leaving the developers to resolve the ambiguity and I'm of the firm belief that developers shouldn't consider themselves psychic.

anaerobicover 6y ago
The main reason, I think, that Swift strings have `count` is that they conform to the `Collection` protocol. Swift's stdlib has a pervasive "generic programming" philosophy, enabled by various protocol heirarchies. So, given that the property is required to be present, some semantic or the other had to be chosen. I am sure there were debates when they were writing `String` about which one was proper.
- beaconstudios 6y ago
  that's very cringe of them. > So, given that the property is required to be present, some semantic or the other had to be chosen. Sounds like an invented solution to an invented problem. The programmer's speciality.
  - samatman 6y ago
    On the contrary it is based! This conforms exactly to our intuition about what a "collection" is. Some (exact) number of items which share some meaningful property in common such that we can consider them "the same" for purposes of enumeration and iteration. In the real world, we also have to decide what our collection is a collection of! Let's say we a pallet of candy bars, each of which is in a display box. If we want to ask "how many X" are on the pallet, we have to decide whether we're asking about the candy bars or the boxes. Clearly we should be able to answer questions about both; just as clearly, operations on the pallet should work with boxes. Because we don't want to open them, and even if we do want to open them, we have to open them, we can't just ignore their existence! I assert that the extended grapheme cluster is the "box" around bytes in a string. Even if you do care about the contents (very often you do not!) you have to know where the boundaries are to do it! Because a Fitzpatric skin tone modifier all on its own has different semantics from one found within an emoji modifier sequence. So it makes perfect sense for Swift to provide one blessed way to iterate over strings, and provide other options for when you're interested in some other level of aggregation. Which is what Swift does.

ChrisSD 6y ago

In summary the two useful measures of unicode length are: * Number of native (preferably UTF-8) code units * Number of extended grapheme clusters The first is useful for basic string operations. The second is good for telling you what a user would consider a "character".

bikeshaving 6y ago
Why do people prefer UTF-8 coordinates? While for storage I think we should use UTF-8, when working with strings live it’s just so much easier to use UTF-16 because it’s predictable: 1 unit for the basic plane and 2 for everything else (the multi-character emoji and modifier stuff aside). I am probably biased because I mostly use and think about DOMStrings which are always UTF-16 but I’m not sure why people who use languages which are more flexible about string representations than JavaScript would not also appreciate this kind of regularity.
- pdpi 6y ago
  I struggle to see why you'd ever want UTF-16. If you're using a variable length encoding, might as well stick to UTF-8. If you want predictable sizes, there's UTF-32 instead. ~Also, DOM Strings are not UTF-16, they're UCS-16.~ EDIT: UCS-2, not UCS-16. Also, I'm confusing the DOM with EcmaScript, and even that hasn't been true in a while.
  - ChrisSD 6y ago
    > Also, DOM Strings are not UTF-16, they're UCS-16. Hm, according to the spec they should be interpreted as UTF-16 but this isn't enforced by the language so it can contain unpaired surrogates: From https://heycam.github.io/webidl/#idl-DOMString https://heycam.github.io/webidl/#idl-DOMString > Such sequences are commonly interpreted as UTF-16 encoded strings [RFC2781] although this is not required... Nothing in this specification requires a DOMString value to be a valid UTF-16 string. From https://262.ecma-international.org/11.0/#sec-ecmascript-language-types-string-type https://262.ecma-international.org/11.0/#sec-ecmascript-lang... > The String type is the set of all ordered sequences of zero or more 16-bit unsigned integer values (“elements”) up to a maximum length of 253 - 1 elements. The String type is generally used to represent textual data in a running ECMAScript program, in which case each element in the String is treated as a UTF-16 code unit value... Operations that do interpret String values treat each element as a single UTF-16 code unit. However, ECMAScript does not restrict the value of or relationships between these code units, so operations that further interpret String contents as sequences of Unicode code points encoded in UTF-16 must account for ill-formed subsequences.

tzs 6y ago

5 in Perl, 17 in PHP.

Rizz 6y ago
17 in php with strlen, which is defined as counting bytes. 1 when you use grapheme_strlen. mb_strlen and iconv_strlen return 5, and 5 is rather useless as the article says.

throwawayffffas 6y ago

Python 3's approach is clearly the best. Because it focuses on the problem at hand, unicode codepoints. A string in python is a sequence of unicode codepoints, it's length should be the number of codepoints in the sequence, it has nothing to do with bytes. To draw an absurd parallel "[emoji]".len() == 17 is equivalent to [1,2].len() == 8 (2 32 bit integers) In my opinion the most useful result in the case the article describes is 5. There should of course be a way to get 1 (the number of extended graphemes), but it should not be the strings "length".

nerdponx 6y ago
Don't Swift and Go support iterating over graphemes? Edit: yes, Swift is mentioned at the bottom of the article. It'd be great to have a function for that in other scripting languages like Python, Ruby, etc. There was an interesting sub-thread here on HN a while ago about how a string-of-codepoints is just as bad as a string-of-bytes (subject to truncation, incorrect iteration, incorrect length calculation) and that we should just have string-of-bytes if we can't have string-of-graphemes. I don't agree, but some people felt very strongly about it.
banthar 6y ago
Defining string as a sequence of unicode codepoints is the mistake. Nobody ever cares about unicode codepoints. You either want the number of bytes or the width of the string on screen. UTF-32 codepoints waste space and give you neither.
- stkdump 6y ago
  The width on the screen is in pixels. Yes, I find monospace fonts increasingly pointless.
anaerobicover 6y ago
But what do you do when you're processing a string with codepoints that compose into one user-visible glyph? >>> len("🇨🇦") 2
- 6y ago

waterside81 6y ago

I like how Go handles this case by providing the utf8.DecodeRune/utf8.DecodeRuneInString functions which return each individual "rune" as well as the size in code points. Coming from python2, for me it was the first time I saw a language handle unicode so gracefully.

eximius 6y ago

Gah, I wish I was done with my crate so I could point to lovely formatted documentaion... But I believe this can be handled explicitly and well and I'm trying to do that in my fuzzy string matching library based on fuzzywuzzy. https://github.com/logannc/fuzzywuzzy-rs/pull/26/files https://github.com/logannc/fuzzywuzzy-rs/pull/26/files

patrickas 6y ago

Raku seems to be more correct (DWIM) in this regard than all the examples given in the post... my \emoji = "\c[FACE PALM]\c[EMOJI MODIFIER FITZPATRICK TYPE-3]\c[ZERO WIDTH JOINER]\c[MALE SIGN]\c[VARIATION SELECTOR-16]"; #one character say emoji.chars; # 1 #Five code points say emoji.codes; # 5 #If I want to know how many bytes that takes up in various encodings... say emoji.encode('UTF8').bytes; # 17 bytes say emoji.encode('UTF16').bytes; # 14 bytes Edit: Updated to use the names of each code point since HN cannot display the emoji

duskwuff 6y ago
You can represent it as a sequence of escapes. If Raku handles this the same way as Perl5, it should be: $a = "\N{FACE PALM}\N{EMOJI MODIFIER FITZPATRICK TYPE-3}\N{ZERO WIDTH JOINER}\N{MALE SIGN}\N{VARIATION SELECTOR-16}";
- csharptwdec19 6y ago
  now do it in YAML
cjm42 6y ago
And if you try to say emoji.length, you'll get an error: No such method 'length' for invocant of type 'Str'. Did you mean any of these: 'codes', 'chars'? Because as the article points out, the "length" of a string is an ambiguous concept these days.

mannykannot 6y ago

In my curmudgeonly way, I suggest zero is the correct value, to reflect the information content of almost all emoji use.

Kaze404 6y ago
With that metric the length of your comment should be zero as well. Probably mine also :)
elliekelly 6y ago
Counterpoint: Appending U+1F609, U+1F61C, or U+1F643 to the end of your comment would have added immense value in communicating your point because it would have lightened the otherwise harsh tone of your words :)
- mannykannot 6y ago
  I almost did, but I was concerned that doing so might be flirting with dissembling, were the inherent irony not be recognized! (BTW: Is ‘!’ the original emoji?)

hateful 6y ago

If you do a formula in Google Sheets and it contains an emoji, this comes into play. For example, if the first character is an emoji and you want to reference it you need to do =LEFT(A1, 2) - and not =LEFT(A1, 1)

maxnoe 6y ago

https://pypi.org/project/grapheme/ https://pypi.org/project/grapheme/

j1elo 6y ago

If you're working with an image, you might have an Image class, that has a Image.width and a Image.height in pixels, regardless of how these pixels are laid out in memory (depends on encoding, colorspace, etc). Most if not all methods operate on these pixels, e.g. cropping, scaling, color filtering, etc. Then, there might be a Image.memory property that provides acces to the underlying, actual bytes. I don't understand why the same is not the obvious solution for strings. len("🇨🇦") should be 1, because we as humans read one emoji as one character, regardless of the memory layout behind. Most if not all methods operate on characters (reversing, concatenating, indexing). And then, if you need access to the low level data, the String.memory would contain the actual bytes... which would be different depending on the actual text encoding.

dnautics 6y ago
The number of bytes necessary is incredibly important for security reasons. It's arguably better to make the number of bytes be the primary value and have a secondary lookup be the number of glyphs. To be fair, some systems distinguish between size and length (with size expected to be O(1) and length allowed to be up to O(n)). For those systems proceed as parent.

bob1029 6y ago

We need to deal with mountains of user input from mobile devices and the worst we have run into is "smart" quotes from iOS devices hosing older business systems. Our end users are pretty good about not typing pirate flags into customer name fields. I still haven't run into a situation where I need to count the logical number of glyphs in a string. Any system that we need to push a string into will be limiting things on the basis of byte counts, so a .Length check is still exactly what we need. Does this cause trouble for anyone else?

ravi-delia 6y ago
I have used glyph counting a handful of times, mostly for width computing before I learned there were better ways. I'm 100% sure my logic was just waiting to fail on any input that didn't use the Latin alphabet.

rurban 6y ago

Rusts graphemes.count sounds nice. In C land it is called ucwidth (libunistring). Length is too arbitrary. Byte? Unicode points? Utf-8 length?

Lightbody 6y ago

Relevant related writeup that was just on HN the other day (I think... or I saw it somewhere else): https://tonsky.me/blog/emoji/ https://tonsky.me/blog/emoji/ Edit: confirmed, it was on HN just half a day prior. Probably why this article arrived too. Just surprised nobody reference this other one in this thread :)