4 ms·

Apparently yes. A quick inspection suggests that this is the same in Ruby, Haskell, SWI-Prolog, Gauche Scheme, SBCL and D. (I might not have the latest version

Apparently yes. A quick inspection suggests that this is the same in Ruby, Haskell, SWI-Prolog, Gauche Scheme, SBCL and D. (I might not have the latest version of everything, so maybe this has been fixed in some of them... assuming it needs fixing. Maybe there's a reason for the answer to be 2 if so many language implementations insist on it. Or, they all use the same faulty algorithm. I don't know.)

a1369209993 7y ago
> they all use the same faulty algorithm Well, yes. To be fair, it's not like any of them make a secret of the fact that they're mistakenly counting unicode code points instead of characters.
- Smithalicious 7y ago
  Are they really doing so "mistakenly"? I feel like there's more to this.
  - dodobirdlord 7y ago
    It's not mistakenly. Unicode's complexity is a bit more than trivial, and since much work has gone into abstracting over it many people are surprised when the complexity rears up at them. Consider, for example, the wonderful piece of writing in the answer to this question. https://stackoverflow.com/questions/1732348/regex-match-open-tags-except-xhtml-self-contained-tags https://stackoverflow.com/questions/1732348/regex-match-open... How many characters do you suppose are in this string? . "TO͇̹̺ͅƝ̴ȳ̳ TH̘Ë͖́̉ ͠P̯͍̭O̚ N̐Y̡ H̸̡̪̯ͨ͊̽̅̾̎Ȩ̬̩̾͛ͪ̈́̀́͘ ̶̧̨̱̹̭̯ͧ̾ͬC̷̙̲̝͖ͭ̏ͥͮ͟Oͮ͏̮̪̝͍M̲̖͊̒ͪͩͬ̚̚͜Ȇ̴̟̟͙̞ͩ͌͝S̨̥̫͎̭ͯ̿̔̀ͅ" . And what should Python tell you the length of this string is?
    - a1369209993 7y ago
      23 if I'm counting correctly (there's a space in "PO NY" for some reason). I would also accept 209 from a language that elected not to deal with large amounts of complexity in string handling. The problem with Unicode is they go to enormous amounts of effort to deliberately give a wrong answer.
    - jrochkind1 7y ago
      Yes, what you said. It's not a mistake. It's a... useful abstraction. Unicode is complicated in some ways because the domain it is dealing with (representing all possible human written communication, basically) is complicated. Unicode is pretty ingenious. It pays to invest in learning about it, rather than assuming your "naive" conclusions are what it "should" do (and unicode's standard docs are pretty readable). Unicode does offer an algorithm for segmenting text into "grapheme clusters", specifically "user-perceived characters." https://unicode.org/reports/tr29/ https://unicode.org/reports/tr29/ It's worth reading that document when deciding what you think the "right" thing to do with "len()" is. The "user-perceived character segmentation" algorithm is complicated, it has a performance cost... and it's implemented in terms of the lower-level codepoint abstraction. Dealing with codepoints is the right thing for most platforms to do, as the basic API. Codepoints are the basic API into unicode. It's true that they ideally ought to also give you access to TR29 character segmentation. And most don't. Cause it's hard and confusing and nobody's done it I guess. It would be nice. If you want to know "well, howe come codepoints are the basic unicode abstraction/API? Why couldn't user-perceived characters be?" Then start reading other unicode docs too, and eventually you'll understand how we got here. (For starters, a "user-perceived character" can actually be locale-dependent, what's two characters in one language may be one in another).
      - a1369209993 7y ago
        > It's not a mistake. It's a... useful abstraction. It is specifially a abstraction that is not useful. > It's worth reading that document when deciding what you think the "right" thing to do with "len()" is. Technically not - the right thing to do is return the number of characters[0] - but the character segmentation parts are worth reading when deciding how to decode UTF-8 bytes into characters in the first place, so the distinction is somewhat academic. > a [character] can actually be locale-dependent, what's two characters in one language may be one in another [citation needed]; ch, ij, dz, etc are not examples, but I'm admitted not exhaustively familiar with non-latin scripts[1], so I would be interested to see what other scripts do. 0: or bytes, but that's trivial 1: Which is why I hate Unicode; I'd prefer to pawn that work off on someone else and just import a library, but Unicode has ensured that all available libraries are always unusably broken.
- deleted 7y ago
  [deleted]

a1369209993 7y ago

> they all use the same faulty algorithm Well, yes. To be fair, it's not like any of them make a secret of the fact that they're mistakenly counting unicode code points instead of characters.

Smithalicious 7y ago
Are they really doing so "mistakenly"? I feel like there's more to this.
- dodobirdlord 7y ago
  It's not mistakenly. Unicode's complexity is a bit more than trivial, and since much work has gone into abstracting over it many people are surprised when the complexity rears up at them. Consider, for example, the wonderful piece of writing in the answer to this question. https://stackoverflow.com/questions/1732348/regex-match-open-tags-except-xhtml-self-contained-tags https://stackoverflow.com/questions/1732348/regex-match-open... How many characters do you suppose are in this string? . "TO͇̹̺ͅƝ̴ȳ̳ TH̘Ë͖́̉ ͠P̯͍̭O̚ N̐Y̡ H̸̡̪̯ͨ͊̽̅̾̎Ȩ̬̩̾͛ͪ̈́̀́͘ ̶̧̨̱̹̭̯ͧ̾ͬC̷̙̲̝͖ͭ̏ͥͮ͟Oͮ͏̮̪̝͍M̲̖͊̒ͪͩͬ̚̚͜Ȇ̴̟̟͙̞ͩ͌͝S̨̥̫͎̭ͯ̿̔̀ͅ" . And what should Python tell you the length of this string is?
  - a1369209993 7y ago
    23 if I'm counting correctly (there's a space in "PO NY" for some reason). I would also accept 209 from a language that elected not to deal with large amounts of complexity in string handling. The problem with Unicode is they go to enormous amounts of effort to deliberately give a wrong answer.
  - jrochkind1 7y ago
    Yes, what you said. It's not a mistake. It's a... useful abstraction. Unicode is complicated in some ways because the domain it is dealing with (representing all possible human written communication, basically) is complicated. Unicode is pretty ingenious. It pays to invest in learning about it, rather than assuming your "naive" conclusions are what it "should" do (and unicode's standard docs are pretty readable). Unicode does offer an algorithm for segmenting text into "grapheme clusters", specifically "user-perceived characters." https://unicode.org/reports/tr29/ https://unicode.org/reports/tr29/ It's worth reading that document when deciding what you think the "right" thing to do with "len()" is. The "user-perceived character segmentation" algorithm is complicated, it has a performance cost... and it's implemented in terms of the lower-level codepoint abstraction. Dealing with codepoints is the right thing for most platforms to do, as the basic API. Codepoints are the basic API into unicode. It's true that they ideally ought to also give you access to TR29 character segmentation. And most don't. Cause it's hard and confusing and nobody's done it I guess. It would be nice. If you want to know "well, howe come codepoints are the basic unicode abstraction/API? Why couldn't user-perceived characters be?" Then start reading other unicode docs too, and eventually you'll understand how we got here. (For starters, a "user-perceived character" can actually be locale-dependent, what's two characters in one language may be one in another).
    - a1369209993 7y ago
      > It's not a mistake. It's a... useful abstraction. It is specifially a abstraction that is not useful. > It's worth reading that document when deciding what you think the "right" thing to do with "len()" is. Technically not - the right thing to do is return the number of characters[0] - but the character segmentation parts are worth reading when deciding how to decode UTF-8 bytes into characters in the first place, so the distinction is somewhat academic. > a [character] can actually be locale-dependent, what's two characters in one language may be one in another [citation needed]; ch, ij, dz, etc are not examples, but I'm admitted not exhaustively familiar with non-latin scripts[1], so I would be interested to see what other scripts do. 0: or bytes, but that's trivial 1: Which is why I hate Unicode; I'd prefer to pawn that work off on someone else and just import a library, but Unicode has ensured that all available libraries are always unusably broken.
deleted 7y ago
[deleted]