3 ms·

I think he meant to bring out defensiveness with that quip. He never says that it's a big deal, just that it's the worst way to get the length of a string conta

I think he meant to bring out defensiveness with that quip. He never says that it's a big deal, just that it's the worst way to get the length of a string containing emoji, presumably of the mainstream languages.

tehsauce 3y ago
Why is it the worst way though?
- deleted 3y ago
  [deleted]
- imron 3y ago
  > Note about Python 3 added on 2019-09-09: Originally this article claimed that Python 3 guaranteed UTF-32 validity. This was in error. Python 3 guarantees that the units of the string stay within the Unicode code point range but does not guarantee the absence of surrogates. It not only allows unpaired surrogates, which might be explained by wishing to be compatible with the value space of potentially-invalid UTF-16, but Python 3 allows materializing even surrogate pairs, which is a truly bizarre design. The previous conclusions stand with the added conclusion that Python 3 is even more messed up than I thought!
- markmark 3y ago
  Perhaps the part about it needing lookups of the unicode database and being dependent on the version of the database used?
  - lmm 3y ago
    That's not true though. It just counts the number of code units, that's not version dependent. It's certainly no worse than counting the number of UTF-16 points (I'd argue it's better since it's less arbitrary - whether something is a unicode scalar is a design decision, whether something is in the BMP or not is mostly an accident of implementation).
- masklinn 3y ago
  Because it’s never actually useful. You can’t use that information to know how much actual space it takes (in storage) as nobody sane stores UTF-32, you can’t use it to know much much logical space it takes (aka the user’s interpretation), you can’t use it to know how much visual space it takes (not that you can ever get that), and you can’t use it to segment or process the text. A length in codepoints gives you nothing that’s really actionable, at least not that you’d need outside of a context where you could easily obtain it otherwise.
  - L3viathan 3y ago
    It is useful: When iterating over a string in Python (which I hope you agree _is_ useful?), you get that many parts.
    - chrismorgan 3y ago
      > When iterating over a string in Python (which I hope you agree _is_ useful?) Not often. There’s almost nothing useful you can correctly do with a sequence of code points.
    - masklinn 3y ago
      > It is useful: When iterating over a string in Python (which I hope you agree _is_ useful?), you get that many parts. That’s… not useful? I can’t say I remember ever caring knowing how many items I would be getting during an iteration[0]. If I want to set an iteration limit I can just… do that, using `islice` or some such. [0] in python anyway, in lower level language there can be a utility in order to pre-allocate an output collection
    - deleted 3y ago
      [deleted]
  - boxed 3y ago
    But 100% of all those complaints also apply to JS, except that UTF-16 is just much more stupid than UTF-32?
  - globular-toast 3y ago

tehsauce 3y ago

Why is it the worst way though?

deleted 3y ago
[deleted]
imron 3y ago
> Note about Python 3 added on 2019-09-09: Originally this article claimed that Python 3 guaranteed UTF-32 validity. This was in error. Python 3 guarantees that the units of the string stay within the Unicode code point range but does not guarantee the absence of surrogates. It not only allows unpaired surrogates, which might be explained by wishing to be compatible with the value space of potentially-invalid UTF-16, but Python 3 allows materializing even surrogate pairs, which is a truly bizarre design. The previous conclusions stand with the added conclusion that Python 3 is even more messed up than I thought!
markmark 3y ago
Perhaps the part about it needing lookups of the unicode database and being dependent on the version of the database used?
- lmm 3y ago
  That's not true though. It just counts the number of code units, that's not version dependent. It's certainly no worse than counting the number of UTF-16 points (I'd argue it's better since it's less arbitrary - whether something is a unicode scalar is a design decision, whether something is in the BMP or not is mostly an accident of implementation).
masklinn 3y ago
Because it’s never actually useful. You can’t use that information to know how much actual space it takes (in storage) as nobody sane stores UTF-32, you can’t use it to know much much logical space it takes (aka the user’s interpretation), you can’t use it to know how much visual space it takes (not that you can ever get that), and you can’t use it to segment or process the text. A length in codepoints gives you nothing that’s really actionable, at least not that you’d need outside of a context where you could easily obtain it otherwise.
- L3viathan 3y ago
  It is useful: When iterating over a string in Python (which I hope you agree _is_ useful?), you get that many parts.
  - chrismorgan 3y ago
    > When iterating over a string in Python (which I hope you agree _is_ useful?) Not often. There’s almost nothing useful you can correctly do with a sequence of code points.
  - masklinn 3y ago
    > It is useful: When iterating over a string in Python (which I hope you agree _is_ useful?), you get that many parts. That’s… not useful? I can’t say I remember ever caring knowing how many items I would be getting during an iteration[0]. If I want to set an iteration limit I can just… do that, using `islice` or some such. [0] in python anyway, in lower level language there can be a utility in order to pre-allocate an output collection
  - deleted 3y ago
    [deleted]
- boxed 3y ago
  But 100% of all those complaints also apply to JS, except that UTF-16 is just much more stupid than UTF-32?
- globular-toast 3y ago