7 ms·

Python3 is a good example of what happens when you represent strings as an index of code points. I always found that a bit ironic in that one of the major justi

Python3 is a good example of what happens when you represent strings as an index of code points. I always found that a bit ironic in that one of the major justifications for python3 was to improve Unicode. In the end it failed to do that because of this.

pxeger1 4y ago
> what happens when you represent strings as an index of code points What happens? Are you referring to a specific problem that Python 3 has?
chrismorgan 4y ago
Worse still, Python strings aren’t even valid Unicode, because they’re sequences of code points rather than scalar values as they should be; and so even UTF–8‐encoding a str is fallible: >>> '\udead'.encode('utf-8') Traceback (most recent call last): File "<stdin>", line 1, in <module> UnicodeEncodeError: 'utf-8' codec can't encode character '\udead' in position 0: surrogates not allowed
- kazinator 4y ago
  That should Just Do What It's Told and produce ED BA AD.
  - chrismorgan 4y ago
    That would be baad, because it would not be valid UTF-8. Surrogates are a mess. Blame UTF-16. It was a terrible idea that never should have been implemented.
    - morelisp 4y ago
      Both you and parent are correct. (It should have been valid UTF-8, in practice it often is treated as such anyway. It should also never have existed in the first place.)
      - chrismorgan 4y ago
        No, it should not have been valid UTF-8, and nothing that validates UTF-8 (which is the considerable majority of things these days, though certainly not all) will accept it. UTF-16 fundamentally can’t represent surrogates: 0x0000 to 0xd7ff represent U+0000 to U+D7FF, and pairs of 0xd800–0xdbff (1024 code units) and 0xdc00–0xdfff (1024 code units) represent U+10000 to U+10FFFF (1024² code point), and there’s nothing left that could represent U+D800–U+DFFF. With UTF-16, there were two possible choices: either make UTF-16 an incomplete encoding, unable to represent some Unicode text, or separate surrogates out as reserved for UTF-16, not encodable and not permitted in Unicode text. Neither solution is good (because too much software doesn’t validate Unicode, and too much software treats UTF-16 more like UCS-2), and the correct answer would have been to throw away UCS-2 as a failed experiment and never invent UTF-16, but here we are now. They decided in favour of consistency between Unicode Transformation Formats, which is fairly clearly the better of the first two choices.
        account42 4y ago
        > No, it should not have been valid UTF-8, and nothing that validates UTF-8 (which is the considerable majority of things these days, though certainly not all) will accept it. Yes it should as those code points should have never been reserved. UTF-8 should only concern itself with encoding arbitrary 24-bit integers. And that those characters are reserved is also useful in practice because lots of things are encoded in UCS2 uhhm I mean "UTF-16" where unpaired surrogates are the reality. Hence: https://simonsapin.github.io/wtf-8/ https://simonsapin.github.io/wtf-8/ That UTF-16 can't represent those code points is UTF-16's problem and should have been solved by those clinging to their UCS2 codebases instead of being enshrined in Unicode. It would have certainly been possible to make a 16-bit Unicode encoding that is as much UCS2-compatible as UTF-16 is but can represent all code points. Also, almost nothing should validate UTF-8.
        chrismorgan 4y ago
        Unicode Transformation Formats are designed to be equivalent in what they can encode. This is simple fact, and sane. Yes, I hate how UTF-16 ruined Unicode, and the typical absence of validation of UTF-16 is a problem, but given the abomination that is UTF-16, surrogates not being part of valid Unicode strings is certainly a better course of action than UTF-16 being unable to represent certain Unicode strings. Just about everything should validate UTF-8, because if you don’t, various invariants that you depend on will break. As a simple example, if you operate on unvalidated UTF-8, then all kinds of operations that care about code points will have to check that they’re in-bounds at every single code unit, because to do otherwise would be memory-unsafe, risking crashing or possibly worse. It’s faster and much more peaceful if you validate UTF-8 in all strings at time of ingress, once for all. Given Rust’s preferences for correctness and performance, it is very clearly in the right in validating UTF-8 in all strings.
mumblemumble 4y ago
They didn't get it perfect, but, compared to what it was like to deal with Unicode in Python 2, I'd say they absolutely succeeded at improving the experience of working with Unicode text.
- afiori 4y ago
  weren't python 2 string essentially unencoded byte-strings? In my uninformed opinion they should have simply added the u"..." and b"..." formats and forbid mixing them or using a u with a b operation and viceversa. python 3 could have simply changed the default from b to u for untagged strings.
  - mumblemumble 4y ago
    This is how Python 2 did it, only with the difference that you didn't need the b prefix for ascii/byte strings. The problem was that this made a muddle of all the string handling functions, which needed to be prepared to handle both types. Including ones you write yourself. And, Python being a dynamic language, getting this right was fiddly and error prone. "Forbid mixing the two" was a workable, if ergonomically annoying, in statically typed languages that predated and then subsequently had to adopt Unicode. For Python, though, it was an enormous minefield of runtime errors.
    - tyingq 4y ago
      And you also had to detect which version of Python2, since Python <= 2.5.x didn't have some of the unicode support that later versions did.
  - akubera 4y ago
    Isn't that what they ended up doing in Python3? $ python2 >>> "abc" + b"123" 'abc123' >>> "abc" + u"123" u'abc123' $ python3 >>> "abc" + b"123" Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: can only concatenate str (not "bytes") to str >>> "abc" + u"123" 'abc123' >>> type(u"123") <class 'str'> P.S. Your use of the word "simply" there is a reminder of how easy it is to underestimate the complexity of handling encoding properly, since it took years (a decade?) to port libraries from Py2 to Py3, since it doesn't just affect string literals, but general IO when reading from files and sockets.

pxeger1 4y ago

> what happens when you represent strings as an index of code points What happens? Are you referring to a specific problem that Python 3 has?

chrismorgan 4y ago

Worse still, Python strings aren’t even valid Unicode, because they’re sequences of code points rather than scalar values as they should be; and so even UTF–8‐encoding a str is fallible: >>> '\udead'.encode('utf-8') Traceback (most recent call last): File "<stdin>", line 1, in <module> UnicodeEncodeError: 'utf-8' codec can't encode character '\udead' in position 0: surrogates not allowed

kazinator 4y ago
That should Just Do What It's Told and produce ED BA AD.
- chrismorgan 4y ago
  That would be baad, because it would not be valid UTF-8. Surrogates are a mess. Blame UTF-16. It was a terrible idea that never should have been implemented.
  - morelisp 4y ago
    Both you and parent are correct. (It should have been valid UTF-8, in practice it often is treated as such anyway. It should also never have existed in the first place.)
    - chrismorgan 4y ago
      No, it should not have been valid UTF-8, and nothing that validates UTF-8 (which is the considerable majority of things these days, though certainly not all) will accept it. UTF-16 fundamentally can’t represent surrogates: 0x0000 to 0xd7ff represent U+0000 to U+D7FF, and pairs of 0xd800–0xdbff (1024 code units) and 0xdc00–0xdfff (1024 code units) represent U+10000 to U+10FFFF (1024² code point), and there’s nothing left that could represent U+D800–U+DFFF. With UTF-16, there were two possible choices: either make UTF-16 an incomplete encoding, unable to represent some Unicode text, or separate surrogates out as reserved for UTF-16, not encodable and not permitted in Unicode text. Neither solution is good (because too much software doesn’t validate Unicode, and too much software treats UTF-16 more like UCS-2), and the correct answer would have been to throw away UCS-2 as a failed experiment and never invent UTF-16, but here we are now. They decided in favour of consistency between Unicode Transformation Formats, which is fairly clearly the better of the first two choices.
      - account42 4y ago
        > No, it should not have been valid UTF-8, and nothing that validates UTF-8 (which is the considerable majority of things these days, though certainly not all) will accept it. Yes it should as those code points should have never been reserved. UTF-8 should only concern itself with encoding arbitrary 24-bit integers. And that those characters are reserved is also useful in practice because lots of things are encoded in UCS2 uhhm I mean "UTF-16" where unpaired surrogates are the reality. Hence: https://simonsapin.github.io/wtf-8/ https://simonsapin.github.io/wtf-8/ That UTF-16 can't represent those code points is UTF-16's problem and should have been solved by those clinging to their UCS2 codebases instead of being enshrined in Unicode. It would have certainly been possible to make a 16-bit Unicode encoding that is as much UCS2-compatible as UTF-16 is but can represent all code points. Also, almost nothing should validate UTF-8.
        chrismorgan 4y ago
        Unicode Transformation Formats are designed to be equivalent in what they can encode. This is simple fact, and sane. Yes, I hate how UTF-16 ruined Unicode, and the typical absence of validation of UTF-16 is a problem, but given the abomination that is UTF-16, surrogates not being part of valid Unicode strings is certainly a better course of action than UTF-16 being unable to represent certain Unicode strings. Just about everything should validate UTF-8, because if you don’t, various invariants that you depend on will break. As a simple example, if you operate on unvalidated UTF-8, then all kinds of operations that care about code points will have to check that they’re in-bounds at every single code unit, because to do otherwise would be memory-unsafe, risking crashing or possibly worse. It’s faster and much more peaceful if you validate UTF-8 in all strings at time of ingress, once for all. Given Rust’s preferences for correctness and performance, it is very clearly in the right in validating UTF-8 in all strings.

mumblemumble 4y ago

They didn't get it perfect, but, compared to what it was like to deal with Unicode in Python 2, I'd say they absolutely succeeded at improving the experience of working with Unicode text.

afiori 4y ago
weren't python 2 string essentially unencoded byte-strings? In my uninformed opinion they should have simply added the u"..." and b"..." formats and forbid mixing them or using a u with a b operation and viceversa. python 3 could have simply changed the default from b to u for untagged strings.
- mumblemumble 4y ago
  This is how Python 2 did it, only with the difference that you didn't need the b prefix for ascii/byte strings. The problem was that this made a muddle of all the string handling functions, which needed to be prepared to handle both types. Including ones you write yourself. And, Python being a dynamic language, getting this right was fiddly and error prone. "Forbid mixing the two" was a workable, if ergonomically annoying, in statically typed languages that predated and then subsequently had to adopt Unicode. For Python, though, it was an enormous minefield of runtime errors.
  - tyingq 4y ago
    And you also had to detect which version of Python2, since Python <= 2.5.x didn't have some of the unicode support that later versions did.
- akubera 4y ago
  Isn't that what they ended up doing in Python3? $ python2 >>> "abc" + b"123" 'abc123' >>> "abc" + u"123" u'abc123' $ python3 >>> "abc" + b"123" Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: can only concatenate str (not "bytes") to str >>> "abc" + u"123" 'abc123' >>> type(u"123") <class 'str'> P.S. Your use of the word "simply" there is a reminder of how easy it is to underestimate the complexity of handling encoding properly, since it took years (a decade?) to port libraries from Py2 to Py3, since it doesn't just affect string literals, but general IO when reading from files and sockets.