3 ms·

> Surrogate pairs are not allowed in UTF8 (only in UTF16), so this error is not surprising. It must be decoded from UTF16, then reencoded to UTF8. The source c

> Surrogate pairs are not allowed in UTF8 (only in UTF16), so this error is not surprising. It must be decoded from UTF16, then reencoded to UTF8. The source code here is whatever your on-disk .py file is, likely ASCII, so UTF-8. No re-encoding is required, in practice or mentally. > The in-memory representation is an implementation detail, and can be anything that works, as you described. The in memory representation is not really the issue; it's the type itself. Consider a type is a set of possible values: bool, for example, is the set {true, false}. u8 is a set containing {0, 1, … 255}, and things like strings are infinite sets. The set of values that `str` represents is not the same set as what UTF-8 can encode / is not the same set of values as a type that encodes any possible sequence of USVs. The set of all possible `str` instances in Rust != the set of all possible `str` instances in Python. Rust is pretty unique in this regard: I don't think I know of another language whose string type is a sequence of USVs. (Though I assume there is likely one out there.) It is worse, too: there's JS's string type, for example, which is a sequence of UTF-16 code units, which is different yet! So (JS string) ≠ (Python str) ≠ (Rust str)

mixmastamyk 2y ago
On disk source-code is not relevant. To normalize surrogate pairs for later export to UTF8, they must be decoded first: >>> '\ud83d\ude04'.encode('utf-16', 'surrogatepass').decode('utf-16') '' >>> _.encode('utf8') b'\xf0\x9f\x98\x84' (The happy face emoji was removed by HN, run line in terminal to see it.) In other words surrogate pairs are only vaild in the context of UTF16 encoded bytes and no where else. If you're adding them into your program by hand on purpose, you're either doing it wrong or specifically to raise an error for illustration purposes.
- deathanatos 2y ago
  I'm well aware that surrogates are only valid in the context of UTF-16's encoding. > If you're adding them into your program by hand on purpose, you're either doing it wrong or specifically to raise an error for illustration purposes. In the comments above, it should be obvious that it is for illustrative purposes. The point is that the string type does not catch these errors. It's not that someone is going to "add them to the program … on purpose" … it's that they're going to slip in from bad data, or bad code, not on purpose but because people write bugs and generate garbage data. A good type does not let you represent illegal states. For the same reason null is a bad idea — here, it's not a valid unicode string. The point of a Unicode string type would be to expose operations that one might want to perform on Unicode strings. By permitting invalid values to exist in the set of values the type can represent, you're basically making every operation fallible or GIGO. For example, take a function that would iterate over the string and yield the USVs within it — a borderline trivial operation, but basically a fundamental building block of any higher Unicode function we might want — it cannot process the example string from my previous comment without either raising, or yielding garbage/invalid USVs. Make illegal values actually illegal, and "iterate over the USVs" becomes an infallible method. This is no different than a bool that permits representing a "2", where a "2" has no well-defined meaning. You're just breeding bugs, at that point.
  - mixmastamyk 2y ago
    Python doesn't do runtime type checking (especially internal ranges) and never will. It does find the error at the first invalid operation, just not as early as you'd like perhaps. If this is a non-theoretical problem at your end, a stricter language might well be called for.
    - deathanatos 2y ago
      > Python doesn't do runtime type checking I don't know what you're defining "runtime type checking" as, but it absolutely does. You can't stuff a bool into an int, you can't stuff any arbitrary byte into a str. For the most part, it's strongly typed, but str falls down by allowing non-Unicode values into its set of values. > If this is a non-theoretical problem at your end, a stricter language might well be called for. … Python is plenty well suited to the task, if not for bugs in the implementations of its types.

mixmastamyk 2y ago

On disk source-code is not relevant. To normalize surrogate pairs for later export to UTF8, they must be decoded first: >>> '\ud83d\ude04'.encode('utf-16', 'surrogatepass').decode('utf-16') '' >>> _.encode('utf8') b'\xf0\x9f\x98\x84' (The happy face emoji was removed by HN, run line in terminal to see it.) In other words surrogate pairs are only vaild in the context of UTF16 encoded bytes and no where else. If you're adding them into your program by hand on purpose, you're either doing it wrong or specifically to raise an error for illustration purposes.

deathanatos 2y ago
I'm well aware that surrogates are only valid in the context of UTF-16's encoding. > If you're adding them into your program by hand on purpose, you're either doing it wrong or specifically to raise an error for illustration purposes. In the comments above, it should be obvious that it is for illustrative purposes. The point is that the string type does not catch these errors. It's not that someone is going to "add them to the program … on purpose" … it's that they're going to slip in from bad data, or bad code, not on purpose but because people write bugs and generate garbage data. A good type does not let you represent illegal states. For the same reason null is a bad idea — here, it's not a valid unicode string. The point of a Unicode string type would be to expose operations that one might want to perform on Unicode strings. By permitting invalid values to exist in the set of values the type can represent, you're basically making every operation fallible or GIGO. For example, take a function that would iterate over the string and yield the USVs within it — a borderline trivial operation, but basically a fundamental building block of any higher Unicode function we might want — it cannot process the example string from my previous comment without either raising, or yielding garbage/invalid USVs. Make illegal values actually illegal, and "iterate over the USVs" becomes an infallible method. This is no different than a bool that permits representing a "2", where a "2" has no well-defined meaning. You're just breeding bugs, at that point.
- mixmastamyk 2y ago
  Python doesn't do runtime type checking (especially internal ranges) and never will. It does find the error at the first invalid operation, just not as early as you'd like perhaps. If this is a non-theoretical problem at your end, a stricter language might well be called for.
  - deathanatos 2y ago
    > Python doesn't do runtime type checking I don't know what you're defining "runtime type checking" as, but it absolutely does. You can't stuff a bool into an int, you can't stuff any arbitrary byte into a str. For the most part, it's strongly typed, but str falls down by allowing non-Unicode values into its set of values. > If this is a non-theoretical problem at your end, a stricter language might well be called for. … Python is plenty well suited to the task, if not for bugs in the implementations of its types.