3 ms·

> In Rust, strings are always valid UTF8, and attempting to create a string with invalid UTF8 will panic at runtime: > [piece of code explicitly calling .unwra

• lucb1e 2y ago
  There are countries where Python 3's stable release is old enough to legally get married, and this change was being planned since 2004. It's not that recent! This is the oldest concrete plan I can quickly spot: https://github.com/python/peps/blob/b5815e3e638834e28233fc2043026707c5bf5e95/pep-3000.txt#L44 https://github.com/python/peps/blob/b5815e3e638834e28233fc20...
  • remram 2y ago
    Julia is not so different, implementation started in 2009 (I don't know how far before it was planned). Why is it new and Python 3 old? https://github.com/JuliaLang/julia/commit/a9cbc036ac62dc5ba5200416ca7b40a2f9aa59ea https://github.com/JuliaLang/julia/commit/a9cbc036ac62dc5ba5...
• mixmastamyk 2y ago
  Python strings are Unicode, not utf8, at least until encoding time. At which point they become bytes.
  • deathanatos 2y ago
    I'm going to define a "Unicode string" as Rust does: a sequence of USVs / content that can be validly represented as UTF-8. Thus, no, sadly, Python's strings are not Unicode, as they're a sequence of Unicode code points. Because of that, a_string.encode('utf-8') … can raise in Python. For example: In [1]: '\uD83D'.encode('utf-8') --------------------------------------------------------------------------- UnicodeEncodeError Traceback (most recent call last) Cell In[1], line 1 ----> 1 '\uD83D'.encode('utf-8') UnicodeEncodeError: 'utf-8' codec can't encode character '\ud83d' in position 0: surrogates not allowed (The underlying encoding of str in Python these days is either [u8], [u16], or [u32], essentially, depending on the value of the largest code point in the string. So, for some values, e.g., 'hello world', the underlying representation is UTF-8, essentially.)
    • mixmastamyk 2y ago
      Surrogate pairs are not allowed in UTF8 (only in UTF16), so this error is not surprising. It must be decoded from UTF16, then reencoded to UTF8. The in-memory representation is an implementation detail, and can be anything that works, as you described.
      • deathanatos 2y ago
        > Surrogate pairs are not allowed in UTF8 (only in UTF16), so this error is not surprising. It must be decoded from UTF16, then reencoded to UTF8. The source code here is whatever your on-disk .py file is, likely ASCII, so UTF-8. No re-encoding is required, in practice or mentally. > The in-memory representation is an implementation detail, and can be anything that works, as you described. The in memory representation is not really the issue; it's the type itself. Consider a type is a set of possible values: bool, for example, is the set {true, false}. u8 is a set containing {0, 1, … 255}, and things like strings are infinite sets. The set of values that `str` represents is not the same set as what UTF-8 can encode / is not the same set of values as a type that encodes any possible sequence of USVs. The set of all possible `str` instances in Rust != the set of all possible `str` instances in Python. Rust is pretty unique in this regard: I don't think I know of another language whose string type is a sequence of USVs. (Though I assume there is likely one out there.) It is worse, too: there's JS's string type, for example, which is a sequence of UTF-16 code units, which is different yet! So (JS string) ≠ (Python str) ≠ (Rust str)
        • mixmastamyk 2y ago
          On disk source-code is not relevant. To normalize surrogate pairs for later export to UTF8, they must be decoded first: >>> '\ud83d\ude04'.encode('utf-16', 'surrogatepass').decode('utf-16') '' >>> _.encode('utf8') b'\xf0\x9f\x98\x84' (The happy face emoji was removed by HN, run line in terminal to see it.) In other words surrogate pairs are only vaild in the context of UTF16 encoded bytes and no where else. If you're adding them into your program by hand on purpose, you're either doing it wrong or specifically to raise an error for illustration purposes.