4 ms·

I'm not a fan of how Python 3 stores Unicode strings internally. In my opinion they should have went with UTF-8. The extra scanning and conversion puts more p

I'm not a fan of how Python 3 stores Unicode strings internally. In my opinion they should have went with UTF-8. The extra scanning and conversion puts more preassure on the processor and caches under load. I agree that Python 2's Unicode handling is broken. That's why I just stored UTF-8 in a normal string and avoided the whole mess. The only thing I have to do is validate any input from the outside world is really UTF-8.

ubernostrum 9y ago
Since the high-level API is supposed to let you treat a string as a sequence of code points, a correct implementation (which Python didn't have until 3.3!) would've imposed the overhead of conversion to something resembling a fixed-width encoding whenever a programmer invoked certain operations. And the vast majority of strings in real-world Python contain only code points also present in latin-1, which means they can be stored in one byte per code point with this approach. And for strings which can't be stored in one byte per code point, you were similarly going to pay the price sooner or later.
- Avernar 9y ago
  > Since the high-level API is supposed to let you treat a string as a sequence of code points, I disagree with that premise. It should operate on grapheme clusters. Operating on code points falls into the same trap as operating on bytes. > a correct implementation (which Python didn't have until 3.3!) would've imposed the overhead of conversion to something resembling a fixed-width encoding whenever a programmer invoked certain operations. Those operations should have been removed. Indexing is the big one that needs fixed width internal representation for speed. Code could have been rewritten to not require indexing. But mechanical translation from Python 2 to 3 was a goal and because of that they couldn't radically change the unicode API for the better. > And the vast majority of strings in real-world Python contain only code points also present in latin-1, which means they can be stored in one byte per code point with this approach. And for strings which can't be stored in one byte per code point, you were similarly going to pay the price sooner or later. You're going to pay the price for 4 byte per codepoint strings quite often. A single emoji will blow up a latin-1 string to 4 times the size.
int_19h 9y ago
> That's why I just stored UTF-8 in a normal string and avoided the whole mess. This only works if every library that you use agrees with you on this, and treats all strings you pass to it as UTF-8 whenever encoding matters. OTOH, if you don't care about that, then you might as well just use bytes everywhere, and get the same thing. At least in Python 3, with bytes, if a library does try to use it as a string, you'll get an error, rather than silent wrong output.
- Avernar 9y ago
  > This only works if every library that you use agrees with you on this, and treats all strings you pass to it as UTF-8 whenever encoding matters. Not exactly. The library just has to treat it as a string and not worry about the encoding (i.e. not try to encode it to/from the unicode type). Only ran into this issue once and the library had an option to return everything as string so not a problem. > At least in Python 3, with bytes, if a library does try to use it as a string, you'll get an error, rather than silent wrong output. Bytes in Python 3 don't support string operators.
  - josteink 9y ago
    > The library just has to treat it as a string and not worry about the encoding (i.e. not try to encode it to/from the unicode type). If I pass a library a string it receives a Unicode string, bytes already decoded using an encoding. It shouldn't be able to re-decode that in any way, whatever that is supposed to mean on a technical level. If a library receives a byte-array representing text, that is a completely different matter and talking about encodings is fully appropriate, even required. But this matter should predominantly exist at your application's barrier, when doing IO. If you're regularly doing encoding and decoding anywhere else, you're doing something wrong (or your language is).
    - Avernar 9y ago
      Look back a few posts. We're discussing using UTF-8 in str and avoiding the unicode type in Python 2. I'n my use case I validate the string as UTF-8 from the internet. To and from the database is UTF-8 so no validation is required there. Output back to the internet requires no additional steps. Nowhere in this method is encode or decode required or desired.

ubernostrum 9y ago

Since the high-level API is supposed to let you treat a string as a sequence of code points, a correct implementation (which Python didn't have until 3.3!) would've imposed the overhead of conversion to something resembling a fixed-width encoding whenever a programmer invoked certain operations. And the vast majority of strings in real-world Python contain only code points also present in latin-1, which means they can be stored in one byte per code point with this approach. And for strings which can't be stored in one byte per code point, you were similarly going to pay the price sooner or later.

Avernar 9y ago
> Since the high-level API is supposed to let you treat a string as a sequence of code points, I disagree with that premise. It should operate on grapheme clusters. Operating on code points falls into the same trap as operating on bytes. > a correct implementation (which Python didn't have until 3.3!) would've imposed the overhead of conversion to something resembling a fixed-width encoding whenever a programmer invoked certain operations. Those operations should have been removed. Indexing is the big one that needs fixed width internal representation for speed. Code could have been rewritten to not require indexing. But mechanical translation from Python 2 to 3 was a goal and because of that they couldn't radically change the unicode API for the better. > And the vast majority of strings in real-world Python contain only code points also present in latin-1, which means they can be stored in one byte per code point with this approach. And for strings which can't be stored in one byte per code point, you were similarly going to pay the price sooner or later. You're going to pay the price for 4 byte per codepoint strings quite often. A single emoji will blow up a latin-1 string to 4 times the size.

int_19h 9y ago

> That's why I just stored UTF-8 in a normal string and avoided the whole mess. This only works if every library that you use agrees with you on this, and treats all strings you pass to it as UTF-8 whenever encoding matters. OTOH, if you don't care about that, then you might as well just use bytes everywhere, and get the same thing. At least in Python 3, with bytes, if a library does try to use it as a string, you'll get an error, rather than silent wrong output.

Avernar 9y ago
> This only works if every library that you use agrees with you on this, and treats all strings you pass to it as UTF-8 whenever encoding matters. Not exactly. The library just has to treat it as a string and not worry about the encoding (i.e. not try to encode it to/from the unicode type). Only ran into this issue once and the library had an option to return everything as string so not a problem. > At least in Python 3, with bytes, if a library does try to use it as a string, you'll get an error, rather than silent wrong output. Bytes in Python 3 don't support string operators.
- josteink 9y ago
  > The library just has to treat it as a string and not worry about the encoding (i.e. not try to encode it to/from the unicode type). If I pass a library a string it receives a Unicode string, bytes already decoded using an encoding. It shouldn't be able to re-decode that in any way, whatever that is supposed to mean on a technical level. If a library receives a byte-array representing text, that is a completely different matter and talking about encodings is fully appropriate, even required. But this matter should predominantly exist at your application's barrier, when doing IO. If you're regularly doing encoding and decoding anywhere else, you're doing something wrong (or your language is).
  - Avernar 9y ago
    Look back a few posts. We're discussing using UTF-8 in str and avoiding the unicode type in Python 2. I'n my use case I validate the string as UTF-8 from the internet. To and from the database is UTF-8 so no validation is required there. Output back to the internet requires no additional steps. Nowhere in this method is encode or decode required or desired.