4 ms·

Well, it could be worse. Python did the whole Py3k own-goal purely for the purpose of forcing everyone onto UTF-16. (Facepalm, yes.)

sterwill 6y ago
I wasn't following the Python 2-to-3 transition that closely, but I don't think Python ever forced anyone to use UTF-16. On Unix platforms (the only place I've used Python for 10 years) the strings are UTF-8 internally. On Windows maybe it's UTF-16, I don't know, but most programmers don't need to know or care. They just bytes(my_string, 'utf-8') to encode it as bytes.
zinekeller 6y ago
> for the purpose of forcing everyone onto UTF-16. Honestly, Py2 was a PITA when handling raw data, it could corrupt your data if you don't know exactly what you are doing. The goal was to separate (Unicode) text from binary data. It wasn't to UTF-16 though. In fact, you should just assume that text variables are encoded in Unicode points and not care whether it is UTF-16 or UTF-8 (and on Unix-like systems, it is definitely represented to UTF-8). If you are converting it into binary, at least you know what encoding is it: no "Oh no my Python code was broken on Windows/Unix" because even Py2 has already the UTF-16/UTF-8 OS split.
- otabdeveloper4 6y ago
  The easy and sane path would have been to just mandate that str is encoded as UTF-8. (Indeed, that's what every other sane language seems to do.) Instead they tried to split every function into a 'wide' (str) and a 'narrow' (bytes) version, like in Windows. The whole idea of 'wide' strings is predicated on the idea that Unicode charpoints are only ever two bytes and that two bytes is all you ever need. This obviously doesn't work in 2020, and the Python folks tried to roll back their broken by design code that has immediately turned into technical debt right out the gate, but the warts still exist. (Like having to choose whether you open files with 'w' or 'wb', etc. No other language does this stupid thing, I think.)
7786655 6y ago
Python uses UCS-4.

sterwill 6y ago

I wasn't following the Python 2-to-3 transition that closely, but I don't think Python ever forced anyone to use UTF-16. On Unix platforms (the only place I've used Python for 10 years) the strings are UTF-8 internally. On Windows maybe it's UTF-16, I don't know, but most programmers don't need to know or care. They just bytes(my_string, 'utf-8') to encode it as bytes.

zinekeller 6y ago

> for the purpose of forcing everyone onto UTF-16. Honestly, Py2 was a PITA when handling raw data, it could corrupt your data if you don't know exactly what you are doing. The goal was to separate (Unicode) text from binary data. It wasn't to UTF-16 though. In fact, you should just assume that text variables are encoded in Unicode points and not care whether it is UTF-16 or UTF-8 (and on Unix-like systems, it is definitely represented to UTF-8). If you are converting it into binary, at least you know what encoding is it: no "Oh no my Python code was broken on Windows/Unix" because even Py2 has already the UTF-16/UTF-8 OS split.

otabdeveloper4 6y ago
The easy and sane path would have been to just mandate that str is encoded as UTF-8. (Indeed, that's what every other sane language seems to do.) Instead they tried to split every function into a 'wide' (str) and a 'narrow' (bytes) version, like in Windows. The whole idea of 'wide' strings is predicated on the idea that Unicode charpoints are only ever two bytes and that two bytes is all you ever need. This obviously doesn't work in 2020, and the Python folks tried to roll back their broken by design code that has immediately turned into technical debt right out the gate, but the warts still exist. (Like having to choose whether you open files with 'w' or 'wb', etc. No other language does this stupid thing, I think.)

7786655 6y ago

Python uses UCS-4.