6 ms·

Have you looked at Python 3 yet? I'm using Python 3 in production for an internationalized website and my experience has been that it handles Unicode pretty wel

Have you looked at Python 3 yet? I'm using Python 3 in production for an internationalized website and my experience has been that it handles Unicode pretty well.

SimonSapin 11y ago
http://lucumr.pocoo.org/2014/1/9/ucs-vs-utf8/ http://lucumr.pocoo.org/2014/1/9/ucs-vs-utf8/ is a nice comparison of Python’s (2 and 3) and Rust’s Unicode handling.
WaxProlix 11y ago
There's some disagreement[1] about the direction that Python3 went in terms of handling unicode. Pretty good read if you have a few minutes. 1 http://lucumr.pocoo.org/2014/1/5/unicode-in-2-and-3/ http://lucumr.pocoo.org/2014/1/5/unicode-in-2-and-3/
- copsarebastards 11y ago
  Not that great of a read. Stuff like: > I have been told multiple times now that my point of view is wrong and I don't understand beginners, or that the “text model” has been changed and my request makes no sense. "The text model has changed" is a perfectly legitimate reason to turn down ideas consistent with the previous text model and inconsistent with the current model. Keeping a coherent, consistent model of your text is a pretty important part of curating a language. One of Python's greatest strengths is that they don't just pile on random features, and keeping old crufty features from previous versions would amount to the same thing. To dismiss this reasoning is extremely shortsighted.
- pekk 11y ago
  Many people who prefer Python3's way of handling Unicode are aware of these arguments. It isn't a position based on ignorance.
  - WaxProlix 11y ago
    Hey, never meant to imply otherwise. In fact, even people who have issues with the py3 way often agree that it's still better than 2's.
DasIch 11y ago
Python 3 doesn't handle Unicode any better than Python 2, it just made it the default string. In all other aspects the situation has stayed as bad as it was in Python 2 or has gotten significantly worse. Good examples for that are paths and anything that relates to local IO when you're locale is C.
- masklinn 11y ago
  That is not quite true, in the sense that more of the standard library has been made unicode-aware, and implicit conversions between unicode and bytestrings have been removed. So if you're working in either domain you get a coherent view, the problem being when you're interacting with systems or concepts which straddle the divide or (even worse) may be in either domain depending on the platform. Filesystem paths is the latter, it's text on OSX and Windows — although possibly ill-formed in Windows — but it's bag-o-bytes in most unices. There Python 2 is only "better" in that issues will probably fly under the radar if you don't prod things too much.
  - copsarebastards 11y ago
    > There Python 2 is only "better" in that issues will probably fly under the radar if you don't prod things too much. Ah yes, the JavaScript solution.
  - DasIch 11y ago
    There is no coherent view at all. Bytes still have methods like .upper() that make no sense at all in that context, while unicode strings with these methods are broken because these are locale dependent operations and there is no appropriate API. You can also index, slice and iterate over strings, all operations that you really shouldn't do unless you really now what you are doing. The API in no way indicates that doing any of these things is a problem. Python 2 handling of paths is not good because there is no good abstraction over different operating systems, treating them as byte strings is a sane lowest common denominator though. Python 3 pretends that paths can be represented as unicode strings on all OSes, that's not true. That is held up with a very leaky abstraction and means that Python code that treats paths as unicode strings and not as paths-that-happen-to-be-unicode-but-really-arent is broken. Most people aren't aware of that at all and it's definitely surprising. On top of that implicit coercions have been replaced with implicit broken guessing of encodings for example when opening files.

Have you looked at Python 3 yet? I'm using Python 3 in production for an internationalized website and my experience has been that it handles Unicode pretty well.

SimonSapin 11y ago

http://lucumr.pocoo.org/2014/1/9/ucs-vs-utf8/ http://lucumr.pocoo.org/2014/1/9/ucs-vs-utf8/ is a nice comparison of Python’s (2 and 3) and Rust’s Unicode handling.

WaxProlix 11y ago

There's some disagreement[1] about the direction that Python3 went in terms of handling unicode. Pretty good read if you have a few minutes. 1 http://lucumr.pocoo.org/2014/1/5/unicode-in-2-and-3/ http://lucumr.pocoo.org/2014/1/5/unicode-in-2-and-3/

copsarebastards 11y ago
Not that great of a read. Stuff like: > I have been told multiple times now that my point of view is wrong and I don't understand beginners, or that the “text model” has been changed and my request makes no sense. "The text model has changed" is a perfectly legitimate reason to turn down ideas consistent with the previous text model and inconsistent with the current model. Keeping a coherent, consistent model of your text is a pretty important part of curating a language. One of Python's greatest strengths is that they don't just pile on random features, and keeping old crufty features from previous versions would amount to the same thing. To dismiss this reasoning is extremely shortsighted.
pekk 11y ago
Many people who prefer Python3's way of handling Unicode are aware of these arguments. It isn't a position based on ignorance.
- WaxProlix 11y ago
  Hey, never meant to imply otherwise. In fact, even people who have issues with the py3 way often agree that it's still better than 2's.

DasIch 11y ago

Python 3 doesn't handle Unicode any better than Python 2, it just made it the default string. In all other aspects the situation has stayed as bad as it was in Python 2 or has gotten significantly worse. Good examples for that are paths and anything that relates to local IO when you're locale is C.

masklinn 11y ago
That is not quite true, in the sense that more of the standard library has been made unicode-aware, and implicit conversions between unicode and bytestrings have been removed. So if you're working in either domain you get a coherent view, the problem being when you're interacting with systems or concepts which straddle the divide or (even worse) may be in either domain depending on the platform. Filesystem paths is the latter, it's text on OSX and Windows — although possibly ill-formed in Windows — but it's bag-o-bytes in most unices. There Python 2 is only "better" in that issues will probably fly under the radar if you don't prod things too much.
- copsarebastards 11y ago
  > There Python 2 is only "better" in that issues will probably fly under the radar if you don't prod things too much. Ah yes, the JavaScript solution.
- DasIch 11y ago
  There is no coherent view at all. Bytes still have methods like .upper() that make no sense at all in that context, while unicode strings with these methods are broken because these are locale dependent operations and there is no appropriate API. You can also index, slice and iterate over strings, all operations that you really shouldn't do unless you really now what you are doing. The API in no way indicates that doing any of these things is a problem. Python 2 handling of paths is not good because there is no good abstraction over different operating systems, treating them as byte strings is a sane lowest common denominator though. Python 3 pretends that paths can be represented as unicode strings on all OSes, that's not true. That is held up with a very leaky abstraction and means that Python code that treats paths as unicode strings and not as paths-that-happen-to-be-unicode-but-really-arent is broken. Most people aren't aware of that at all and it's definitely surprising. On top of that implicit coercions have been replaced with implicit broken guessing of encodings for example when opening files.