3 ms·

This is a really bad way of thinking. The distinction in Python 3 is between text (str) and bytes. str is not Unicode in fact if you don't use fancy characters

by takeda 7y ago

This is a really bad way of thinking. The distinction in Python 3 is between text (str) and bytes. str is not Unicode in fact if you don't use fancy characters internally it stores text as a byte array. You should think of text the same as of image or sound, what you see in the screen or hear in the speaker is the actual thing, but if you need to save it on disk you encode it as for example png or wav.

owl57 7y ago
You can just read "requires text when it shouldn't". But I don't recommend this terminology: in most modern computer programs, including Python 3 implementations, "text" and "Unicode" mean the same thing, but outside of this context Unicode is just more precise: sometimes "text" means ASCII and sometimes it means things non-represantable in current version of Unicode.
pdonis 7y ago
> The distinction in Python 3 is between text (str) and bytes. Feel free to s/Unicode/str/ in what I posted if you prefer that terminology. The problem is still the same. An example of the problem: Python's standard streams (stdin|out|err) in Python 2 are streams of bytes, but in Python 3 they're streams of Unicode (or str if you prefer that terminology) characters. The problem is twofold: first, if my standard streams are hooked to a console, Python can't always properly detect the encoding of the bytes coming from the console, so it can give me the wrong Unicode characters; second, if my standard streams are hooked to pipes, there is no encoding it can pick that is right, since the bytes aren't even coming from a console (where at least there is some plausible argument for saying the user meant to type Unicode characters, not bytes). What Python 3 should have done was keep the standard streams as bytes, since that's the only common denominator you can rely on, and then let the application decide how to decode them if it decides it needs to, just as in Python 2.
- takeda 7y ago
  I believe the behavior is correct though. Python uses encoding specified through LANG/LC_* which is the encoding that supposed to be used, and all properly behaved applications use it. If your application works on binary data, you can use sys.stdin/out/err.buffer to get binary version. Most people will use it for text, so the defaults make sense. Personally I would like if there was no automatic conversion when using files/network/pipes etc. but I guess that would make it more confusing for new users, and would be unnecessary boilerplate for most use cases.
  - pdonis 7y ago
    > Python uses encoding specified through LANG/LC_ Yes, that's the best you can do, but it's still not always correct. I agree that it should be, but "should be" and "is" aren't always the same. > If your application works on binary data, you can use sys.stdin/out/err.buffer to get binary version. Yes, but there are still standard library functions that will use the regular streams, and that might conflict with what your application is doing. There is no way to tell Python as a whole "use binary streams everywhere because they are pipes for this application". > Personally I would like if there was no automatic conversion when using files/network/pipes etc. That would work if (a) Python could always detect that condition (it can't) and (b) the entire standard library adjusted itself accordingly. > I guess that would make it more confusing for new users, and would be unnecessary boilerplate for most use cases. Python 2 worked fine with the standard streams being binary, and applications wrapping them to decode to Unicode when necessary. Python 2.7 even back ported the TextIOWrapper and similar classes to make the wrapping as simple as possible. A similar approach could have been taken in Python 3 (binary streams and a simple wrapper class), but it wasn't.

by takeda 7y ago

owl57 7y ago

You can just read "requires text when it shouldn't". But I don't recommend this terminology: in most modern computer programs, including Python 3 implementations, "text" and "Unicode" mean the same thing, but outside of this context Unicode is just more precise: sometimes "text" means ASCII and sometimes it means things non-represantable in current version of Unicode.

pdonis 7y ago

> The distinction in Python 3 is between text (str) and bytes. Feel free to s/Unicode/str/ in what I posted if you prefer that terminology. The problem is still the same. An example of the problem: Python's standard streams (stdin|out|err) in Python 2 are streams of bytes, but in Python 3 they're streams of Unicode (or str if you prefer that terminology) characters. The problem is twofold: first, if my standard streams are hooked to a console, Python can't always properly detect the encoding of the bytes coming from the console, so it can give me the wrong Unicode characters; second, if my standard streams are hooked to pipes, there is no encoding it can pick that is right, since the bytes aren't even coming from a console (where at least there is some plausible argument for saying the user meant to type Unicode characters, not bytes). What Python 3 should have done was keep the standard streams as bytes, since that's the only common denominator you can rely on, and then let the application decide how to decode them if it decides it needs to, just as in Python 2.

takeda 7y ago
I believe the behavior is correct though. Python uses encoding specified through LANG/LC_* which is the encoding that supposed to be used, and all properly behaved applications use it. If your application works on binary data, you can use sys.stdin/out/err.buffer to get binary version. Most people will use it for text, so the defaults make sense. Personally I would like if there was no automatic conversion when using files/network/pipes etc. but I guess that would make it more confusing for new users, and would be unnecessary boilerplate for most use cases.
- pdonis 7y ago
  > Python uses encoding specified through LANG/LC_ Yes, that's the best you can do, but it's still not always correct. I agree that it should be, but "should be" and "is" aren't always the same. > If your application works on binary data, you can use sys.stdin/out/err.buffer to get binary version. Yes, but there are still standard library functions that will use the regular streams, and that might conflict with what your application is doing. There is no way to tell Python as a whole "use binary streams everywhere because they are pipes for this application". > Personally I would like if there was no automatic conversion when using files/network/pipes etc. That would work if (a) Python could always detect that condition (it can't) and (b) the entire standard library adjusted itself accordingly. > I guess that would make it more confusing for new users, and would be unnecessary boilerplate for most use cases. Python 2 worked fine with the standard streams being binary, and applications wrapping them to decode to Unicode when necessary. Python 2.7 even back ported the TextIOWrapper and similar classes to make the wrapping as simple as possible. A similar approach could have been taken in Python 3 (binary streams and a simple wrapper class), but it wasn't.