3 ms·

> The only problem with Python 3’s str is that you don’t grok it Sadly this quote shows a fundamental lack of understanding of the problems with the Python 3 s

> The only problem with Python 3’s str is that you don’t grok it Sadly this quote shows a fundamental lack of understanding of the problems with the Python 3 string type.

nicky0 10y ago
Is this a comment on the article or just its title?
- the_mitsuhiko 10y ago
  It's a comment on the title but also the article is not even talking about the various issues with Unicode in Python.
  - ageofwant 10y ago
    The only issue I ever had with Python's Unicode support was due to my own lack of understanding, a direct consequence of failing to read the docs, because I'm lazy and opinionated, and have a bad reaction when reality differs from what it obviously should be, also I'm intolerant to adversity.
ThePhysicist 10y ago
The problem is that in Python 2, "str" is a bytestring and "unicode" is a Unicode string, whereas in Python 3 "str" is a Unicode string and "bytes" is a bytestring. In my experience this is the main reason why Python 2 users get confused. I can understand why the renaming might have been a good idea (to avoid changing the semantic meaning of an expression while leaving the syntax intact), but I can understand why people don't "grok it" immediately, though it is rather simple if you know it.
- the_mitsuhiko 10y ago
  > The problem is that in Python 2, "str" is a bytestring and "unicode" is a Unicode string, whereas in Python 3 "str" is a Unicode string and "bytes" is a bytestring. `str` in Python 3 is `unicode` in Python 2. The bytestring type was removed and replaced with a bytes type. Not sure what's particularly confusing about this. None of this however is the issue with Unicode in Python in general. I think all these articles fail to understand that the actual story behind unicode in the language is very complex.
  - ThePhysicist 10y ago
    Yes that's what I said, and there's nothing "wrong" with it, I just think that it's where most of the confusion comes from, as many people that come from Python 2 think of `str` as a byte-string, while in Python 3 it's a Unicode string. I think the "unicode-by-default" approach of Python 3 is a great reason to switch to it (and I did so for almost all my codebases), converting code can sometimes cause surprises though, especially due to the lack of type checks in Python combined with the more stricter checking that Python 3 does when working with strings (i.e. adding Unicode and byte strings throws an exception in Python 3, whereas Python 2 would simply try to convert the byte-string implicitly into a string and only throw an exception if this fails). All theses changes combined make converting code from 2 to 3 more risky. I think especially the handling of type coercion between Unicode and bytes objects could have been handled differently, as it is (IMO) a big semantic change. As an example, just today I ran into a problem with WTForms and Flask (which is a great framework btw, thanks for writing it!), where I used a cookie-based session store provided by `itsdangerous`, which deserializes a value stored in the session either into a byte-string or a Unicode string. After switching my code from Python 2 to Python 3, there were still some clients in the wild that had a session object in their cookie generated by Python 2 containing byte-strings, and `itsdangerous` happily deserialized it into a byte-string as well, causing an exception when my Python 3 code tried adding it to a Unicode string. With new sessions this didn't happen of course (as the objects were serialized as Unicode), which made this thing quite fun to debug. Granted, this is not a problem with Unicode in Python 3, but I think it illustrates the kind of challenges you face when migrating codebases from one to the other. What I can agree on is that we'd be much better off if everyone was using Python 3, and eventually we'll get there.
  - rtpg 10y ago
    I really wish they named it something different. Like `text` and `bytes` In that scenario you have no name link between Py2 objects and Py3 objects. The `str` name reuse for different objects makes the 2-3 transition that much harder.
jstimpfle 10y ago
Downvoted for surprising lack of substance. Note I do read many of your blog posts -- and thanks for writing them, there is a lot of insight for me. I also did read most or all of your blog posts about python3 unicode handling. The thing is, while most or all of the facts presented there are "true", many of the negative conclusions there are just your opinion that stems from years and years of doing it "your way" (I would call it FUD, but I have strong opinions too). Python2's unpredictable implicit conversions are hardly "sane" (that's one of the unjustified claims made there). Do you also call Javascript's "==" sane? For me Python3 str has worked like a charm for years. I like the strict separation of high and low level affairs. I like how I can treat files as text files (deal only with python3 strs) and don't have to think about low level affairs, or types, or conversions, which is a great boon for scripting. I like how I can drop down to the bytes level when needed and know exactly what I'm at. For some balance, I wouldn't expect that conversion to python3 is always easy. But I would blame python2 for missing clean concepts, not python3. Also, it's hardly convenient to code most of a Python3 app at the byte level. But it could be argued that python should not be used for these things.
- ThePhysicist 10y ago
  I agree that having stricter coercion rules makes sense in situations where the coercion might be ambiguous, whether adding byte-strings and Unicode strings together is such a case is debatable though (I think it is), as in many cases implicit conversion yields an acceptable outcome. Again, this is more a question of design philosophy, but the thing with Python (2) is that implicit coercion was the default behavior in many cases, so changing that is painful. Personally I think we should double-down on type annotations and stronger (optional) typing for Python, because the lack of a good type system is by far the biggest obstacle for building robust, large systems in Python.
ageofwant 10y ago
Your comment would be more constructive if you could expand on what, in your opinion, Python 3's str failings are.
- cwyers 10y ago
  This blog post may illuminate you (commenter is the author, IIRC): http://lucumr.pocoo.org/2014/5/12/everything-about-unicode/ http://lucumr.pocoo.org/2014/5/12/everything-about-unicode/

> The only problem with Python 3’s str is that you don’t grok it Sadly this quote shows a fundamental lack of understanding of the problems with the Python 3 string type.

nicky0 10y ago

Is this a comment on the article or just its title?

the_mitsuhiko 10y ago
It's a comment on the title but also the article is not even talking about the various issues with Unicode in Python.
- ageofwant 10y ago
  The only issue I ever had with Python's Unicode support was due to my own lack of understanding, a direct consequence of failing to read the docs, because I'm lazy and opinionated, and have a bad reaction when reality differs from what it obviously should be, also I'm intolerant to adversity.

ThePhysicist 10y ago

The problem is that in Python 2, "str" is a bytestring and "unicode" is a Unicode string, whereas in Python 3 "str" is a Unicode string and "bytes" is a bytestring. In my experience this is the main reason why Python 2 users get confused. I can understand why the renaming might have been a good idea (to avoid changing the semantic meaning of an expression while leaving the syntax intact), but I can understand why people don't "grok it" immediately, though it is rather simple if you know it.

the_mitsuhiko 10y ago
> The problem is that in Python 2, "str" is a bytestring and "unicode" is a Unicode string, whereas in Python 3 "str" is a Unicode string and "bytes" is a bytestring. `str` in Python 3 is `unicode` in Python 2. The bytestring type was removed and replaced with a bytes type. Not sure what's particularly confusing about this. None of this however is the issue with Unicode in Python in general. I think all these articles fail to understand that the actual story behind unicode in the language is very complex.
- ThePhysicist 10y ago
  Yes that's what I said, and there's nothing "wrong" with it, I just think that it's where most of the confusion comes from, as many people that come from Python 2 think of `str` as a byte-string, while in Python 3 it's a Unicode string. I think the "unicode-by-default" approach of Python 3 is a great reason to switch to it (and I did so for almost all my codebases), converting code can sometimes cause surprises though, especially due to the lack of type checks in Python combined with the more stricter checking that Python 3 does when working with strings (i.e. adding Unicode and byte strings throws an exception in Python 3, whereas Python 2 would simply try to convert the byte-string implicitly into a string and only throw an exception if this fails). All theses changes combined make converting code from 2 to 3 more risky. I think especially the handling of type coercion between Unicode and bytes objects could have been handled differently, as it is (IMO) a big semantic change. As an example, just today I ran into a problem with WTForms and Flask (which is a great framework btw, thanks for writing it!), where I used a cookie-based session store provided by `itsdangerous`, which deserializes a value stored in the session either into a byte-string or a Unicode string. After switching my code from Python 2 to Python 3, there were still some clients in the wild that had a session object in their cookie generated by Python 2 containing byte-strings, and `itsdangerous` happily deserialized it into a byte-string as well, causing an exception when my Python 3 code tried adding it to a Unicode string. With new sessions this didn't happen of course (as the objects were serialized as Unicode), which made this thing quite fun to debug. Granted, this is not a problem with Unicode in Python 3, but I think it illustrates the kind of challenges you face when migrating codebases from one to the other. What I can agree on is that we'd be much better off if everyone was using Python 3, and eventually we'll get there.
- rtpg 10y ago
  I really wish they named it something different. Like `text` and `bytes` In that scenario you have no name link between Py2 objects and Py3 objects. The `str` name reuse for different objects makes the 2-3 transition that much harder.

jstimpfle 10y ago

Downvoted for surprising lack of substance. Note I do read many of your blog posts -- and thanks for writing them, there is a lot of insight for me. I also did read most or all of your blog posts about python3 unicode handling. The thing is, while most or all of the facts presented there are "true", many of the negative conclusions there are just your opinion that stems from years and years of doing it "your way" (I would call it FUD, but I have strong opinions too). Python2's unpredictable implicit conversions are hardly "sane" (that's one of the unjustified claims made there). Do you also call Javascript's "==" sane? For me Python3 str has worked like a charm for years. I like the strict separation of high and low level affairs. I like how I can treat files as text files (deal only with python3 strs) and don't have to think about low level affairs, or types, or conversions, which is a great boon for scripting. I like how I can drop down to the bytes level when needed and know exactly what I'm at. For some balance, I wouldn't expect that conversion to python3 is always easy. But I would blame python2 for missing clean concepts, not python3. Also, it's hardly convenient to code most of a Python3 app at the byte level. But it could be argued that python should not be used for these things.

ThePhysicist 10y ago
I agree that having stricter coercion rules makes sense in situations where the coercion might be ambiguous, whether adding byte-strings and Unicode strings together is such a case is debatable though (I think it is), as in many cases implicit conversion yields an acceptable outcome. Again, this is more a question of design philosophy, but the thing with Python (2) is that implicit coercion was the default behavior in many cases, so changing that is painful. Personally I think we should double-down on type annotations and stronger (optional) typing for Python, because the lack of a good type system is by far the biggest obstacle for building robust, large systems in Python.

ageofwant 10y ago

Your comment would be more constructive if you could expand on what, in your opinion, Python 3's str failings are.

cwyers 10y ago
This blog post may illuminate you (commenter is the author, IIRC): http://lucumr.pocoo.org/2014/5/12/everything-about-unicode/ http://lucumr.pocoo.org/2014/5/12/everything-about-unicode/