5 ms·

Apparently you don't deal with much non-ascii data if you don't like python3's string handling. It's literally the reason I first upgraded, because reading in s

Apparently you don't deal with much non-ascii data if you don't like python3's string handling. It's literally the reason I first upgraded, because reading in simple text data on Python2 can break completely. Seeing as literally no language on earth can be written entirely in ascii characters (since even English text can contain words with accents or old fashioned things like the long s ſ) this is a pretty big issue. Yes it is possibly to laboriously add code to check and encode/decode text, that might sometime be text and sometimes binary pretending to be text pretending to be binary. But why go through that pain when you could just use Python3 where things are clear and generally actually work.

eeZi 11y ago
I fully agree. Python 3 makes life much, much easier with non-ASCII languages and prevents the annoying Unicode conversion bugs you see in most Python 2 code bases. It's also very intuitive, which is great for teaching - I've always had a hard time explaining the difference between "strings" and "unicode" objects in Python 2, but the clear boundary makes it straight-forward in Python 3.
- SeanDav 11y ago
  Not always straight forward in Python 3. Try explaining to your class how to ignore case in string comparisons.
  - zo1 11y ago
    That depends on the type of strings you are comparing. If you include the full Unicode set, you're going to run in to huge troubles in probably all languages simply due to the nature of the Unicode character set, rather than the specific language's handling of strings and Unicode code points. But, personally, if you're teaching Python to a class, whether it's 2.x or 3.x. You should probably stick to using some variation of string.upper() == string.upper() to explain simple comparisons, due to simplicity.
  - squeaky-clean 11y ago
    >Try explaining to your class how to ignore case in string comparisons Okay: You can't. You'll need to convert both strings to the same case, and then compare those. In the majority of cases, you can do: return string_a.lower() == string_b.lower() But sometimes that won't work, because converting case isn't always easy. And that's a unicode issue, not a python issue. It's not even completely a unicode issue, just a "writing can be very weird" issue. For example, "ß" (U+00DF) is a character that doesn't/shouldn't have an uppercase form. Unicode does support it as U+1E9E, but all the examples I know of will convert it to "SS" ([U+0053, U+0053]) instead. Try this in a javascript console, for example. > "ß".toUpperCase().toLowerCase() === "ß".toLowerCase() false > "ß".toUpperCase() === "SS".toUpperCase() true > "ß".toLowerCase() === "SS".toLowerCase() false > "ß".length 1 > "ß".toLowerCase().length 1 > "ß".toUpperCase().length 2 > "ß".toUpperCase().toLowerCase() "ss" Also, try ctrl+f'ing for "ss" on the Wiki page for ß [0] and see what it matches. Or ctrl+f this page for "ß" and it will match every "ss". I think it's pretty neat. [0] https://en.wikipedia.org/wiki/%C3%9F https://en.wikipedia.org/wiki/%C3%9F (edit: You might already know everything in this comment, but I started writing and it ended up being a cool exercise for myself, so I figured I'd submit).
    - throw_away 11y ago
      Interesting that the "ss" search only works on safari and chrome, but not on firefox (which happens to be the only browser that lets you easily toggle case-matching AFAICT). I presume this means that the first two went with a toUpperCase() approach while FF went with toLowerCase(). I wonder which choice is better in the face of unicode type oddities like this one.
    - mangecoeur 11y ago
      > "writing can be very weird" Absolutely! I think there's a lot of baggage in computing because it happened to emerge in places where language could mostly be written in simple characters with surprisingly simple rules. When exported internationally suddenly you deal with "oddities" that are really rather the norm in other languages. In French for example is customary NOT to put accents on capital letters, even though it's possible. But then again sometime people do it anyway - it's not a hard rule. So often converting to lowercase may require contextual understanding because some words are written the same but with different accents, and would capitalize to the same string. Which word was meant would only be evident from the context of the sentence. I suspect that if computing had rather developed somewhere where these things were commonplace we would have much better handling of it today.
  - c-oreills 11y ago
    This is exactly what str.casefold is for: https://docs.python.org/3/library/stdtypes.html#str.casefold https://docs.python.org/3/library/stdtypes.html#str.casefold
    - SeanDav 11y ago
      casefold is only part of the story and by itself does not supply a complete solution.
      - c-oreills 11y ago
        Could you expand on where it falls short? I'm interested, having not yet relied on it heavily.
- takeda 11y ago
  This is also significant reason why people have problem when migrating py2 code to py3. The code is actually buggy and mixes strings with bytes. Once they run it in Py3 all of the bugs show up because Py3 is stricter about it. For the same reason if you need to write code that should work on Py2 and Py3, it's much easier to write it on Py3 and then add backward compatibility through "from __future__ import ..." and the six module.
  - eeZi 11y ago
    I've heard lots of good things about this project, by the way: http://python-future.org/ http://python-future.org/ It's much less clunky than six. Instead of manually adding the compatibility layers to your code, it automates the process, which results in cleaner code. It can even translate (some) Python 2 modules on-the-fly. Many projects switched to it from six.
yepguy 11y ago
Python 3 is okay-ish for developers who don't think about encodings unless forced to, but it actually makes it more difficult to do "the right thing": http://lucumr.pocoo.org/2014/5/12/everything-about-unicode/ http://lucumr.pocoo.org/2014/5/12/everything-about-unicode/
- vetinari 11y ago
  Having to thing about encoding all the time, without being forced to, brings in a lot of unwanted accidental complexity. Did you know, that there is an operating system, where a simple loop that reads lines from file and prints them to the console will end up with errors? (that being Windows and some European code pages, where console is cp85x, filesystem read and write cp125x).
  - mark-r 11y ago
    The accidental complexity here is due to Windows, not Python. Python would rather generate an error than give you garbled output without your knowledge.
    - vetinari 11y ago
      Most developers are not aware of that and write code, that just bombs out for the user. The user would prefer at least garbled text instead of exception, because he is used to that anyway in other apps. He has at least some result, instead of stack trace. Well, and that's before the user tries to redirect stdout to file, in which case it will end up with error too, because that is going to be 7-bit ascii. Anyway, Python3 does the right thing with defining the encoding at the edges and everything works right for the user out of the box.
krick 11y ago
I would suspect that you don't deal with much non-ascii data if you like python3's string handling. In Python 2 you have control over encodings. You can work with unicode-strings internally, but you receive and produce bytes, which is actually the only truth in the computer world. Everything else is a lie. It would be nice to pretend that there's only 1 encoding in the world and that encoding is roughly equivalent to unicode itself, but this isn't true. World doesn't work on UTF-8. By abstracting encodings Python 3 obscures what's actually happening.
- zo1 11y ago
  Not at all, it forces you to unify to storing character data as Unicode code-points. Eventually if you go down far enough, you're reading the raw bytes you speak of from some sort of external system/file/DB. It's at that point that you define your encoding, and never worry about it every again. Except for when you touch those external systems/files again, again with full knowledge about what encoding is required. Either way, if you're not dealing with differently encoded character data in Python 2.x as noted above, you're causing yourself some unnecessary headaches.
- vetinari 11y ago
  In Python2, I couldn't just print the string I just read from file - such were the joys of Windows console, which has different encoding than the rest of the system (cp85x in console, cp125x in filesystem & gui)... Compared to that, Python3 was like gift from heaven.
- takeda 11y ago
  Something tells me you did not experience the pre-unicode world around 90s. I did, and I didn't like it. I prefer it is abstracted from me thankyouverymuch. But seriously, all what python (and as a matter of the fact majority of languages) is placing a clear distinction between a byte and a character. This forces developers to keep in mind what they are dealing with and reduces number of bugs. The truth is that if you treat everything as bytes, chances are that your program has bugs. Actually good test is to convert your existing Py2 code to Py3, if you have encoding/decoding errors then your code was buggy.

eeZi 11y ago

I fully agree. Python 3 makes life much, much easier with non-ASCII languages and prevents the annoying Unicode conversion bugs you see in most Python 2 code bases. It's also very intuitive, which is great for teaching - I've always had a hard time explaining the difference between "strings" and "unicode" objects in Python 2, but the clear boundary makes it straight-forward in Python 3.

SeanDav 11y ago
Not always straight forward in Python 3. Try explaining to your class how to ignore case in string comparisons.
- zo1 11y ago
  That depends on the type of strings you are comparing. If you include the full Unicode set, you're going to run in to huge troubles in probably all languages simply due to the nature of the Unicode character set, rather than the specific language's handling of strings and Unicode code points. But, personally, if you're teaching Python to a class, whether it's 2.x or 3.x. You should probably stick to using some variation of string.upper() == string.upper() to explain simple comparisons, due to simplicity.
- squeaky-clean 11y ago
  >Try explaining to your class how to ignore case in string comparisons Okay: You can't. You'll need to convert both strings to the same case, and then compare those. In the majority of cases, you can do: return string_a.lower() == string_b.lower() But sometimes that won't work, because converting case isn't always easy. And that's a unicode issue, not a python issue. It's not even completely a unicode issue, just a "writing can be very weird" issue. For example, "ß" (U+00DF) is a character that doesn't/shouldn't have an uppercase form. Unicode does support it as U+1E9E, but all the examples I know of will convert it to "SS" ([U+0053, U+0053]) instead. Try this in a javascript console, for example. > "ß".toUpperCase().toLowerCase() === "ß".toLowerCase() false > "ß".toUpperCase() === "SS".toUpperCase() true > "ß".toLowerCase() === "SS".toLowerCase() false > "ß".length 1 > "ß".toLowerCase().length 1 > "ß".toUpperCase().length 2 > "ß".toUpperCase().toLowerCase() "ss" Also, try ctrl+f'ing for "ss" on the Wiki page for ß [0] and see what it matches. Or ctrl+f this page for "ß" and it will match every "ss". I think it's pretty neat. [0] https://en.wikipedia.org/wiki/%C3%9F https://en.wikipedia.org/wiki/%C3%9F (edit: You might already know everything in this comment, but I started writing and it ended up being a cool exercise for myself, so I figured I'd submit).
  - throw_away 11y ago
    Interesting that the "ss" search only works on safari and chrome, but not on firefox (which happens to be the only browser that lets you easily toggle case-matching AFAICT). I presume this means that the first two went with a toUpperCase() approach while FF went with toLowerCase(). I wonder which choice is better in the face of unicode type oddities like this one.
  - mangecoeur 11y ago
    > "writing can be very weird" Absolutely! I think there's a lot of baggage in computing because it happened to emerge in places where language could mostly be written in simple characters with surprisingly simple rules. When exported internationally suddenly you deal with "oddities" that are really rather the norm in other languages. In French for example is customary NOT to put accents on capital letters, even though it's possible. But then again sometime people do it anyway - it's not a hard rule. So often converting to lowercase may require contextual understanding because some words are written the same but with different accents, and would capitalize to the same string. Which word was meant would only be evident from the context of the sentence. I suspect that if computing had rather developed somewhere where these things were commonplace we would have much better handling of it today.
- c-oreills 11y ago
  This is exactly what str.casefold is for: https://docs.python.org/3/library/stdtypes.html#str.casefold https://docs.python.org/3/library/stdtypes.html#str.casefold
  - SeanDav 11y ago
    casefold is only part of the story and by itself does not supply a complete solution.
    - c-oreills 11y ago
      Could you expand on where it falls short? I'm interested, having not yet relied on it heavily.
takeda 11y ago
This is also significant reason why people have problem when migrating py2 code to py3. The code is actually buggy and mixes strings with bytes. Once they run it in Py3 all of the bugs show up because Py3 is stricter about it. For the same reason if you need to write code that should work on Py2 and Py3, it's much easier to write it on Py3 and then add backward compatibility through "from __future__ import ..." and the six module.
- eeZi 11y ago
  I've heard lots of good things about this project, by the way: http://python-future.org/ http://python-future.org/ It's much less clunky than six. Instead of manually adding the compatibility layers to your code, it automates the process, which results in cleaner code. It can even translate (some) Python 2 modules on-the-fly. Many projects switched to it from six.

yepguy 11y ago

Python 3 is okay-ish for developers who don't think about encodings unless forced to, but it actually makes it more difficult to do "the right thing": http://lucumr.pocoo.org/2014/5/12/everything-about-unicode/ http://lucumr.pocoo.org/2014/5/12/everything-about-unicode/

vetinari 11y ago
Having to thing about encoding all the time, without being forced to, brings in a lot of unwanted accidental complexity. Did you know, that there is an operating system, where a simple loop that reads lines from file and prints them to the console will end up with errors? (that being Windows and some European code pages, where console is cp85x, filesystem read and write cp125x).
- mark-r 11y ago
  The accidental complexity here is due to Windows, not Python. Python would rather generate an error than give you garbled output without your knowledge.
  - vetinari 11y ago
    Most developers are not aware of that and write code, that just bombs out for the user. The user would prefer at least garbled text instead of exception, because he is used to that anyway in other apps. He has at least some result, instead of stack trace. Well, and that's before the user tries to redirect stdout to file, in which case it will end up with error too, because that is going to be 7-bit ascii. Anyway, Python3 does the right thing with defining the encoding at the edges and everything works right for the user out of the box.

krick 11y ago

I would suspect that you don't deal with much non-ascii data if you like python3's string handling. In Python 2 you have control over encodings. You can work with unicode-strings internally, but you receive and produce bytes, which is actually the only truth in the computer world. Everything else is a lie. It would be nice to pretend that there's only 1 encoding in the world and that encoding is roughly equivalent to unicode itself, but this isn't true. World doesn't work on UTF-8. By abstracting encodings Python 3 obscures what's actually happening.

zo1 11y ago
Not at all, it forces you to unify to storing character data as Unicode code-points. Eventually if you go down far enough, you're reading the raw bytes you speak of from some sort of external system/file/DB. It's at that point that you define your encoding, and never worry about it every again. Except for when you touch those external systems/files again, again with full knowledge about what encoding is required. Either way, if you're not dealing with differently encoded character data in Python 2.x as noted above, you're causing yourself some unnecessary headaches.
vetinari 11y ago
In Python2, I couldn't just print the string I just read from file - such were the joys of Windows console, which has different encoding than the rest of the system (cp85x in console, cp125x in filesystem & gui)... Compared to that, Python3 was like gift from heaven.
takeda 11y ago
Something tells me you did not experience the pre-unicode world around 90s. I did, and I didn't like it. I prefer it is abstracted from me thankyouverymuch. But seriously, all what python (and as a matter of the fact majority of languages) is placing a clear distinction between a byte and a character. This forces developers to keep in mind what they are dealing with and reduces number of bugs. The truth is that if you treat everything as bytes, chances are that your program has bugs. Actually good test is to convert your existing Py2 code to Py3, if you have encoding/decoding errors then your code was buggy.