6 ms·

You missed a whole subclass of people. Those that don't like Python3. Personally I believe Python3 took a wrong turn with the approach of converting everything

You missed a whole subclass of people. Those that don't like Python3. Personally I believe Python3 took a wrong turn with the approach of converting everything to UTF-32 and making a big deal about the philosophical difference between "strings" and "bytes". I very much suspect the future will be something like how Julia deals with the issue, where UTF-8 can be just left as UTF-8.

maxerickson 11y ago
Recent versions only convert to UTF-32 when a wide character is present (they also use 2 byte characters when that is sufficient): https://docs.python.org/3.3/whatsnew/3.3.html#pep-393-flexible-string-representation https://docs.python.org/3.3/whatsnew/3.3.html#pep-393-flexib... Glancing at Julia, I'm not sure "invalid character index" is really a great thing to have to handle when dealing with text. It does seem that systems for dealing with text will eventually offer various apis beyond just a somewhat disfunctional character api (which I think is a fair description of str in Python 3).
- upofadown 11y ago
  >Recent versions only convert to UTF-32 when a wide character is present (they also use 2 byte characters when that is sufficient) ... I happen to know this fact. Are you really asking that everyone that comments on a language acknowledge all the possible implementation details? I at no point complained about the memory required to hold strings in CPython so I can't really understand why you would bring this up.
  - maxerickson 11y ago
    Because someone else reading "converting everything to UTF-32" might not be aware of the nuance. That's all.
    - teach 11y ago
      I'm a casual user of Python 2, and I certainly didn't know this, so I appreciated the clarification.
    - upofadown 11y ago
      Sorry for the snark but this has become a trigger for me. Every time anyone disagrees with the way Py3 does Unicode someone will immediately assume that that person is just confused about the implementation.
- Avernar 11y ago
  > Recent versions only convert to UTF-32 when a wide character is present It's non-BMP characters that trigger the conversion to 4 bytes per codepoint. Unfortunately Emoji are non-BMP and are super common.
  - rspeer 11y ago
    If you're dealing with emoji -- and yes, they are incredibly common -- you'd better not use any Python before 3.3. Non-BMP characters in 2.x-3.2 give you implementation-specific behavior. It can vary from one system to the next. Some string methods fail on "narrow builds" of Python (the ones that store their characters in UCS-2), while others return different results than they would on a "wide build".
    - Avernar 11y ago
      Not an issue because I don't use the Unicode type. All my data comes in as UTF-8 on which I do a quick validation check. Then it's just handled as bytes all the way to and from my database.
      - rspeer 11y ago
        Sure, that is an option that's always available. You don't have to worry if Python's string methods are wrong if you're opting not to use them.
mangecoeur 11y ago
Apparently you don't deal with much non-ascii data if you don't like python3's string handling. It's literally the reason I first upgraded, because reading in simple text data on Python2 can break completely. Seeing as literally no language on earth can be written entirely in ascii characters (since even English text can contain words with accents or old fashioned things like the long s ſ) this is a pretty big issue. Yes it is possibly to laboriously add code to check and encode/decode text, that might sometime be text and sometimes binary pretending to be text pretending to be binary. But why go through that pain when you could just use Python3 where things are clear and generally actually work.
- eeZi 11y ago
  I fully agree. Python 3 makes life much, much easier with non-ASCII languages and prevents the annoying Unicode conversion bugs you see in most Python 2 code bases. It's also very intuitive, which is great for teaching - I've always had a hard time explaining the difference between "strings" and "unicode" objects in Python 2, but the clear boundary makes it straight-forward in Python 3.
  - SeanDav 11y ago
    Not always straight forward in Python 3. Try explaining to your class how to ignore case in string comparisons.
    - zo1 11y ago
      That depends on the type of strings you are comparing. If you include the full Unicode set, you're going to run in to huge troubles in probably all languages simply due to the nature of the Unicode character set, rather than the specific language's handling of strings and Unicode code points. But, personally, if you're teaching Python to a class, whether it's 2.x or 3.x. You should probably stick to using some variation of string.upper() == string.upper() to explain simple comparisons, due to simplicity.
    - squeaky-clean 11y ago
      >Try explaining to your class how to ignore case in string comparisons Okay: You can't. You'll need to convert both strings to the same case, and then compare those. In the majority of cases, you can do: return string_a.lower() == string_b.lower() But sometimes that won't work, because converting case isn't always easy. And that's a unicode issue, not a python issue. It's not even completely a unicode issue, just a "writing can be very weird" issue. For example, "ß" (U+00DF) is a character that doesn't/shouldn't have an uppercase form. Unicode does support it as U+1E9E, but all the examples I know of will convert it to "SS" ([U+0053, U+0053]) instead. Try this in a javascript console, for example. > "ß".toUpperCase().toLowerCase() === "ß".toLowerCase() false > "ß".toUpperCase() === "SS".toUpperCase() true > "ß".toLowerCase() === "SS".toLowerCase() false > "ß".length 1 > "ß".toLowerCase().length 1 > "ß".toUpperCase().length 2 > "ß".toUpperCase().toLowerCase() "ss" Also, try ctrl+f'ing for "ss" on the Wiki page for ß [0] and see what it matches. Or ctrl+f this page for "ß" and it will match every "ss". I think it's pretty neat. [0] https://en.wikipedia.org/wiki/%C3%9F https://en.wikipedia.org/wiki/%C3%9F (edit: You might already know everything in this comment, but I started writing and it ended up being a cool exercise for myself, so I figured I'd submit).
Avernar 11y ago
I agree. Everything is so much simpler when you use UTF-8 everywhere. Being effectively forced to use the ASCII/UCS2/UCS4 unicode type in Python3 is the only thing that has kept me from switching to it.
- 1st1 11y ago
  The only thing you are forced to think about is whether you're working with strings or with bytes. Unicode is handled transparently (but you do need to know the encoding when you're decoding bytes). How it's stored internally is none of your concern.
  - Avernar 11y ago
    Yes it is. Every conversion to and from the unicode type and every expansion of the unicode type from 1 to 2 to 4 bytes eats up processor time, cache/memory bandwidth and puts more pressure on the garbage collector. And I don't want to be forced to distinguish between strings and bytes. To me they are one and the same as I use UTF-8 for everything. Not once in my code do I need to encode/decode anything.
    - mark-r 11y ago
      Most of us aren't so lucky. I really do wish we lived in a 100% UTF-8 world. But even if we did, I don't see how e.g. regular expressions would work.
      - Avernar 11y ago
        The world seems to be headed to UTF-8 for everything. That's why having Python3 force us away from it doesn't make sense to me. As for regular expressions, they work perfectly if your delimiters are in the ASCII range. If you're trying to match a single character then they don't work for Unicode even when using Python3's unicode type. The reason is that code points are not characters.
        11y ago

maxerickson 11y ago

Recent versions only convert to UTF-32 when a wide character is present (they also use 2 byte characters when that is sufficient): https://docs.python.org/3.3/whatsnew/3.3.html#pep-393-flexible-string-representation https://docs.python.org/3.3/whatsnew/3.3.html#pep-393-flexib... Glancing at Julia, I'm not sure "invalid character index" is really a great thing to have to handle when dealing with text. It does seem that systems for dealing with text will eventually offer various apis beyond just a somewhat disfunctional character api (which I think is a fair description of str in Python 3).

upofadown 11y ago
>Recent versions only convert to UTF-32 when a wide character is present (they also use 2 byte characters when that is sufficient) ... I happen to know this fact. Are you really asking that everyone that comments on a language acknowledge all the possible implementation details? I at no point complained about the memory required to hold strings in CPython so I can't really understand why you would bring this up.
- maxerickson 11y ago
  Because someone else reading "converting everything to UTF-32" might not be aware of the nuance. That's all.
  - teach 11y ago
    I'm a casual user of Python 2, and I certainly didn't know this, so I appreciated the clarification.
  - upofadown 11y ago
    Sorry for the snark but this has become a trigger for me. Every time anyone disagrees with the way Py3 does Unicode someone will immediately assume that that person is just confused about the implementation.
Avernar 11y ago
> Recent versions only convert to UTF-32 when a wide character is present It's non-BMP characters that trigger the conversion to 4 bytes per codepoint. Unfortunately Emoji are non-BMP and are super common.
- rspeer 11y ago
  If you're dealing with emoji -- and yes, they are incredibly common -- you'd better not use any Python before 3.3. Non-BMP characters in 2.x-3.2 give you implementation-specific behavior. It can vary from one system to the next. Some string methods fail on "narrow builds" of Python (the ones that store their characters in UCS-2), while others return different results than they would on a "wide build".
  - Avernar 11y ago
    Not an issue because I don't use the Unicode type. All my data comes in as UTF-8 on which I do a quick validation check. Then it's just handled as bytes all the way to and from my database.
    - rspeer 11y ago
      Sure, that is an option that's always available. You don't have to worry if Python's string methods are wrong if you're opting not to use them.

mangecoeur 11y ago

Apparently you don't deal with much non-ascii data if you don't like python3's string handling. It's literally the reason I first upgraded, because reading in simple text data on Python2 can break completely. Seeing as literally no language on earth can be written entirely in ascii characters (since even English text can contain words with accents or old fashioned things like the long s ſ) this is a pretty big issue. Yes it is possibly to laboriously add code to check and encode/decode text, that might sometime be text and sometimes binary pretending to be text pretending to be binary. But why go through that pain when you could just use Python3 where things are clear and generally actually work.

eeZi 11y ago
I fully agree. Python 3 makes life much, much easier with non-ASCII languages and prevents the annoying Unicode conversion bugs you see in most Python 2 code bases. It's also very intuitive, which is great for teaching - I've always had a hard time explaining the difference between "strings" and "unicode" objects in Python 2, but the clear boundary makes it straight-forward in Python 3.
- SeanDav 11y ago
  Not always straight forward in Python 3. Try explaining to your class how to ignore case in string comparisons.
  - zo1 11y ago
    That depends on the type of strings you are comparing. If you include the full Unicode set, you're going to run in to huge troubles in probably all languages simply due to the nature of the Unicode character set, rather than the specific language's handling of strings and Unicode code points. But, personally, if you're teaching Python to a class, whether it's 2.x or 3.x. You should probably stick to using some variation of string.upper() == string.upper() to explain simple comparisons, due to simplicity.
  - squeaky-clean 11y ago
    >Try explaining to your class how to ignore case in string comparisons Okay: You can't. You'll need to convert both strings to the same case, and then compare those. In the majority of cases, you can do: return string_a.lower() == string_b.lower() But sometimes that won't work, because converting case isn't always easy. And that's a unicode issue, not a python issue. It's not even completely a unicode issue, just a "writing can be very weird" issue. For example, "ß" (U+00DF) is a character that doesn't/shouldn't have an uppercase form. Unicode does support it as U+1E9E, but all the examples I know of will convert it to "SS" ([U+0053, U+0053]) instead. Try this in a javascript console, for example. > "ß".toUpperCase().toLowerCase() === "ß".toLowerCase() false > "ß".toUpperCase() === "SS".toUpperCase() true > "ß".toLowerCase() === "SS".toLowerCase() false > "ß".length 1 > "ß".toLowerCase().length 1 > "ß".toUpperCase().length 2 > "ß".toUpperCase().toLowerCase() "ss" Also, try ctrl+f'ing for "ss" on the Wiki page for ß [0] and see what it matches. Or ctrl+f this page for "ß" and it will match every "ss". I think it's pretty neat. [0] https://en.wikipedia.org/wiki/%C3%9F https://en.wikipedia.org/wiki/%C3%9F (edit: You might already know everything in this comment, but I started writing and it ended up being a cool exercise for myself, so I figured I'd submit).

Avernar 11y ago

I agree. Everything is so much simpler when you use UTF-8 everywhere. Being effectively forced to use the ASCII/UCS2/UCS4 unicode type in Python3 is the only thing that has kept me from switching to it.

1st1 11y ago
The only thing you are forced to think about is whether you're working with strings or with bytes. Unicode is handled transparently (but you do need to know the encoding when you're decoding bytes). How it's stored internally is none of your concern.
- Avernar 11y ago
  Yes it is. Every conversion to and from the unicode type and every expansion of the unicode type from 1 to 2 to 4 bytes eats up processor time, cache/memory bandwidth and puts more pressure on the garbage collector. And I don't want to be forced to distinguish between strings and bytes. To me they are one and the same as I use UTF-8 for everything. Not once in my code do I need to encode/decode anything.
  - mark-r 11y ago
    Most of us aren't so lucky. I really do wish we lived in a 100% UTF-8 world. But even if we did, I don't see how e.g. regular expressions would work.
    - Avernar 11y ago
      The world seems to be headed to UTF-8 for everything. That's why having Python3 force us away from it doesn't make sense to me. As for regular expressions, they work perfectly if your delimiters are in the ASCII range. If you're trying to match a single character then they don't work for Unicode even when using Python3's unicode type. The reason is that code points are not characters.
      - 11y ago