7 ms·

Python 3 did not break compatibilty for "pureness" sake. They broke it, because for 6 billion people in the world, for whom English is not the only language, th

Python 3 did not break compatibilty for "pureness" sake. They broke it, because for 6 billion people in the world, for whom English is not the only language, there was a very real pain (bordering on stuff that was impossible). Most of the world is moving to unicode exactly because of this (I know Sillicon Vally companies are probably moving to unicode, so that their users can use cute emojis). Most people also need to handle stuff that are in different encodings (I work daily in both Latin1 (north-western European) and UTF8), translating between that was painful/borderline-impossible in the old days (python 2). The sharp distinction between bytes and strings are essential for any non-english/american/aussie person. Sadly python2 got that wrong, and hence had to die.. Which is probably also why Perl6 choose to go the same route.. If you really need a good reason to go to python 3 other than 6 billion extra potential customers, try "oh look, we have emojis" <sarcasm/>

detaro 9y ago
One might argue that making unicode strings the default instead of a special datatype (which they were in 2.x) is also a "pureness" decision (since you can handle unicode as much in 2.7 as you can in 3.x)
prevedmedved 9y ago
I'm one of those people for whom English is not the only language. Python 3 is objectively harder and more confusing to use in a multilingual environment. Python 2 didn't pretend that only Unicode exists and Python 2 was encoding-agnostic. This is the correct default for the real world. Ironically, the effect of Python 3 was the opposite: it made life really hard for those that aren't exclusive anglophones and made life a lot easier for American programmers who now don't have to bother with the whole character encoding mess. Personally, Python 3's broken support for internationalisation made me drop Python altogether. Python 2 was confusing, but the defaults were at least workable.
- svennek 9y ago
  Really? I find it so much easier (I am also not an English language programmer), because it is very explicit about what is the codepoints (i.e. string) and what is the bytes (i.e. on the disk or the network). The encode/decode functions is basically the only change, but it gives you the reason to explicitly tell it about the encoding you expect.. But as you left python, the point is moot :)
  - saurik 9y ago
    Python 2 was also explicit: it had unicode and it had str. I never had a problem managing complex character encoding issues in Python 2.
  - prevedmedved 9y ago
    You're right that you can get Python 3 to behave correctly if you jump through some hoops. However, wasn't the point of Python 3 to remove the hoopjumping in the first place? Anyways, the point is that Python 3's design was broken from the start. Anglophone programmers think that internationalisation means "upgrading" from ASCII to Unicode. In reality, though, the rest of the world's programmers got along just fine long before Unicode was invented and will continue to do fine long after Unicode gets replaced by something else. True internationalisation means mechanisms to deal with the world's text encodings in a neutral and culture-agnostic way. (And "Unicode only everywhere and no exceptions" is definitely not that.)
    - catamorphismic 9y ago
      The bytes and bytestring types in Python 3 are basically the str type of Python 2 and I don't see that as much of a hoop to jump through. The Unicode default seems like a very sane decision from my point of view. Your argument primarily seems based around an anti-Unicode viewpoint. Am I missing something?
    - svennek 9y ago
      If you see the converting as hoop-jumping then I agree with you. If you do recognize that there is such a thing called text, that human understand, and for example the text "hello" has five "letters".... The point that that text might be represented in five different (and equally valid) ways with different amounts of bytes is irrelevant to the "human user"... Hence the sharp distinction between makes sense to most humans. The danish word "rødgrød" for example is seven letters (i.e. every person would say it is 7 letters long), a computer would call it 7, 7 9 or 16 bytes (in cp865 (nordics) latin1("wester european", utf8 and utf16...) Unless you work in the ideal "text" mode getting the correct length of the text is not that trivial. Equally, the pair 2-3 and and 7-8 bytes in utf8 must not be split (as either half has not meaning on its own).. hence (in utf8) a function "give me the next letter" will return 1,2,1,1,1,2,1 bytes in its revocations.. Also casefolding (upper/lower case) is hard when working in bytes (some glyphs might not even have one or the other).. I am unable to say if unicode fits all languages in the world as needed, but it is much better as byte-wrangling if you have multiple possible encodings at once.. EDIT: for reference the bytes of "rødgrød" are cp865 : b'r\x9bdgr\x9bd' latin1 : b'r\xf8dgr\xf8d' utf8 : b'r\xc3\xb8dgr\xc3\xb8d' utf16le: b'r\x00\xf8\x00d\x00g\x00r\x00\xf8\x00d\x00' utf16be: b'\x00r\x00\xf8\x00d\x00g\x00r\x00\xf8\x00d' (and it cannot be written in ASCII ("C")) Anyone saying that bytes and text are the same are nuts...
- sametmax 9y ago
  Objectively os a badly choosen word since as a french i strongly disagreewith you. All my dev and all my new commers training are much much easier in python 3
patrec 9y ago
I've written commercial code to handle decidedly non-english languages in python2 just fine. Have you? Do you understand unicode and it's encodings well enough to have an informed opinion that python3's braindamaged unicode implementation is superior to working mostly at the byte string level in python2? Hint: this is much closer to what string handling in sensibly designed languages like Rust and Go or Erlang looks like.
- svennek 9y ago
  Yes, I have (for more than a decade in four different languages, not all of those who are latin1)... One project was even both LTR and RTL :) Byte handling is definately not fun in variable encodings, and even fixed-length multi-byte encodings require a lot of support functionality when working with bytes.. Such a simple thing as "len" is suddenly hard (for the human definition of length and not the byte-length), the latter is also easily attainable by converting the string to bytes and get the length of that in the given encoding.
  - patrec 9y ago
    len isn't a great example because it doesn't do anything remotely useful on python3 strs in the general case. In particular it doesn't tell you how many characters the str has, which is what you probably wanted to know -- possibly as an approximation for figuring out how many columns in fixed-width it would take, in which case of course you are even more likely to be disappointed. In python2 len(str) will tell you exactly how many bytes some text takes, which is useful.
- masklinn 9y ago
  > Hint: this is much closer to what string handling in sensibly designed languages like Rust and Go or Erlang looks like. * Rust has a very strict split between bytes and text * Go has fucked up bytestrings which depending on validity may break at any point depending on the API you're applying * Erlang's strings are widely considered awful, and are literally just linked lists of integers making them slow, memory-heavy and prone to pretty-printer whims (as the pretty printer decides on the fly whether a given list looks like a string — and should be printed thus — or does not — and should be printed as a regular list of integers)
netheril96 9y ago
I am a Chinese and according to my observation Python 2 are still dominant in China. I don't know any companies on Python 3. The idea that a cleaner Unicode handling trumps backwards compatibility for the non-English speaking is disproved by reality. Or at least disproved in China.
- deleted 9y ago
  [deleted]
- svennek 9y ago
  Porting from py2 to py3 is a "lot" of work and has a lot of risk. See (was it?) Instagrams keynote at PyCon of how one missing "b" prefix cost them 10% or so percent of runtime performance (can't remember the exact number). For every project that ports, the risk goes down for the rest (of those who still need to move), as more bugs are uncovered and more lessons/best-practices have been learned. Hence porting late is an advantage from a cost/risk point.. Also, if there are no strong technical people advocating for the change, most organisation only see porting as a (unnecessary) costly exercise.. I think the judge is still out on whether that is really disproved, but right now you are likely correct :)
xorcist 9y ago
> for 6 billion people in the world, for whom English is not the only language, there was a very real pain Python 3 is different but not necessarily better in this regard. While I hated the encode/decode on .. every .. single .. function call, py3 pretends a lot of bytestreams are instead unicode (file and path names, for example). This can lead to bugs. It's not that it wasn't worth it, but it could have been done a lot better. > Which is probably also why Perl6 choose to go the same route Not at all. Perl 5 did unicode right. The difference between character, glyph, code point and byte is clear and uniform throughout the entire stack. Python could have copied this and it would have been awesome.
- dom0 9y ago
  The issue w.r.t. paths is that paths have different types depending on the operating system. On Linux and most BSDs paths are bytes. On some Unices they are bytes, but must be in a specific encoding (e.g. macOS => UTF-8). On others, they are strings, not bytes (e.g. Windows). It's implausible to expect Python programmers to correctly handle this, hence fsencode/fsdecode and surrogate escaping.
  - xorcist 9y ago
    It's part of the problem of cross platform software. Sometimes you have to go with the least common denominator, in this case byte strings. Do not be smart about it and let the programmer handle the errors. It is unexpected to blow up my software because a user supplied completely valid file name happens to be invalid by Python rules. Yes, fsencode was finally added to work around parts of this problem which is good, but it shouldn't have been there in the first place. The problem is universal to every time you interface with the operating system. Sometimes it's locale dependent too and you never know beforehand so a program might blow up in the hands of a customer. While the encode/decode process was tiresome at least it gave predictable results. If you have to work around your runtime in order to give the parameters you want to the underlying syscalls, you are doing it wrong. The premise to move to unicode was a laudable one, but I wish more attention had been paid to how other languages do it, because anything worse than what Perl 5 did more than a decade earlier isn't that great.
est 9y ago
> because for 6 billion people in the world, for whom English is not the only language, there was a very real pain (bordering on stuff that was impossible). Python2 unicode is not broken, the basic type str() is broken because it defaults to ascii.
nothrabannosir 9y ago
I don't understand this. Support was fine: first class Unicode objects and first class tools to encode and decode them to/from byte strings. Not mega intuitively named, but definitely not worth breaking backwards compatibility for an entire language over. Notably the semantics didn't even change; they still have Unicode objects and byte strings in 3, just with different names now. And you still need to encode/decode them. You always will, that's the thing with encodings. What specific issues do you mean?
- svennek 9y ago
  If you don't think the sharp divide between bytes and str in py3 is something new, you haven't grasped the value of it.. I am not sure, you are correct that py2-unicode == py3-str and py3-bytes == py2-str ... But even if (and I must admit that it is more then 5 years since I wrote py2 of any length).. The value is... in py3 "str" is the ideal string, which does things the way humans expect.. When you need to pump in or out bytes, the class is called "bytes" and requires an explicit conversion, where you get prompted to think about encodings.. (but you don't need to think about it all the time as you would with byte-wrangling) This way, people not well-versed in the horrible mess of encodings are going to be safe by default... If they do something unsupported they get an exception, not something that is wrong... That alone is worth it.. Also, the full stack of python is working correctly with unicode, you don't have to handle two different "string" classes ("str" and "unicode") where one of them is like to do the wrong thing.. One often overlook thing is that py3 defaults to utf8 source code, in the old days integrating stuff from files with non-ascii chars in different encodings was a nightmare...
  - netheril96 9y ago
    Sigh, that could have easily been implemented with a "use strict" directive. Or, `from __future__ import strict_unicode`. JavaScript has this strict mode, albeit to solve a different problem, and its success despite its messiness alludes to the notion that gradual improvement is better than sudden breakage.
    - acdha 9y ago

detaro 9y ago

One might argue that making unicode strings the default instead of a special datatype (which they were in 2.x) is also a "pureness" decision (since you can handle unicode as much in 2.7 as you can in 3.x)

prevedmedved 9y ago

I'm one of those people for whom English is not the only language. Python 3 is objectively harder and more confusing to use in a multilingual environment. Python 2 didn't pretend that only Unicode exists and Python 2 was encoding-agnostic. This is the correct default for the real world. Ironically, the effect of Python 3 was the opposite: it made life really hard for those that aren't exclusive anglophones and made life a lot easier for American programmers who now don't have to bother with the whole character encoding mess. Personally, Python 3's broken support for internationalisation made me drop Python altogether. Python 2 was confusing, but the defaults were at least workable.

svennek 9y ago
Really? I find it so much easier (I am also not an English language programmer), because it is very explicit about what is the codepoints (i.e. string) and what is the bytes (i.e. on the disk or the network). The encode/decode functions is basically the only change, but it gives you the reason to explicitly tell it about the encoding you expect.. But as you left python, the point is moot :)
- saurik 9y ago
  Python 2 was also explicit: it had unicode and it had str. I never had a problem managing complex character encoding issues in Python 2.
- prevedmedved 9y ago
  You're right that you can get Python 3 to behave correctly if you jump through some hoops. However, wasn't the point of Python 3 to remove the hoopjumping in the first place? Anyways, the point is that Python 3's design was broken from the start. Anglophone programmers think that internationalisation means "upgrading" from ASCII to Unicode. In reality, though, the rest of the world's programmers got along just fine long before Unicode was invented and will continue to do fine long after Unicode gets replaced by something else. True internationalisation means mechanisms to deal with the world's text encodings in a neutral and culture-agnostic way. (And "Unicode only everywhere and no exceptions" is definitely not that.)
  - catamorphismic 9y ago
    The bytes and bytestring types in Python 3 are basically the str type of Python 2 and I don't see that as much of a hoop to jump through. The Unicode default seems like a very sane decision from my point of view. Your argument primarily seems based around an anti-Unicode viewpoint. Am I missing something?
  - svennek 9y ago
    If you see the converting as hoop-jumping then I agree with you. If you do recognize that there is such a thing called text, that human understand, and for example the text "hello" has five "letters".... The point that that text might be represented in five different (and equally valid) ways with different amounts of bytes is irrelevant to the "human user"... Hence the sharp distinction between makes sense to most humans. The danish word "rødgrød" for example is seven letters (i.e. every person would say it is 7 letters long), a computer would call it 7, 7 9 or 16 bytes (in cp865 (nordics) latin1("wester european", utf8 and utf16...) Unless you work in the ideal "text" mode getting the correct length of the text is not that trivial. Equally, the pair 2-3 and and 7-8 bytes in utf8 must not be split (as either half has not meaning on its own).. hence (in utf8) a function "give me the next letter" will return 1,2,1,1,1,2,1 bytes in its revocations.. Also casefolding (upper/lower case) is hard when working in bytes (some glyphs might not even have one or the other).. I am unable to say if unicode fits all languages in the world as needed, but it is much better as byte-wrangling if you have multiple possible encodings at once.. EDIT: for reference the bytes of "rødgrød" are cp865 : b'r\x9bdgr\x9bd' latin1 : b'r\xf8dgr\xf8d' utf8 : b'r\xc3\xb8dgr\xc3\xb8d' utf16le: b'r\x00\xf8\x00d\x00g\x00r\x00\xf8\x00d\x00' utf16be: b'\x00r\x00\xf8\x00d\x00g\x00r\x00\xf8\x00d' (and it cannot be written in ASCII ("C")) Anyone saying that bytes and text are the same are nuts...
sametmax 9y ago
Objectively os a badly choosen word since as a french i strongly disagreewith you. All my dev and all my new commers training are much much easier in python 3

patrec 9y ago

I've written commercial code to handle decidedly non-english languages in python2 just fine. Have you? Do you understand unicode and it's encodings well enough to have an informed opinion that python3's braindamaged unicode implementation is superior to working mostly at the byte string level in python2? Hint: this is much closer to what string handling in sensibly designed languages like Rust and Go or Erlang looks like.

svennek 9y ago
Yes, I have (for more than a decade in four different languages, not all of those who are latin1)... One project was even both LTR and RTL :) Byte handling is definately not fun in variable encodings, and even fixed-length multi-byte encodings require a lot of support functionality when working with bytes.. Such a simple thing as "len" is suddenly hard (for the human definition of length and not the byte-length), the latter is also easily attainable by converting the string to bytes and get the length of that in the given encoding.
- patrec 9y ago
  len isn't a great example because it doesn't do anything remotely useful on python3 strs in the general case. In particular it doesn't tell you how many characters the str has, which is what you probably wanted to know -- possibly as an approximation for figuring out how many columns in fixed-width it would take, in which case of course you are even more likely to be disappointed. In python2 len(str) will tell you exactly how many bytes some text takes, which is useful.
masklinn 9y ago
> Hint: this is much closer to what string handling in sensibly designed languages like Rust and Go or Erlang looks like. * Rust has a very strict split between bytes and text * Go has fucked up bytestrings which depending on validity may break at any point depending on the API you're applying * Erlang's strings are widely considered awful, and are literally just linked lists of integers making them slow, memory-heavy and prone to pretty-printer whims (as the pretty printer decides on the fly whether a given list looks like a string — and should be printed thus — or does not — and should be printed as a regular list of integers)

netheril96 9y ago

I am a Chinese and according to my observation Python 2 are still dominant in China. I don't know any companies on Python 3. The idea that a cleaner Unicode handling trumps backwards compatibility for the non-English speaking is disproved by reality. Or at least disproved in China.

deleted 9y ago
[deleted]
svennek 9y ago
Porting from py2 to py3 is a "lot" of work and has a lot of risk. See (was it?) Instagrams keynote at PyCon of how one missing "b" prefix cost them 10% or so percent of runtime performance (can't remember the exact number). For every project that ports, the risk goes down for the rest (of those who still need to move), as more bugs are uncovered and more lessons/best-practices have been learned. Hence porting late is an advantage from a cost/risk point.. Also, if there are no strong technical people advocating for the change, most organisation only see porting as a (unnecessary) costly exercise.. I think the judge is still out on whether that is really disproved, but right now you are likely correct :)

xorcist 9y ago

> for 6 billion people in the world, for whom English is not the only language, there was a very real pain Python 3 is different but not necessarily better in this regard. While I hated the encode/decode on .. every .. single .. function call, py3 pretends a lot of bytestreams are instead unicode (file and path names, for example). This can lead to bugs. It's not that it wasn't worth it, but it could have been done a lot better. > Which is probably also why Perl6 choose to go the same route Not at all. Perl 5 did unicode right. The difference between character, glyph, code point and byte is clear and uniform throughout the entire stack. Python could have copied this and it would have been awesome.

dom0 9y ago
The issue w.r.t. paths is that paths have different types depending on the operating system. On Linux and most BSDs paths are bytes. On some Unices they are bytes, but must be in a specific encoding (e.g. macOS => UTF-8). On others, they are strings, not bytes (e.g. Windows). It's implausible to expect Python programmers to correctly handle this, hence fsencode/fsdecode and surrogate escaping.
- xorcist 9y ago
  It's part of the problem of cross platform software. Sometimes you have to go with the least common denominator, in this case byte strings. Do not be smart about it and let the programmer handle the errors. It is unexpected to blow up my software because a user supplied completely valid file name happens to be invalid by Python rules. Yes, fsencode was finally added to work around parts of this problem which is good, but it shouldn't have been there in the first place. The problem is universal to every time you interface with the operating system. Sometimes it's locale dependent too and you never know beforehand so a program might blow up in the hands of a customer. While the encode/decode process was tiresome at least it gave predictable results. If you have to work around your runtime in order to give the parameters you want to the underlying syscalls, you are doing it wrong. The premise to move to unicode was a laudable one, but I wish more attention had been paid to how other languages do it, because anything worse than what Perl 5 did more than a decade earlier isn't that great.

est 9y ago

> because for 6 billion people in the world, for whom English is not the only language, there was a very real pain (bordering on stuff that was impossible). Python2 unicode is not broken, the basic type str() is broken because it defaults to ascii.

nothrabannosir 9y ago

I don't understand this. Support was fine: first class Unicode objects and first class tools to encode and decode them to/from byte strings. Not mega intuitively named, but definitely not worth breaking backwards compatibility for an entire language over. Notably the semantics didn't even change; they still have Unicode objects and byte strings in 3, just with different names now. And you still need to encode/decode them. You always will, that's the thing with encodings. What specific issues do you mean?

svennek 9y ago
If you don't think the sharp divide between bytes and str in py3 is something new, you haven't grasped the value of it.. I am not sure, you are correct that py2-unicode == py3-str and py3-bytes == py2-str ... But even if (and I must admit that it is more then 5 years since I wrote py2 of any length).. The value is... in py3 "str" is the ideal string, which does things the way humans expect.. When you need to pump in or out bytes, the class is called "bytes" and requires an explicit conversion, where you get prompted to think about encodings.. (but you don't need to think about it all the time as you would with byte-wrangling) This way, people not well-versed in the horrible mess of encodings are going to be safe by default... If they do something unsupported they get an exception, not something that is wrong... That alone is worth it.. Also, the full stack of python is working correctly with unicode, you don't have to handle two different "string" classes ("str" and "unicode") where one of them is like to do the wrong thing.. One often overlook thing is that py3 defaults to utf8 source code, in the old days integrating stuff from files with non-ascii chars in different encodings was a nightmare...
- netheril96 9y ago
  Sigh, that could have easily been implemented with a "use strict" directive. Or, `from __future__ import strict_unicode`. JavaScript has this strict mode, albeit to solve a different problem, and its success despite its messiness alludes to the notion that gradual improvement is better than sudden breakage.
  - acdha 9y ago