6 ms·

> and the result was better Unicode support Different Unicode support. And worse bytes support. What could previously be done using python -c "..." is now lon

> and the result was better Unicode support Different Unicode support. And worse bytes support. What could previously be done using python -c "..." is now long, horrible and ugly.

mehrdadn 6y ago
> Different Unicode support. And worse bytes support. I feel like you're the first person I've seen on the planet to echo my sentiments on these. I expect a lot of people will jump here to tell you you're wrong like they have to me, so just wanted to let you know I've felt exactly these pains and agree with you.
- harikb 6y ago
  I am hoping this is in agreement. py2’s flexibility to handle utf8 bytes without fuss is amazing. Then people come up with all kind of purity reasons to make it more complicated.
  - mehrdadn 6y ago
    Take out "utf8" and I'll agree ;) The fundamental problem as I see it is that "string" is a grossly leaky and misunderstood abstraction. The string type is not the same thing as a "text" type. It's being used in all the wrong places for that purpose. People treat "string" like it means "text", but in so many places where we deal with them, they just aren't (and should never be) text. Everything from stdio to argv to file paths to environment variables to "text" files to basically any interface with the outside world needs to be dealt with in bytes rather than text if you care about actually producing correct code that doesn't lose, corrupt, or otherwise choke on data. C++ understood this and got it right, preferring to focus on optimizing rather than constraining the string type. Many other languages did pretty well by avoiding enforcing encodings on strings, too. And Python 2 defaulted to bytes as well, and only really cared about encoding/decoding at I/O boundaries where it thought it can assume it's dealing with text (though it sometimes didn't behave well there, and yes it got painful as a result). Then Python 3 came along and just made everyone start treating most data as if they're inherently (Unicode) text by default, when they really had no such constraints to begin with. It boggles my mind that Python 3 folks like to beat the drum on how Python 3 got the bytes/unicode right without taking a single moment to even notice that most strings people deal with aren't (and never were!) actually guaranteed to be in a specific, known textual encoding a priori. They were just arrays of code units with few restrictions on them, and if you want to write correct code, you're going to have to deal with bytes by default (or something else with similar flexibility) instead of text. It would've been totally fine to introduce a text type, but it fundamentally can't take the place of a blob type, which is the language of the outside world.
    - int_19h 6y ago
      "The outside world", by and large, also speaks Unicode. Java uses UTF-16 throughout, including file paths. So does .NET. All Apple platforms are UTF-16. C++ - if you just look at stdlib, sure, it's byte-centric; but then look at popular frameworks such as Qt. In practice, this means that, yeah, you can have that odd filename that is technically not Unicode. But the vast majority of code running on the most popular desktop and mobile platforms is going to handle it in a way that expects it to be Unicode. Why should Python go against the trend, and make life more complicated for developers using it in the process?
      - scoot_718 6y ago
        Many of us deal in bytes that simply aren't UTF8 and never could be. Because they're just bytes. How many things are stored as binary files? > All Apple platforms are UTF-16. I'm glad all their executable files are apparently text files. How amazing. > Why should Python go against the trend, and make life more complicated for developers using it in the process? You tell me why Python3 did that.
      - mehrdadn 6y ago
        File names? I listed so much more for you than file names. That HTML you just fetched? How do you know it's Unicode? That .txt file the user just asked to load? How do you know that's Unicode? For heaven's sake, when can you actually guarantee that even sys.stdin.read() is going to read Unicode? You can only do that when you're the one piping your own stdin... which is not the common case. What do you do when your fundamentally invalid assumptions break? Do you just not care and simply present a stack trace to the user and tell them to get lost? I've gotten tired of these debates though, so just a heads up I may not have the energy to reply if you continue...
        gvjddbnvdrbv 6y ago
        In the real world Python2 gave stack traces by default when presented with common strings. Python3 doesn't.
        6y ago
  - int_19h 6y ago
    Except for that part where it would happily implicitly convert them to/from a Unicode string in any context where one was needed or present... using ASCII, rather than UTF-8, as the encoding.
  - raverbashing 6y ago
    > py2’s flexibility to handle utf8 bytes without fuss is amazing Without fuzz? No, sorry, it was anything but. First of all it would default encoding to "ASCII". Have any whiff of non-explicitly handled UTF-8 and it would just go bang at the worse time possible. That was a stupid decision "Oh but there was setdefaultencoding" Yeah here's the first result for that https://stackoverflow.com/questions/3828723/why-should-we-not-use-sys-setdefaultencodingutf-8-in-a-py-script https://stackoverflow.com/questions/3828723/why-should-we-no... So no, Python2 way of dealing with Unicode was the most annoying way possible, because hey who needs anything but ASCII right?
    - scoot_718 6y ago
      > Have any whiff of non-explicitly handled UTF-8 and it would just go bang at the worse time possible. Whereas python3 is just waiting to explode the moment there are bytes in your UTF-8 that are invalid. Oh the http request got truncated to leave invalid utf-8 in an otherwise fine utf-8 response? FUCKING ERROR.
    - ehsankia 6y ago
      I agree that it was easy to shoot yourself in the foot, and if you did have to deal with unicode, it was often a pain, but at the same time, Python's simplicity and ease of use is what makes it great, with the ability to do something cleaner if you choose to. You can choose to type annotate all your code and make it better. You can choose to organize your code in however package structure you want or keep it all in one file. The language doesn't force any of that onto you. That's the approach I think would've been much nicer and Pythonic in my mind. Now you're forced to use a much clunkier bytes/str paradigm, which yes will make your life much nicer in the 10% of the time when you'll need it, but the other 90% of the time will just be slightly more annoying. Similarly, I may be alone in this, but having to put parens around print statements is also annoying 99% of the time, but nice that 1% of the time I need to pass it as a function or pass it some extra arguments.
    - pdonis 6y ago
      > Python2 way of dealing with Unicode was the most annoying way possible The part about defaulting to ASCII is annoying, yes. And using sys.setdefaultencoding to change the default would still be annoying, yes. The reason for that is that any default encoding will be annoying whenever the actual encoding when the program is running doesn't match the default. The correct way to fix this problem is to not have a default encoding at all. Don't try to auto-detect encodings; don't try to guess encodings. Force every encode and decode operation to explicitly specify an encoding. That way the issue of what the encoding is, how to detect it, etc., is handled in the right place--in the code of the particular application that needs to use Unicode. It should not be handled in a language runtime or a standard library, precisely because there is no way for a language runtime or a library to properly deal with all use cases of all applications. What Python 3 did, instead, was to change the rules of default encodings and auto-detection/guessing of encodings, so that they were nicer to some use cases, and even more annoying than before to others.
- pdonis 6y ago
  > I feel like you're the first person I've seen on the planet to echo my sentiments on these. There have been plenty of people with similar sentiments. I'm one of them. I have felt ever since I first looked at Python 3 that the ways in which it broke backward incompatibility were heavily skewed towards a few particular use cases and did not take into account the needs of all of the Python community.
Too 6y ago
Got any examples?

> and the result was better Unicode support Different Unicode support. And worse bytes support. What could previously be done using python -c "..." is now long, horrible and ugly.

mehrdadn 6y ago

> Different Unicode support. And worse bytes support. I feel like you're the first person I've seen on the planet to echo my sentiments on these. I expect a lot of people will jump here to tell you you're wrong like they have to me, so just wanted to let you know I've felt exactly these pains and agree with you.

harikb 6y ago
I am hoping this is in agreement. py2’s flexibility to handle utf8 bytes without fuss is amazing. Then people come up with all kind of purity reasons to make it more complicated.
- mehrdadn 6y ago
  Take out "utf8" and I'll agree ;) The fundamental problem as I see it is that "string" is a grossly leaky and misunderstood abstraction. The string type is not the same thing as a "text" type. It's being used in all the wrong places for that purpose. People treat "string" like it means "text", but in so many places where we deal with them, they just aren't (and should never be) text. Everything from stdio to argv to file paths to environment variables to "text" files to basically any interface with the outside world needs to be dealt with in bytes rather than text if you care about actually producing correct code that doesn't lose, corrupt, or otherwise choke on data. C++ understood this and got it right, preferring to focus on optimizing rather than constraining the string type. Many other languages did pretty well by avoiding enforcing encodings on strings, too. And Python 2 defaulted to bytes as well, and only really cared about encoding/decoding at I/O boundaries where it thought it can assume it's dealing with text (though it sometimes didn't behave well there, and yes it got painful as a result). Then Python 3 came along and just made everyone start treating most data as if they're inherently (Unicode) text by default, when they really had no such constraints to begin with. It boggles my mind that Python 3 folks like to beat the drum on how Python 3 got the bytes/unicode right without taking a single moment to even notice that most strings people deal with aren't (and never were!) actually guaranteed to be in a specific, known textual encoding a priori. They were just arrays of code units with few restrictions on them, and if you want to write correct code, you're going to have to deal with bytes by default (or something else with similar flexibility) instead of text. It would've been totally fine to introduce a text type, but it fundamentally can't take the place of a blob type, which is the language of the outside world.
  - int_19h 6y ago
    "The outside world", by and large, also speaks Unicode. Java uses UTF-16 throughout, including file paths. So does .NET. All Apple platforms are UTF-16. C++ - if you just look at stdlib, sure, it's byte-centric; but then look at popular frameworks such as Qt. In practice, this means that, yeah, you can have that odd filename that is technically not Unicode. But the vast majority of code running on the most popular desktop and mobile platforms is going to handle it in a way that expects it to be Unicode. Why should Python go against the trend, and make life more complicated for developers using it in the process?
    - scoot_718 6y ago
      Many of us deal in bytes that simply aren't UTF8 and never could be. Because they're just bytes. How many things are stored as binary files? > All Apple platforms are UTF-16. I'm glad all their executable files are apparently text files. How amazing. > Why should Python go against the trend, and make life more complicated for developers using it in the process? You tell me why Python3 did that.
    - mehrdadn 6y ago
      File names? I listed so much more for you than file names. That HTML you just fetched? How do you know it's Unicode? That .txt file the user just asked to load? How do you know that's Unicode? For heaven's sake, when can you actually guarantee that even sys.stdin.read() is going to read Unicode? You can only do that when you're the one piping your own stdin... which is not the common case. What do you do when your fundamentally invalid assumptions break? Do you just not care and simply present a stack trace to the user and tell them to get lost? I've gotten tired of these debates though, so just a heads up I may not have the energy to reply if you continue...
      - gvjddbnvdrbv 6y ago
        In the real world Python2 gave stack traces by default when presented with common strings. Python3 doesn't.
      - 6y ago
- int_19h 6y ago
  Except for that part where it would happily implicitly convert them to/from a Unicode string in any context where one was needed or present... using ASCII, rather than UTF-8, as the encoding.
- raverbashing 6y ago
  > py2’s flexibility to handle utf8 bytes without fuss is amazing Without fuzz? No, sorry, it was anything but. First of all it would default encoding to "ASCII". Have any whiff of non-explicitly handled UTF-8 and it would just go bang at the worse time possible. That was a stupid decision "Oh but there was setdefaultencoding" Yeah here's the first result for that https://stackoverflow.com/questions/3828723/why-should-we-not-use-sys-setdefaultencodingutf-8-in-a-py-script https://stackoverflow.com/questions/3828723/why-should-we-no... So no, Python2 way of dealing with Unicode was the most annoying way possible, because hey who needs anything but ASCII right?
  - scoot_718 6y ago
    > Have any whiff of non-explicitly handled UTF-8 and it would just go bang at the worse time possible. Whereas python3 is just waiting to explode the moment there are bytes in your UTF-8 that are invalid. Oh the http request got truncated to leave invalid utf-8 in an otherwise fine utf-8 response? FUCKING ERROR.
  - ehsankia 6y ago
    I agree that it was easy to shoot yourself in the foot, and if you did have to deal with unicode, it was often a pain, but at the same time, Python's simplicity and ease of use is what makes it great, with the ability to do something cleaner if you choose to. You can choose to type annotate all your code and make it better. You can choose to organize your code in however package structure you want or keep it all in one file. The language doesn't force any of that onto you. That's the approach I think would've been much nicer and Pythonic in my mind. Now you're forced to use a much clunkier bytes/str paradigm, which yes will make your life much nicer in the 10% of the time when you'll need it, but the other 90% of the time will just be slightly more annoying. Similarly, I may be alone in this, but having to put parens around print statements is also annoying 99% of the time, but nice that 1% of the time I need to pass it as a function or pass it some extra arguments.
  - pdonis 6y ago
    > Python2 way of dealing with Unicode was the most annoying way possible The part about defaulting to ASCII is annoying, yes. And using sys.setdefaultencoding to change the default would still be annoying, yes. The reason for that is that any default encoding will be annoying whenever the actual encoding when the program is running doesn't match the default. The correct way to fix this problem is to not have a default encoding at all. Don't try to auto-detect encodings; don't try to guess encodings. Force every encode and decode operation to explicitly specify an encoding. That way the issue of what the encoding is, how to detect it, etc., is handled in the right place--in the code of the particular application that needs to use Unicode. It should not be handled in a language runtime or a standard library, precisely because there is no way for a language runtime or a library to properly deal with all use cases of all applications. What Python 3 did, instead, was to change the rules of default encodings and auto-detection/guessing of encodings, so that they were nicer to some use cases, and even more annoying than before to others.
pdonis 6y ago
> I feel like you're the first person I've seen on the planet to echo my sentiments on these. There have been plenty of people with similar sentiments. I'm one of them. I have felt ever since I first looked at Python 3 that the ways in which it broke backward incompatibility were heavily skewed towards a few particular use cases and did not take into account the needs of all of the Python community.

Too 6y ago

Got any examples?