5 ms·

I do similarly as you (maintain 2/3 code without dependencies) but every time I have to do string encoding/decoding it kills me to find a way that half works, a

I do similarly as you (maintain 2/3 code without dependencies) but every time I have to do string encoding/decoding it kills me to find a way that half works, and I don't have a ready solution in my mind for these that doesn't break half the time. How do you handle non-ASCII in a compatible manner? Like Unicode stdio? Unicode file paths? Unicode sys.argv? string_escape/unicode_escape? I feel like Python 3 completely wrecked strings instead of making them better.

jteppinette 7y ago
You have to invest some time in understanding how strings (byte strings and unicode) work in Python. I put this off for way too long. After A TON OF PAIN I decided to put the time in to watch the talk "Pragmatic Unicode, or, How do I stop the pain?"[0] by Ned Batchelder and it all just clicked. Now, I make unicode sandwiches like a boss. [0] https://youtu.be/sgHbC6udIqc https://youtu.be/sgHbC6udIqc
- dataflow 7y ago
  Uhm I think you misunderstood the problem. I already understand Unicode, and I can handle Unicode just fine as far as the facilities are there, in both Python 2 and Python 3 -- individually. The places where I have trouble are (1) where either or both languages do something nonsensical, (2) where the facilities just don't seem to be there (sys.argv?), (3) where there are bugs (try writing binary to stdout with IDLE open), or (4) where there seems no reasonable way to write code that behaves correctly in both languages without making your own little translation layer. Like as an example for #1, "a".encode('unicode_escape') == b'a' makes no sense. Why should escaping a string suddenly turn it into bytes? I never even specified an encoding for those bytes. And similarly why should "a".decode('unicode_escape') be an error? It makes perfect sense. What they did looked nice in theory (no implicit conversions between bytes and strings etc.) but IMHO they practically completely wrecked strings in Python 3.
  - dataflow 7y ago
    Also, I think the facilities are lacking for case-insensitive comparisons in Python 2, so I guess this is one thing they improved. Still don't know how to do it "correctly" when writing Python 2-compatible code.
    - mgkimsal 7y ago
      > Still don't know how to do it "correctly" when writing Python 2-compatible code. Does writing python 2 code stop being 'worth it' at some point? If so, where is that point? It sort of sounds like you're already there. There's still people using P2 at least in part because of all the hoops people are jumping through to keep supporting it, no? I'm not a professional python dev, so it doesn't impact me as much as some of my colleagues, but 'backwards compatible' issues crop up in other platforms/langauges as well. Wordpress might be the biggest example in PHP. They've kept a minimum target language which is far behing 'current' or even 'currently supported', and it's been a catch 22. Hosts keep supporting PHP 5.4, for example, far later than they 'should' have, because people kept writing new stuff targeting PHP 5.4. WP 5 was, alas, a missed opportunity to target PHP7 as a minimum. :/
      - dataflow 7y ago
        > Does writing python 2 code stop being 'worth it' at some point? If so, where is that point? It sort of sounds like you're already there. How do you know I'm there? If nothing else I still have my own previous Python 2 code that I've spent time on and that's useful to me! Why would I just throw them all away or waste massive amounts of time rewriting them into Python 3? Is my goal supposed to be to please the masses here? Honestly that's sufficient reason already. But if you want a more "standard"/politically-correct response: Python 2 is still being supported, Ubuntu 14.04 LTS literally only just reached EOL last month and can be found in the wild, Python 3 support is lagging in a lot of places (e.g. PyPy is still on 3.5, although I'm just using it as an example; I don't use it much), and I still come across Python 2-based tools and packages once in a while. So I guess the answer to your question is it "stops being worth it" when I stop coming across situations where I'd regret abandoning Python 2. > There's still people using P2 at least in part because of all the hoops people are jumping through to keep supporting it, no? "Because" is an odd way to put it... it's true that if they didn't support P2 then people would use P2 less, but people use it because it benefits them, not "because it's supported". I mean, you're also alive "at least in part because" of all the hoops people jump through to grow and bring food within your reach, but I doubt your conclusion is that this should stop being the case... > They've kept a minimum target language which is far behind 'current' or even 'currently supported', and it's been a catch 22. What I don't understand is why are people supposed to keep abandoning good software just because someone made something shiny and declared it "unsupported"? I hate this "you have to like my updates... or I will force you" attitude that every organization seems to have nowadays. People are trying to solve their own problems, not please the leader they're following. > Hosts keep supporting PHP 5.4, for example, far later than they 'should' have, because people kept writing new stuff targeting PHP 5.4. I mean, this isn't even the same situation? I'm not targeting Python 2 or introducing a dependency on it generally. I try hard to keep my code both Python 3 and Python 2 compatible. So the decision as to whether to move on or not is left to the client (which is often myself) and there's no obstacle either way.
deathanatos 7y ago
For the most part, when you do I/O to some external system, if it's text, you encode/decode at the border to that system. Interally, all text data is `str` (or `unicode` in 2) and all binary data is `bytes` (in both). In some cases of common OS-induced pain, I'd say "do whatever 3 does" in 2, since that'll make migration easier in the long run. (But I understand that can be hard, and I think my responses to your examples below even demonstrate that to be hard.) To your specific pain points: > Unicode stdio? Mostly, `io` should handle this in both 3/2. You might need to help it get the right encoding in 2. > Unicode file paths This is going to be a mess in any language, because file paths really aren't text. On nix, they're byte strings that don't have nuls in them. Hopefully* they're encoded according to LANG, and hopefully LANG is a UTF-8 variant, but it isn't required, and it isn't required that two users on the same system use compatible LANGs, so you get a Tower of Babel. I really wish OSs would just start enforcing a. Unicode filenames, and b. no newlines in filenames; those two alone would make life so much easier. Hopefully you've seen os.fsencode / fsdecode, but alas those aren't in 2, so I'm not sure they really help you. Often one is not really munging paths that much, and can just pass through whatever value/type you get, but it does happen, of course. (E.g., adding or removing extensions) > Unicode sys.argv This is also a pain point, since again, the underlying type in nix is a byte string without nuls. I'd hope it decodes w/ the LANG encoding, but since the user could easily tab-complete a filename, fsencode/decode might be more appropriate. I think I'd say "do whatever 3 does". 1 Jan 2020 is nearly here. Forget about 2 / assume UTF-8 in 2 and don't support anything else? > I feel like Python 3 completely wrecked strings instead of making them better.* A clear separation of text and binary is needed in the long run, and makes other operations much clearer and saner. The pain you're feeling is introduced from the OS not having the same clarity.
- dataflow 7y ago
  >> Unicode stdio? > Mostly, `io` should handle this in both 3/2. You might need to help it get the right encoding in 2. Does io handle stdio? I was referring to standard input and standard error/output. How do you read/write Unicode in a cross-2/3 way from/to standard input/output/error without adding your own translation layer? >> Unicode file paths > This is going to be a mess in any language, because file paths really aren't text. Sorry I need to be more clear. That general mess is not the aspect of it I was referring to. I'm specifically referring to a 2/3 compatibility mess. I meant that, for example, to have any semblance of Unicode handling, in Python 2 you do os.listdir(u"."), whereas in Python 3 you do os.listdir(b"."). I know how to handle it in both Python 2 and Python 3 in a way that's Good Enough (TM), but how do I even get that with cross-compatible code? I'd need to write a translation layer of sorts for every single I/O function I might use. >> Unicode sys.argv > I think I'd say "do whatever 3 does". Hmm okay thanks, I'll need to try to see what the implications are again. I think the problems I recalled from this may have been just been a result of the other issues, not sure. I can't "forget about 2" though, it's still on Ubuntu LTS systems and there are still packages in 2 that haven't been ported to 3. >> I feel like Python 3 completely wrecked strings instead of making them better. > A clear separation of text and binary is needed in the long run, and makes other operations much clearer and saner. The pain you're feeling is introduced from the OS not having the same clarity. Again, I think this is "cleaner" in theory, not in practice. What happened to the string_escape/unicode_escape nonsense I pointed out with the new system? Any rebuttal to that one? ;-)
maxnoe 7y ago
Everytime I hear somebody claim python 3 broke strings I just cringe. No. No on just so many levels. It finally fixed it by introducing two (or three if you count bytearray) types for completely different semantics of data. Python 2 was just a mess whenever you left ascii.
- dataflow 7y ago
  Meanwhile I cringe every time I see people say "No" while ignoring the issues people like me point out.
  - maxnoe 7y ago
    I don't say the transition is easy or maintaining a codebase that handles these issues correct that supports python 3 and 2 at the same time. But looking solely at python3 , everything byte and unicode related just got so much easier and better testable and with better error messages at better points in your code than in python 2. So claiming python 3 broke stuff just not what the case is. Python 2 had a deeply flawed, boundryless view of unicode vs bytes and python 3 fixed that.
    - dataflow 7y ago
      So for example what is your rebuttal to my actual example pointing out that "a".encode('unicode_escape') returns bytes instead of strings? Why/how the heck is it even encoding anything when it never even asked me for an encoding? If for nonsensical reasons it really wants to encode things, shouldn't it at least be asking me for an encoding if it wants a "clean separation" between strings and bytes? You find Python 2's behavior of returning a string to be somehow deeply flawed but Python 3's returning of bytes to be sensible? Really? What problem are they solving here?
      - lozenge 7y ago
        Encode and decode always move between str and bytes. Otherwise you could never write a function that takes an encoding as the argument and uses it in the middle of its code (because the types of your locals would vary at runtime in incompatible ways). What is your reason for using this obscure encoding anyway?

jteppinette 7y ago

You have to invest some time in understanding how strings (byte strings and unicode) work in Python. I put this off for way too long. After A TON OF PAIN I decided to put the time in to watch the talk "Pragmatic Unicode, or, How do I stop the pain?"[0] by Ned Batchelder and it all just clicked. Now, I make unicode sandwiches like a boss. [0] https://youtu.be/sgHbC6udIqc https://youtu.be/sgHbC6udIqc

dataflow 7y ago
Uhm I think you misunderstood the problem. I already understand Unicode, and I can handle Unicode just fine as far as the facilities are there, in both Python 2 and Python 3 -- individually. The places where I have trouble are (1) where either or both languages do something nonsensical, (2) where the facilities just don't seem to be there (sys.argv?), (3) where there are bugs (try writing binary to stdout with IDLE open), or (4) where there seems no reasonable way to write code that behaves correctly in both languages without making your own little translation layer. Like as an example for #1, "a".encode('unicode_escape') == b'a' makes no sense. Why should escaping a string suddenly turn it into bytes? I never even specified an encoding for those bytes. And similarly why should "a".decode('unicode_escape') be an error? It makes perfect sense. What they did looked nice in theory (no implicit conversions between bytes and strings etc.) but IMHO they practically completely wrecked strings in Python 3.
- dataflow 7y ago
  Also, I think the facilities are lacking for case-insensitive comparisons in Python 2, so I guess this is one thing they improved. Still don't know how to do it "correctly" when writing Python 2-compatible code.
  - mgkimsal 7y ago
    > Still don't know how to do it "correctly" when writing Python 2-compatible code. Does writing python 2 code stop being 'worth it' at some point? If so, where is that point? It sort of sounds like you're already there. There's still people using P2 at least in part because of all the hoops people are jumping through to keep supporting it, no? I'm not a professional python dev, so it doesn't impact me as much as some of my colleagues, but 'backwards compatible' issues crop up in other platforms/langauges as well. Wordpress might be the biggest example in PHP. They've kept a minimum target language which is far behing 'current' or even 'currently supported', and it's been a catch 22. Hosts keep supporting PHP 5.4, for example, far later than they 'should' have, because people kept writing new stuff targeting PHP 5.4. WP 5 was, alas, a missed opportunity to target PHP7 as a minimum. :/
    - dataflow 7y ago
      > Does writing python 2 code stop being 'worth it' at some point? If so, where is that point? It sort of sounds like you're already there. How do you know I'm there? If nothing else I still have my own previous Python 2 code that I've spent time on and that's useful to me! Why would I just throw them all away or waste massive amounts of time rewriting them into Python 3? Is my goal supposed to be to please the masses here? Honestly that's sufficient reason already. But if you want a more "standard"/politically-correct response: Python 2 is still being supported, Ubuntu 14.04 LTS literally only just reached EOL last month and can be found in the wild, Python 3 support is lagging in a lot of places (e.g. PyPy is still on 3.5, although I'm just using it as an example; I don't use it much), and I still come across Python 2-based tools and packages once in a while. So I guess the answer to your question is it "stops being worth it" when I stop coming across situations where I'd regret abandoning Python 2. > There's still people using P2 at least in part because of all the hoops people are jumping through to keep supporting it, no? "Because" is an odd way to put it... it's true that if they didn't support P2 then people would use P2 less, but people use it because it benefits them, not "because it's supported". I mean, you're also alive "at least in part because" of all the hoops people jump through to grow and bring food within your reach, but I doubt your conclusion is that this should stop being the case... > They've kept a minimum target language which is far behind 'current' or even 'currently supported', and it's been a catch 22. What I don't understand is why are people supposed to keep abandoning good software just because someone made something shiny and declared it "unsupported"? I hate this "you have to like my updates... or I will force you" attitude that every organization seems to have nowadays. People are trying to solve their own problems, not please the leader they're following. > Hosts keep supporting PHP 5.4, for example, far later than they 'should' have, because people kept writing new stuff targeting PHP 5.4. I mean, this isn't even the same situation? I'm not targeting Python 2 or introducing a dependency on it generally. I try hard to keep my code both Python 3 and Python 2 compatible. So the decision as to whether to move on or not is left to the client (which is often myself) and there's no obstacle either way.

deathanatos 7y ago

For the most part, when you do I/O to some external system, if it's text, you encode/decode at the border to that system. Interally, all text data is `str` (or `unicode` in 2) and all binary data is `bytes` (in both). In some cases of common OS-induced pain, I'd say "do whatever 3 does" in 2, since that'll make migration easier in the long run. (But I understand that can be hard, and I think my responses to your examples below even demonstrate that to be hard.) To your specific pain points: > Unicode stdio? Mostly, `io` should handle this in both 3/2. You might need to help it get the right encoding in 2. > Unicode file paths This is going to be a mess in any language, because file paths really aren't text. On nix, they're byte strings that don't have nuls in them. Hopefully* they're encoded according to LANG, and hopefully LANG is a UTF-8 variant, but it isn't required, and it isn't required that two users on the same system use compatible LANGs, so you get a Tower of Babel. I really wish OSs would just start enforcing a. Unicode filenames, and b. no newlines in filenames; those two alone would make life so much easier. Hopefully you've seen os.fsencode / fsdecode, but alas those aren't in 2, so I'm not sure they really help you. Often one is not really munging paths that much, and can just pass through whatever value/type you get, but it does happen, of course. (E.g., adding or removing extensions) > Unicode sys.argv This is also a pain point, since again, the underlying type in nix is a byte string without nuls. I'd hope it decodes w/ the LANG encoding, but since the user could easily tab-complete a filename, fsencode/decode might be more appropriate. I think I'd say "do whatever 3 does". 1 Jan 2020 is nearly here. Forget about 2 / assume UTF-8 in 2 and don't support anything else? > I feel like Python 3 completely wrecked strings instead of making them better.* A clear separation of text and binary is needed in the long run, and makes other operations much clearer and saner. The pain you're feeling is introduced from the OS not having the same clarity.

dataflow 7y ago
>> Unicode stdio? > Mostly, `io` should handle this in both 3/2. You might need to help it get the right encoding in 2. Does io handle stdio? I was referring to standard input and standard error/output. How do you read/write Unicode in a cross-2/3 way from/to standard input/output/error without adding your own translation layer? >> Unicode file paths > This is going to be a mess in any language, because file paths really aren't text. Sorry I need to be more clear. That general mess is not the aspect of it I was referring to. I'm specifically referring to a 2/3 compatibility mess. I meant that, for example, to have any semblance of Unicode handling, in Python 2 you do os.listdir(u"."), whereas in Python 3 you do os.listdir(b"."). I know how to handle it in both Python 2 and Python 3 in a way that's Good Enough (TM), but how do I even get that with cross-compatible code? I'd need to write a translation layer of sorts for every single I/O function I might use. >> Unicode sys.argv > I think I'd say "do whatever 3 does". Hmm okay thanks, I'll need to try to see what the implications are again. I think the problems I recalled from this may have been just been a result of the other issues, not sure. I can't "forget about 2" though, it's still on Ubuntu LTS systems and there are still packages in 2 that haven't been ported to 3. >> I feel like Python 3 completely wrecked strings instead of making them better. > A clear separation of text and binary is needed in the long run, and makes other operations much clearer and saner. The pain you're feeling is introduced from the OS not having the same clarity. Again, I think this is "cleaner" in theory, not in practice. What happened to the string_escape/unicode_escape nonsense I pointed out with the new system? Any rebuttal to that one? ;-)

maxnoe 7y ago

Everytime I hear somebody claim python 3 broke strings I just cringe. No. No on just so many levels. It finally fixed it by introducing two (or three if you count bytearray) types for completely different semantics of data. Python 2 was just a mess whenever you left ascii.

dataflow 7y ago
Meanwhile I cringe every time I see people say "No" while ignoring the issues people like me point out.
- maxnoe 7y ago
  I don't say the transition is easy or maintaining a codebase that handles these issues correct that supports python 3 and 2 at the same time. But looking solely at python3 , everything byte and unicode related just got so much easier and better testable and with better error messages at better points in your code than in python 2. So claiming python 3 broke stuff just not what the case is. Python 2 had a deeply flawed, boundryless view of unicode vs bytes and python 3 fixed that.
  - dataflow 7y ago
    So for example what is your rebuttal to my actual example pointing out that "a".encode('unicode_escape') returns bytes instead of strings? Why/how the heck is it even encoding anything when it never even asked me for an encoding? If for nonsensical reasons it really wants to encode things, shouldn't it at least be asking me for an encoding if it wants a "clean separation" between strings and bytes? You find Python 2's behavior of returning a string to be somehow deeply flawed but Python 3's returning of bytes to be sensible? Really? What problem are they solving here?
    - lozenge 7y ago
      Encode and decode always move between str and bytes. Otherwise you could never write a function that takes an encoding as the argument and uses it in the middle of its code (because the types of your locals would vary at runtime in incompatible ways). What is your reason for using this obscure encoding anyway?