5 ms·

Consider this, for example: https://click.palletsprojects.com/en/7.x/python3/ https://click.palletsprojects.com/en/7.x/python3/ and here, have a nice sandwich

Consider this, for example: https://click.palletsprojects.com/en/7.x/python3/ https://click.palletsprojects.com/en/7.x/python3/ and here, have a nice sandwich of this: https://stackoverflow.com/questions/57652720/runtimeerror-click-will-abort-further-execution-because-python-3-was-configured https://stackoverflow.com/questions/57652720/runtimeerror-cl... or this bullshit: https://bugzilla.redhat.com/show_bug.cgi?id=1404918 https://bugzilla.redhat.com/show_bug.cgi?id=1404918 The truth no one wants to hear is that Python2 did Unicode right (or as well as could be expected), while Python3 is a train wreck, even if you just stick to LANG=C and ASCII.

joshuamorton 7y ago
The surrogate escape issue was fixed in 3.7 and you shouldn't use LANG=C as a locale, you should use LANG=utf-8. Python doesn't have bad unicode handling, the system does when you set LANG=C. And no, for people who actually have to use unicode, python2 didn't do unicode anything like "right". Python2 did unicode "right" for the people with LANG=C and ASCII. Those people are generally wrong. Even pallets notes >Misconfigured environments can currently cause a wide range of Unicode problems The keyword there is misconfigured. Your system is misconfigured. Fix it (by making your locale correct). Then the problems go away.
- downerending 7y ago
  So, I was right to scorn 3.0, 3.1, 3.2, 3.3, 3.4, 3.5, and 3.6? And no, LANG=C isn't wrong. Your software has one job, to at least be correct for LANG=C. If it can't do that, it sucks.
  - joshuamorton 7y ago
    Lang=C is broken if you have non-ascii unicode filenames (or many other things, but those are the most obvious). That's not pythons fault, it's your bad encoding. Python trusts the system, and the system lies. It's your fault for lying to python. Configure your system correctly, don't lie to your tools and then blame them. Lang=C is the realm of English only 70s crap programs. You shouldn't use it. Use a modern encoding that supports modern characters.
    - downerending 7y ago
      POSIX filenames are a series of bytes. That's all you get, and it makes things strikingly simple. Trying to wedge Unicode in at this level is a recipe for disaster, and after all these years, Python3 still doesn't have a reliable, coherent story on this.
      - joshuamorton 7y ago
        > POSIX filenames are a series of bytes. That's all you get, and it makes things strikingly simple. But not windows, or a number of filesystems. Hell the vast majority of the time posix filenames are ASCII, and therefore utf8 compatible. The rare cases where they aren't break all kinds of other tools because... > Trying to wedge Unicode in at this level is a recipe for disaster, Blame every terminal emulator and os in common use. They all display filenames in unicode. So pythons in good company here. Since you have to display filenames, they need an encoding.
        downerending 7y ago
        My chosen encoding is usually LANG=C. That is, just show me the bytes, escaped as needed (as with Python 2). Ń̸̗̮̯͇͈͇̑͒o̸̧̭̯̖̾̐̔̈́̋̈́̈́͘t̵̖̹͙̟͑̈́̎̀͝ͅ ̴̡͒͌̋̾͆̒̎ȧ̴̬̰̪̮̩͉̐̈́̋̇̀̊̕ ̷̫͑́͑̕͠f̸̬̥̼̣̙͎͍͙̤͗͆ͅā̸̛̼̱̠̯͖̫͓͙͒̔͂̅͜͝ͅń̴̨̨̨̦̺̺͗͗̉̎̓̀͘͘͠ ̵̢̖̞͍̻̮͈̘͎̹̉̇̈͐̍́̂̂̀̂o̴̳͙̿͆͝f̵̮̙̞̰͖͉̰̞͕͑̍̀̄̎̍͜ ̵̛̗̙͓̰̹͇̍̒̋͆ͅU̶̧̯͇̰̞̮͋n̵͉̰̰̳̗͎̂͑̉͛̏͛̌i̵̢̢̪͂́̽̀͝͠c̴̢͇͙̙̲̯̫͈̪̀̌̓̂͗̚͜͠ơ̷̬͖̙̯͇̻̱͕͐̎d̶̛̤͉̻͆̋͋̂̀e̵̦̺̹̅͆̀͛̎̎̅.̸̡̟̰͙̮̱͍̼̾͊̑̅͊̚͝
        joshuamorton 7y ago
        > That is, just show me the bytes, escaped as needed this isn't actually how python 2 works either. `print("Ń̸̗̮̯͇͈͇̑͒o̸̧̭̯̖̾̐̔̈́̋̈́̈́͘)` will print the same string, since my machine's locale is utf-8. So python2 does the same thing as 3 with regard to machine encoding.
    - downerending 7y ago
      In addition, the whole idea that a system must have a privileged locale is silly in our connected world. Even assuming a single process (which can be talking to endpoints in many countries) should be locked to a specific locale is nonsense. The beauty of LANG=C is that it's the only true lingua franca of modern computing. It's a place where all software and programmers can "meet" without privileging any particular locale. We have 'gettext' and friends for speaking to humans in particular locales. Embedding i18n into core system design just doesn't work.

joshuamorton 7y ago

The surrogate escape issue was fixed in 3.7 and you shouldn't use LANG=C as a locale, you should use LANG=utf-8. Python doesn't have bad unicode handling, the system does when you set LANG=C. And no, for people who actually have to use unicode, python2 didn't do unicode anything like "right". Python2 did unicode "right" for the people with LANG=C and ASCII. Those people are generally wrong. Even pallets notes >Misconfigured environments can currently cause a wide range of Unicode problems The keyword there is misconfigured. Your system is misconfigured. Fix it (by making your locale correct). Then the problems go away.

downerending 7y ago
So, I was right to scorn 3.0, 3.1, 3.2, 3.3, 3.4, 3.5, and 3.6? And no, LANG=C isn't wrong. Your software has one job, to at least be correct for LANG=C. If it can't do that, it sucks.
- joshuamorton 7y ago
  Lang=C is broken if you have non-ascii unicode filenames (or many other things, but those are the most obvious). That's not pythons fault, it's your bad encoding. Python trusts the system, and the system lies. It's your fault for lying to python. Configure your system correctly, don't lie to your tools and then blame them. Lang=C is the realm of English only 70s crap programs. You shouldn't use it. Use a modern encoding that supports modern characters.
  - downerending 7y ago
    POSIX filenames are a series of bytes. That's all you get, and it makes things strikingly simple. Trying to wedge Unicode in at this level is a recipe for disaster, and after all these years, Python3 still doesn't have a reliable, coherent story on this.
    - joshuamorton 7y ago
      > POSIX filenames are a series of bytes. That's all you get, and it makes things strikingly simple. But not windows, or a number of filesystems. Hell the vast majority of the time posix filenames are ASCII, and therefore utf8 compatible. The rare cases where they aren't break all kinds of other tools because... > Trying to wedge Unicode in at this level is a recipe for disaster, Blame every terminal emulator and os in common use. They all display filenames in unicode. So pythons in good company here. Since you have to display filenames, they need an encoding.
      - downerending 7y ago
        My chosen encoding is usually LANG=C. That is, just show me the bytes, escaped as needed (as with Python 2). Ń̸̗̮̯͇͈͇̑͒o̸̧̭̯̖̾̐̔̈́̋̈́̈́͘t̵̖̹͙̟͑̈́̎̀͝ͅ ̴̡͒͌̋̾͆̒̎ȧ̴̬̰̪̮̩͉̐̈́̋̇̀̊̕ ̷̫͑́͑̕͠f̸̬̥̼̣̙͎͍͙̤͗͆ͅā̸̛̼̱̠̯͖̫͓͙͒̔͂̅͜͝ͅń̴̨̨̨̦̺̺͗͗̉̎̓̀͘͘͠ ̵̢̖̞͍̻̮͈̘͎̹̉̇̈͐̍́̂̂̀̂o̴̳͙̿͆͝f̵̮̙̞̰͖͉̰̞͕͑̍̀̄̎̍͜ ̵̛̗̙͓̰̹͇̍̒̋͆ͅU̶̧̯͇̰̞̮͋n̵͉̰̰̳̗͎̂͑̉͛̏͛̌i̵̢̢̪͂́̽̀͝͠c̴̢͇͙̙̲̯̫͈̪̀̌̓̂͗̚͜͠ơ̷̬͖̙̯͇̻̱͕͐̎d̶̛̤͉̻͆̋͋̂̀e̵̦̺̹̅͆̀͛̎̎̅.̸̡̟̰͙̮̱͍̼̾͊̑̅͊̚͝
        joshuamorton 7y ago
        > That is, just show me the bytes, escaped as needed this isn't actually how python 2 works either. `print("Ń̸̗̮̯͇͈͇̑͒o̸̧̭̯̖̾̐̔̈́̋̈́̈́͘)` will print the same string, since my machine's locale is utf-8. So python2 does the same thing as 3 with regard to machine encoding.
  - downerending 7y ago
    In addition, the whole idea that a system must have a privileged locale is silly in our connected world. Even assuming a single process (which can be talking to endpoints in many countries) should be locked to a specific locale is nonsense. The beauty of LANG=C is that it's the only true lingua franca of modern computing. It's a place where all software and programmers can "meet" without privileging any particular locale. We have 'gettext' and friends for speaking to humans in particular locales. Embedding i18n into core system design just doesn't work.