6 ms·

So, I was right to scorn 3.0, 3.1, 3.2, 3.3, 3.4, 3.5, and 3.6? And no, LANG=C isn't wrong. Your software has one job, to at least be correct for LANG=C. If

So, I was right to scorn 3.0, 3.1, 3.2, 3.3, 3.4, 3.5, and 3.6? And no, LANG=C isn't wrong. Your software has one job, to at least be correct for LANG=C. If it can't do that, it sucks.

joshuamorton 7y ago
Lang=C is broken if you have non-ascii unicode filenames (or many other things, but those are the most obvious). That's not pythons fault, it's your bad encoding. Python trusts the system, and the system lies. It's your fault for lying to python. Configure your system correctly, don't lie to your tools and then blame them. Lang=C is the realm of English only 70s crap programs. You shouldn't use it. Use a modern encoding that supports modern characters.
- downerending 7y ago
  POSIX filenames are a series of bytes. That's all you get, and it makes things strikingly simple. Trying to wedge Unicode in at this level is a recipe for disaster, and after all these years, Python3 still doesn't have a reliable, coherent story on this.
  - joshuamorton 7y ago
    > POSIX filenames are a series of bytes. That's all you get, and it makes things strikingly simple. But not windows, or a number of filesystems. Hell the vast majority of the time posix filenames are ASCII, and therefore utf8 compatible. The rare cases where they aren't break all kinds of other tools because... > Trying to wedge Unicode in at this level is a recipe for disaster, Blame every terminal emulator and os in common use. They all display filenames in unicode. So pythons in good company here. Since you have to display filenames, they need an encoding.
    - downerending 7y ago
      My chosen encoding is usually LANG=C. That is, just show me the bytes, escaped as needed (as with Python 2). Ń̸̗̮̯͇͈͇̑͒o̸̧̭̯̖̾̐̔̈́̋̈́̈́͘t̵̖̹͙̟͑̈́̎̀͝ͅ ̴̡͒͌̋̾͆̒̎ȧ̴̬̰̪̮̩͉̐̈́̋̇̀̊̕ ̷̫͑́͑̕͠f̸̬̥̼̣̙͎͍͙̤͗͆ͅā̸̛̼̱̠̯͖̫͓͙͒̔͂̅͜͝ͅń̴̨̨̨̦̺̺͗͗̉̎̓̀͘͘͠ ̵̢̖̞͍̻̮͈̘͎̹̉̇̈͐̍́̂̂̀̂o̴̳͙̿͆͝f̵̮̙̞̰͖͉̰̞͕͑̍̀̄̎̍͜ ̵̛̗̙͓̰̹͇̍̒̋͆ͅU̶̧̯͇̰̞̮͋n̵͉̰̰̳̗͎̂͑̉͛̏͛̌i̵̢̢̪͂́̽̀͝͠c̴̢͇͙̙̲̯̫͈̪̀̌̓̂͗̚͜͠ơ̷̬͖̙̯͇̻̱͕͐̎d̶̛̤͉̻͆̋͋̂̀e̵̦̺̹̅͆̀͛̎̎̅.̸̡̟̰͙̮̱͍̼̾͊̑̅͊̚͝
      - joshuamorton 7y ago
        > That is, just show me the bytes, escaped as needed this isn't actually how python 2 works either. `print("Ń̸̗̮̯͇͈͇̑͒o̸̧̭̯̖̾̐̔̈́̋̈́̈́͘)` will print the same string, since my machine's locale is utf-8. So python2 does the same thing as 3 with regard to machine encoding.
- downerending 7y ago
  In addition, the whole idea that a system must have a privileged locale is silly in our connected world. Even assuming a single process (which can be talking to endpoints in many countries) should be locked to a specific locale is nonsense. The beauty of LANG=C is that it's the only true lingua franca of modern computing. It's a place where all software and programmers can "meet" without privileging any particular locale. We have 'gettext' and friends for speaking to humans in particular locales. Embedding i18n into core system design just doesn't work.
  - joshuamorton 7y ago
    > Even assuming a single process (which can be talking to endpoints in many countries) should be locked to a specific locale is nonsense. This doesn't matter. I'm not sure why you think this matters or what problems will be caused, but a server using a unicode locale (C.utf8 or en.utf8) doesn't localize things for those clients, it just is able to not crash when printing their usernames to log files.
    - downerending 7y ago
      > it just is able to not crash when printing their usernames to log files The usernames could be from a variety of locales. If showing them in that form is seen as valuable, they'll each have to be decoded in their appropriate locale. Using LANG=C doesn't prevent this, it encourages it. Beyond that, as you point out, Unicode provides lots of new opportunities for crashes. Indeed, one of my favorite ways to crash software (software almost invariably running under a "proper" locale) is to feed it a string of (non-ASCII) Unicode. Boom. No matter the approach, you must code for correctness and deal with all input. For Unicode, you must always allow for decoding failure.
      - joshuamorton 7y ago
        > The usernames could be from a variety of locales. If showing them in that form is seen as valuable, they'll each have to be decoded in their appropriate locale. You realize that locales and encodings are different right? En.utf8, c.utf8, and fr.utf8 are all different locales, but are all utf-8 encoded. They'll all work correctly when printed on a machine with any other utf8 locale. The differences arise in things like sort order and regex, where c.utf-8 might be best for a server if you want C or ASCII like functionality. > Using LANG=C doesn't prevent this, it encourages it. Well no, because you don't know what encoding each person is using. Either you guess which is error prone, or you have to control the other endpoint so that they include the encoding. But if you control the client, just use utf-8. Pretty much no software does what you suggest. Go and rust, for example, go further and entirely ignore the system locale and just always use utf-8 for everything. People don't complain.

So, I was right to scorn 3.0, 3.1, 3.2, 3.3, 3.4, 3.5, and 3.6? And no, LANG=C isn't wrong. Your software has one job, to at least be correct for LANG=C. If it can't do that, it sucks.

joshuamorton 7y ago

Lang=C is broken if you have non-ascii unicode filenames (or many other things, but those are the most obvious). That's not pythons fault, it's your bad encoding. Python trusts the system, and the system lies. It's your fault for lying to python. Configure your system correctly, don't lie to your tools and then blame them. Lang=C is the realm of English only 70s crap programs. You shouldn't use it. Use a modern encoding that supports modern characters.

downerending 7y ago
POSIX filenames are a series of bytes. That's all you get, and it makes things strikingly simple. Trying to wedge Unicode in at this level is a recipe for disaster, and after all these years, Python3 still doesn't have a reliable, coherent story on this.
- joshuamorton 7y ago
  > POSIX filenames are a series of bytes. That's all you get, and it makes things strikingly simple. But not windows, or a number of filesystems. Hell the vast majority of the time posix filenames are ASCII, and therefore utf8 compatible. The rare cases where they aren't break all kinds of other tools because... > Trying to wedge Unicode in at this level is a recipe for disaster, Blame every terminal emulator and os in common use. They all display filenames in unicode. So pythons in good company here. Since you have to display filenames, they need an encoding.
  - downerending 7y ago
    My chosen encoding is usually LANG=C. That is, just show me the bytes, escaped as needed (as with Python 2). Ń̸̗̮̯͇͈͇̑͒o̸̧̭̯̖̾̐̔̈́̋̈́̈́͘t̵̖̹͙̟͑̈́̎̀͝ͅ ̴̡͒͌̋̾͆̒̎ȧ̴̬̰̪̮̩͉̐̈́̋̇̀̊̕ ̷̫͑́͑̕͠f̸̬̥̼̣̙͎͍͙̤͗͆ͅā̸̛̼̱̠̯͖̫͓͙͒̔͂̅͜͝ͅń̴̨̨̨̦̺̺͗͗̉̎̓̀͘͘͠ ̵̢̖̞͍̻̮͈̘͎̹̉̇̈͐̍́̂̂̀̂o̴̳͙̿͆͝f̵̮̙̞̰͖͉̰̞͕͑̍̀̄̎̍͜ ̵̛̗̙͓̰̹͇̍̒̋͆ͅU̶̧̯͇̰̞̮͋n̵͉̰̰̳̗͎̂͑̉͛̏͛̌i̵̢̢̪͂́̽̀͝͠c̴̢͇͙̙̲̯̫͈̪̀̌̓̂͗̚͜͠ơ̷̬͖̙̯͇̻̱͕͐̎d̶̛̤͉̻͆̋͋̂̀e̵̦̺̹̅͆̀͛̎̎̅.̸̡̟̰͙̮̱͍̼̾͊̑̅͊̚͝
    - joshuamorton 7y ago
      > That is, just show me the bytes, escaped as needed this isn't actually how python 2 works either. `print("Ń̸̗̮̯͇͈͇̑͒o̸̧̭̯̖̾̐̔̈́̋̈́̈́͘)` will print the same string, since my machine's locale is utf-8. So python2 does the same thing as 3 with regard to machine encoding.
downerending 7y ago
In addition, the whole idea that a system must have a privileged locale is silly in our connected world. Even assuming a single process (which can be talking to endpoints in many countries) should be locked to a specific locale is nonsense. The beauty of LANG=C is that it's the only true lingua franca of modern computing. It's a place where all software and programmers can "meet" without privileging any particular locale. We have 'gettext' and friends for speaking to humans in particular locales. Embedding i18n into core system design just doesn't work.
- joshuamorton 7y ago
  > Even assuming a single process (which can be talking to endpoints in many countries) should be locked to a specific locale is nonsense. This doesn't matter. I'm not sure why you think this matters or what problems will be caused, but a server using a unicode locale (C.utf8 or en.utf8) doesn't localize things for those clients, it just is able to not crash when printing their usernames to log files.
  - downerending 7y ago
    > it just is able to not crash when printing their usernames to log files The usernames could be from a variety of locales. If showing them in that form is seen as valuable, they'll each have to be decoded in their appropriate locale. Using LANG=C doesn't prevent this, it encourages it. Beyond that, as you point out, Unicode provides lots of new opportunities for crashes. Indeed, one of my favorite ways to crash software (software almost invariably running under a "proper" locale) is to feed it a string of (non-ASCII) Unicode. Boom. No matter the approach, you must code for correctness and deal with all input. For Unicode, you must always allow for decoding failure.
    - joshuamorton 7y ago
      > The usernames could be from a variety of locales. If showing them in that form is seen as valuable, they'll each have to be decoded in their appropriate locale. You realize that locales and encodings are different right? En.utf8, c.utf8, and fr.utf8 are all different locales, but are all utf-8 encoded. They'll all work correctly when printed on a machine with any other utf8 locale. The differences arise in things like sort order and regex, where c.utf-8 might be best for a server if you want C or ASCII like functionality. > Using LANG=C doesn't prevent this, it encourages it. Well no, because you don't know what encoding each person is using. Either you guess which is error prone, or you have to control the other endpoint so that they include the encoding. But if you control the client, just use utf-8. Pretty much no software does what you suggest. Go and rust, for example, go further and entirely ignore the system locale and just always use utf-8 for everything. People don't complain.