4 ms·

Python2 supports unicode. Chávez and Çelik are covered. I'll take Python2's POSIX text model over Microsoft's default unicode strings every time because I pref

• vetinari 10y ago
  > Python2 supports unicode. Chávez and Çelik are covered. Most python2 code doesn't. It has a nice property of exploding spectacularly when the string you are writing out to console or file contains Chávez or Çelik.
  • the_mitsuhiko 10y ago
    > Most python2 code doesn't. Citation needed. I can't say I ever had practical issues with not being able to handle unicode on Python if I needed. No matter which library. Please give practical examples of where you cannot deal with unicode on Python 2.
    • vetinari 10y ago
      just plain for line in yourfile: print line with yourfile containing non-ascii characters and running it in windows console.
      • colejohnson66 10y ago
        That sounds like a limitation of the Windows console, not the programming language.
        • comex 10y ago
          It's a limitation of how the programming language interacts with the Windows console, and it's a mistake. If you are writing potentially-garbled strings directly to the console, there is already no guarantee that the output will properly reflect the string or e.g. copying and pasting will work correctly, because the string could include backspace characters, or a ton of newlines. Therefore it doesn't make sense to do anything other than make a 'best effort' attempt when rendering strings to the console, including with Unicode handling. If it works, great, but if the string is invalid Unicode, that's only one of multiple problematic cases, so what exactly is throwing an exception supposed to accomplish? Python should use replacement characters by default in this case.
          • rjurney 10y ago
            Stop using the Windows console. Or fix it. More rational than a breaking upgrade to Python 3.
            • paulddraper 10y ago
              Fixing the Windows console rather than upgrading to Python 3 is the only sensible choice. </s>
        • Animats 10y ago
          It is, and Java has similar problems on Windows. The Windows console can be switched to Unicode, but by default, it emulates the IBM 5150 PC character set, circa 1980. It's not even ISO LATIN-1. Works fine on Linux.
      • the_mitsuhiko 10y ago
        Windows console shpport is broken on python 3 too on many versions. Click the library can print unicode to the console on windows however. No python 3 needed.
    • takeda 10y ago
      > Citation needed. I'm sorry to say, but your code for example. While I love your click library it has many small issues. Which annoy the heck out of me. I would submit a patch but this is not a simple change and requires a bit of effort. Perhaps I'll find some time to work on it, but I'm afraid it might possibly break compatibility.
      • the_mitsuhiko 10y ago
        Can you talk specifics about what does not work? Because if there is an actual unicode problem in click i want to fix it.
        • takeda 10y ago
          Would have to go over the rest of code, but the code in ClickException is broken: class ClickException(Exception): """An exception that Click can handle and show to the user.""" #: The exit code for this exception exit_code = 1 def __init__(self, message): ctor_msg = message if PY2: if ctor_msg is not None: ctor_msg = ctor_msg.encode('utf-8') Exception.__init__(self, ctor_msg) self.message = message def format_message(self): return self.message def __unicode__(self): return self.message def __str__(self): return self.message.encode('utf-8') def show(self, file=None): if file is None: file = get_text_stderr() echo('Error: %s' % self.format_message(), file=file) This code is arguably wrong. I think I understand what you were trying to do, but I'm not certain. There is a possibility to use it correctly in python 2 though, but many people might be not aware of it. In python 3 you pass the message as is (it might cause another issue, but about that later). In python 2 you immediately encode the passed variable using utf-8. This means that you're expecting argument to be of unicode type, but at the same time you're discouraging users to use unicode_literals, and in most situations users will pass a regular string. In python 3 __str__ is trying to convert text to bytes, while the message would already be text. This most of the times will look correct, but it might spew garbage when there are non ascii characters. Here's corrected code (did not test it though): if PY2: unicode = str class ClickException(Exception): """An exception that Click can handle and show to the user.""" #: The exit code for this exception exit_code = 1 def __init__(self, message): if PY2 and isinstance(self, unicode): message = message.encode() Exception.__init__(self, message) self.message = message def format_message(self): return self.message def __unicode__(self): return str(self).decode() def __str__(self): return str(self.message) def show(self, file=None): if file is None: file = get_text_stderr() echo('Error: %s' % self.format_message(), file=file) There are few other things that gives headaches (not necessarily python 2 only). In python 3 for example click refuses to run if LANG and LC_CTYPE are not defined. Why doesn't it simply do what all other applications are doing and simply fall back to latin-1 (ISO-8859-1) instead of printing the error. Also, another issue (and my above code is also is affected by this). Click is checking for environmental variables before continuing, yet encode and decode have hardcoded utf-8. It probably should use whatever locale.getdefaultlocale() returns.
  • mark-r 10y ago
    Yes, it will explode if you try to write a Unicode string to something that doesn't have UTF-8 encoding by default. But it won't explode if you write byte strings, or if your default encoding is UTF-8.
    • vetinari 10y ago
      Windows console is that case which doesn't default to UTF-8. At the time when I was working with it, py2 defaulted to ascii. For some locales Windows has additional bonus, the Ansi (GUI) encoding is different than OEM (console) and it was heroic undertaking to make your program work correctly for both. Python3 works out of the box.
      • mark-r 10y ago
        I can sympathize, Windows is my environment too. But I'm not blind to other environments where some simplifying assumptions can be made. For true insanity try adding IDLE to the mix. IDLE under Windows in Python 2 will let you output UTF-8, but won't accept it as input. Python 3 is more consistent.
• kstrauser 10y ago
  I had thought you were being sarcastic in your grandparent post.
• int_19h 10y ago
  Python2 supports Unicode, but it requires careful thought and deliberation to do it right - and at the same time makes it very easy to do it wrong (by assuming all bytes are strings etc). As a result, when such code is written by people with little awareness of the world beyond ASCII or Latin-1, it tends to be buggy with locales that need more than that. And yes, it's not really a Python issue, it's a more generic issue with treating bytearrays as strings (what you refer to as POSIX text model). Until UTF-8 became the standard encoding everywhere, there were many Linux apps that couldn't handle non-Latin1 encodings properly, either. It was also the case on Windows in 9x days, for all the same reasons.
• the_why_of_y 10y ago
  Windows doesn't actually support Unicode, not really (for backwards compatibility reasons): it's just UCS-2 at the OS level, and hence you can't represent OS level strings as UTF-8 without data loss, because in practice you need to handle lone UTF-16 surrogate code units. This is why WTF-8 exists. https://simonsapin.github.io/wtf-8/ https://simonsapin.github.io/wtf-8/