3 ms·

Nice littel overview, but... "Unicode sucks. This is a universal truth." And here I stopped reading. Unicode is the best thing around, what sucks it that Py2

by yaix 14y ago

Nice littel overview, but... "Unicode sucks. This is a universal truth." And here I stopped reading. Unicode is the best thing around, what sucks it that Py2 does not implement it very well. Py3 does, and that is the main reason I really would like to use it and hope Django will soon fully support it. (Because: there are other languages beside the European ones)

eevee 14y ago
Fair point. Correction: working with encodings sucks. Py3 isn't magnitudes better with Unicode; it's better about converting incoming bytes, and it makes literals into real strings. But the former isn't necessarily a good thing, and you can get the latter with a `__future__` import. Always dying on implicit conversions is definitely nice, though.
- adeelk 14y ago
  Working with encodings is indeed annoying in Python, but it’s not a “universal” problem.
  - anuraj 14y ago
    There is a language called 'Java' where every string is in unicode, and I do not remember ever running into an issue!
    - duskwuff 14y ago
      Except that Java strings aren't in Unicode. They're in UTF-16, which is the worst-of-all-worlds encoding. (It's big and heavy, and it still has multibyte sequences. They're just rare enough that you're likely to forget about them during testing.)
      - 0x09 14y ago
        UTF-16 is most certainly Unicode, even if it isn't the preferred flavor.
        anuraj 14y ago
        "UCS-2 (2-byte Universal Character Set) is a character encoding that was superseded by UTF-16 in version 2.0 of the Unicode standard in July 1996". Java adopted UCS-2 which was later supplemented with UTF-16 support. There are at least a few languages in this world which do not use roman letters and are better represented as multibyte sequences :). That is why Java added support for supplementary UTF-16 characters over and above UCS-2. That said, use of UTF-8 would have been optimal for western languages, but sub optimal for several other languages.
        slowpoke 14y ago
        Wrong. UTF-16 is an encoding, Unicode is the abstract representation. You can encode Unicode strings into UTF-16, but that doesn't make UTF-16 Unicode. Python's Unicode strings are actually really just Unicode, that's why you can't write them to a file or anything - you need to encode them first (which defaults to UTF-8).
        kami8845 14y ago
        Unfortunately it only defaults to utf-8 in Python 3
        duskwuff 14y ago
        In Java, you end up seeing the guts of UTF-16 far more often than you should. Most notably, the String APIs often index strings by UTF-16 code units, not characters, so string "lengths" don't always correspond to the number of Unicode characters, and you can end up cutting surrogate pairs in half if you aren't careful.
- jeltz 14y ago
  Working with encoding is not that bad in for example Ruby 1.9. What makes them horrible to work with are the bad design decisions in Python, especially Python 2.
  - eevee 14y ago
    I've only read about Ruby strings, not used them, but they seem like a middle ground between Perl's approach and Python's approach. What do you perceive as the problem with Python?

by yaix 14y ago

eevee 14y ago

Fair point. Correction: working with encodings sucks. Py3 isn't magnitudes better with Unicode; it's better about converting incoming bytes, and it makes literals into real strings. But the former isn't necessarily a good thing, and you can get the latter with a `__future__` import. Always dying on implicit conversions is definitely nice, though.

adeelk 14y ago
Working with encodings is indeed annoying in Python, but it’s not a “universal” problem.
- anuraj 14y ago
  There is a language called 'Java' where every string is in unicode, and I do not remember ever running into an issue!
  - duskwuff 14y ago
    Except that Java strings aren't in Unicode. They're in UTF-16, which is the worst-of-all-worlds encoding. (It's big and heavy, and it still has multibyte sequences. They're just rare enough that you're likely to forget about them during testing.)
    - 0x09 14y ago
      UTF-16 is most certainly Unicode, even if it isn't the preferred flavor.
      - anuraj 14y ago
        "UCS-2 (2-byte Universal Character Set) is a character encoding that was superseded by UTF-16 in version 2.0 of the Unicode standard in July 1996". Java adopted UCS-2 which was later supplemented with UTF-16 support. There are at least a few languages in this world which do not use roman letters and are better represented as multibyte sequences :). That is why Java added support for supplementary UTF-16 characters over and above UCS-2. That said, use of UTF-8 would have been optimal for western languages, but sub optimal for several other languages.
      - slowpoke 14y ago
        Wrong. UTF-16 is an encoding, Unicode is the abstract representation. You can encode Unicode strings into UTF-16, but that doesn't make UTF-16 Unicode. Python's Unicode strings are actually really just Unicode, that's why you can't write them to a file or anything - you need to encode them first (which defaults to UTF-8).
        kami8845 14y ago
        Unfortunately it only defaults to utf-8 in Python 3
      - duskwuff 14y ago
        In Java, you end up seeing the guts of UTF-16 far more often than you should. Most notably, the String APIs often index strings by UTF-16 code units, not characters, so string "lengths" don't always correspond to the number of Unicode characters, and you can end up cutting surrogate pairs in half if you aren't careful.
jeltz 14y ago
Working with encoding is not that bad in for example Ruby 1.9. What makes them horrible to work with are the bad design decisions in Python, especially Python 2.
- eevee 14y ago
  I've only read about Ruby strings, not used them, but they seem like a middle ground between Perl's approach and Python's approach. What do you perceive as the problem with Python?