3 ms·

Python supports full Unicode on all platforms now (x). Who cares about in-memory representation as long as the user doesn't have to suffer from surrogates and a

by zurn 14y ago

Python supports full Unicode on all platforms now (x). Who cares about in-memory representation as long as the user doesn't have to suffer from surrogates and all that horror. (x) pre-PEP393 there was a build option to use the limited 16-bit Unicode, was unfortunately popular on Windows

asveikau 14y ago
Hm. I picked Python because I knew it was an outlier, but I don't know all the details. Does Python still suffer from the horrors of comparison between diacritics made up of combining characters and the same glyph as a pre-composed character? Seems like even if you expose strings as UTF-32 you'd still have that issue.
- zurn 14y ago
  Afaik the combining character problem hasn't changed, you still have to use unicodedata.normalize() for that. But at least you can pass through Unicode strings cleanly. Are there other languages that handle this better?
  - grey-area 14y ago
    The same problem occurs in Ruby, at least when dealing with the Mac OS X file system (which uses decomposed strings internally), you may receive NFC or NFD and don't know which you're getting.
  - asveikau 14y ago
    It's not really clear to me what "handle better" would be, which is why I asked if Python did anything, as your post suggested it hides some ugliness of Unicode (a claim like this I am always skeptical of, although reading PEP393 it seems kind of clever in that "why don't more libraries do this" kind of way). I suppose one form of "handle better" would be to always normalize all the time, but then, there are those times when you want payload to be bit-identical after a round-trip. I suspect it's just rooted in Unicode sucking (even if it sucks less than the encodings it replaced); it's funny how Unicode is supposed to make things simpler, but carries its own baggage and complexity and nonsense to worry about. Every time I read about these details in Unicode it seems to suck more than I remembered the last time.
    - plorkyeran 14y ago
      Other than the great UCS-2 cockup most of the issues with Unicode are either due to that human languages are a pain or are unpleasant backwards compatiblity things. Only having the fully normalized forms would make things designed around Unicode a lot simpler, but the denormalized forms let things not designed for Unicode mostly work, which was key for getting people to actually use Unicode.
      - asveikau 14y ago
        Oh, I totally understand the reasons. Doesn't mean it still doesn't suck. It's one of those "sucks less" type of scenarios.

by zurn 14y ago

asveikau 14y ago

Hm. I picked Python because I knew it was an outlier, but I don't know all the details. Does Python still suffer from the horrors of comparison between diacritics made up of combining characters and the same glyph as a pre-composed character? Seems like even if you expose strings as UTF-32 you'd still have that issue.

zurn 14y ago
Afaik the combining character problem hasn't changed, you still have to use unicodedata.normalize() for that. But at least you can pass through Unicode strings cleanly. Are there other languages that handle this better?
- grey-area 14y ago
  The same problem occurs in Ruby, at least when dealing with the Mac OS X file system (which uses decomposed strings internally), you may receive NFC or NFD and don't know which you're getting.
- asveikau 14y ago
  It's not really clear to me what "handle better" would be, which is why I asked if Python did anything, as your post suggested it hides some ugliness of Unicode (a claim like this I am always skeptical of, although reading PEP393 it seems kind of clever in that "why don't more libraries do this" kind of way). I suppose one form of "handle better" would be to always normalize all the time, but then, there are those times when you want payload to be bit-identical after a round-trip. I suspect it's just rooted in Unicode sucking (even if it sucks less than the encodings it replaced); it's funny how Unicode is supposed to make things simpler, but carries its own baggage and complexity and nonsense to worry about. Every time I read about these details in Unicode it seems to suck more than I remembered the last time.
  - plorkyeran 14y ago
    Other than the great UCS-2 cockup most of the issues with Unicode are either due to that human languages are a pain or are unpleasant backwards compatiblity things. Only having the fully normalized forms would make things designed around Unicode a lot simpler, but the denormalized forms let things not designed for Unicode mostly work, which was key for getting people to actually use Unicode.
    - asveikau 14y ago
      Oh, I totally understand the reasons. Doesn't mean it still doesn't suck. It's one of those "sucks less" type of scenarios.