3 ms·

No, it is really UCS-2: >>> unichr(0x10000) ------------------------------------------------------------ Traceback (most recent call last): File "<ip

by ot 16y ago

No, it is really UCS-2: >>> unichr(0x10000) ------------------------------------------------------------ Traceback (most recent call last): File "<ipython console>", line 1, in <module> ValueError: unichr() arg not in range(0x10000) (narrow Python build) If you want to support codepoints greater than 0x10000 you have to recompile with the option UTF32. I think it must be a constant-lenght encoding to allow s[i] to be constant time.

Locke1689 16y ago
Guido has a different opinion: http://mail.python.org/pipermail/python-dev/2008-July/080895.html http://mail.python.org/pipermail/python-dev/2008-July/080895...
- ot 16y ago
  You are completely right, I'm sorry about my previous comment. The strange thing is that I couldn't find any reference to surrogate pairs in the Python documentation, so I was assuming that the elements of an unicode strings were complete codepoints. Instead this is not the case: >>> list(u'\U00010000') [u'\ud800', u'\udc00'] If I had Python compiled with the UTF32 option, this would return a single element, so Python is leaking an implementation detail that can change across builds. That's really really bad...
  - Locke1689 16y ago
    No, that's the correct behavior. list only incidentally returns a single character in ASCII strings -- it's not required to. You shouldn't be using list on raw unicode strings. u'\U00010000'.encode('utf-8') should produce the same result on every Python version.
    - ot 16y ago
      > You shouldn't be using list on raw unicode strings. Why? I am using list only to show what are the values of s[0] and s[1]. What I am saying is that it returns the list of characters of the underlying representation, so a list of wide chars (possibly surrogate) if compiled with UTF16 or a list of 32bit characters if compiled with UTF16. Are you suggesting that all the string processing (including iteration) should be done on a str encoded in UTF8 instead of using the native unicode type?

by ot 16y ago

Locke1689 16y ago

Guido has a different opinion: http://mail.python.org/pipermail/python-dev/2008-July/080895.html http://mail.python.org/pipermail/python-dev/2008-July/080895...

ot 16y ago
You are completely right, I'm sorry about my previous comment. The strange thing is that I couldn't find any reference to surrogate pairs in the Python documentation, so I was assuming that the elements of an unicode strings were complete codepoints. Instead this is not the case: >>> list(u'\U00010000') [u'\ud800', u'\udc00'] If I had Python compiled with the UTF32 option, this would return a single element, so Python is leaking an implementation detail that can change across builds. That's really really bad...
- Locke1689 16y ago
  No, that's the correct behavior. list only incidentally returns a single character in ASCII strings -- it's not required to. You shouldn't be using list on raw unicode strings. u'\U00010000'.encode('utf-8') should produce the same result on every Python version.
  - ot 16y ago
    > You shouldn't be using list on raw unicode strings. Why? I am using list only to show what are the values of s[0] and s[1]. What I am saying is that it returns the list of characters of the underlying representation, so a list of wide chars (possibly surrogate) if compiled with UTF16 or a list of 32bit characters if compiled with UTF16. Are you suggesting that all the string processing (including iteration) should be done on a str encoded in UTF8 instead of using the native unicode type?