4 ms·

Can you provide some examples of Python 3 getting strings wrong? Between strings being native unicode code points (you have to encode to bytes to get UTF-8) an

by wbond 13y ago

Can you provide some examples of Python 3 getting strings wrong? Between strings being native unicode code points (you have to encode to bytes to get UTF-8) and unicodedata for normalization and decomposition (http://docs.python.org/3.3/library/unicodedata.html http://docs.python.org/3.3/library/unicodedata.html) I've found Python 3 pretty robust. Python 3.3 also uses appropriate Unicode data for regular expressions, as mentioned on http://docs.python.org/3.3/howto/regex.html http://docs.python.org/3.3/howto/regex.html. If you want to compare strings you should really normalize them first, which is where unicodedata comes in. In my programming situations it would be wrong to conflate different decomposition of the same unicode string. Why is this? Because other software you interact with uses encodings and the UTF-8 encoding of two different decompositions if different. I've run into this with UTF-8 filenames on OS X when working with Subversion.

lelf 13y ago
Did you read the comment you're replying to at all? You can start at “It's sad 99% comments”. PS: Python 3.3.2 (default, Nov 27 2013, 20:04:48) [GCC 4.2.1 Compatible Apple LLVM 5.0 (clang-500.2.79)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> 'öo̧'[1:] '̈o̧' And sorry, those new regexes don't even support \X (grapheme matching) Edit: python version
- wbond 13y ago
  Yes, I did, and you did not provide a single example. You just said "“oh see, I can run some examples from page just fine. So everything's all right, I've got full Unicode!". Taking the time to actually prove your point it useful. However, your recent example seems to be running fine on Python 3.3. You did not include any version info in your example output. Python 3.3.0 (default, Mar 11 2013, 00:32:12) [GCC 4.7.2] on linux Type "help", "copyright", "credits" or "license" for more information. >>> "öo̧"[1:] 'o̧' >>> I haven't run across any situations where Python 3.3 is doing wrong, which is why I am asking for some examples.
  - lelf 13y ago
    3.3.2. No, it is not. Use 'o\u0308o\u0327'
    - wbond 13y ago
      Oh, I see the issue here. You are expecting the string class to function via graphemes rather than characters. It should be possible to implement grapheme support since character support is there, but I imagine the reverse is not true. A little googling turned this up. https://mail.python.org/pipermail/python-ideas/2013-July/021916.html https://mail.python.org/pipermail/python-ideas/2013-July/021...
      - tekacs 13y ago
        TL;DR of parent comment here for those skimming: x = 'o\u0308o\u0327' len(x) == 4 x == "öo̧" x[:2] == "ö" x[2:] == "o̧" x[0] == x[2] == "o"
- wbond 13y ago
  You got me curious about grapheme matching in Python with regex. It looks like it is not in the stdlib yet with 3.3. However, it you install https://pypi.python.org/pypi/regex https://pypi.python.org/pypi/regex and then replace: import re with import regex as re Then if you want to get into using graphemes slicing, you could use something like: import regex as re decomposed_str = 'o\u0308o\u0327' graphemes = re.findall('(\\X)', decomposed_str) sub_graphemes = grapheme[1:] decomposed_substr = ''.join(sub_graphemes)
- e12e 13y ago
  But what is that sequence (I know the unicode sequence is listed below -- but is it some wierd edge-case)? — because if I manually compose/type those (and a few other characters) everything seems to work fine: [edit: Python 3.2.3] [edit: [GCC 4.7.2] on linux2] >>> 'öo̧'[1:] #copy-paste 'o̧' >>> 'öo̧'[::-1] # "reverse" also breaks '̧oö' #But for Japanese: >>> '日本語'[1:] '本語' >>> '日本語'[:-1] '日本' >>> '日本語'[-1:] '語' >>> '日本語'[::-1] '語本日' # And Norwegian >>> 'æåø'[::-1] 'øåæ' # And a few "French" characters (in this case # manually typed as alt+~+e, etc >>> 'ẽêèe'[::-1] 'eèêẽ' # And crucially for your example, typed as # alt+"+o >>> 'öo'[::-1] 'oö' So is your initial example some kind of unicode-without-bom(b) or something? [edit2: I gather, that working with "pre-composed" characters work, and working with "de-composed" ones break. Which, while expected, is a little sad, I agree.]

by wbond 13y ago

lelf 13y ago

Did you read the comment you're replying to at all? You can start at “It's sad 99% comments”. PS: Python 3.3.2 (default, Nov 27 2013, 20:04:48) [GCC 4.2.1 Compatible Apple LLVM 5.0 (clang-500.2.79)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> 'öo̧'[1:] '̈o̧' And sorry, those new regexes don't even support \X (grapheme matching) Edit: python version

wbond 13y ago
Yes, I did, and you did not provide a single example. You just said "“oh see, I can run some examples from page just fine. So everything's all right, I've got full Unicode!". Taking the time to actually prove your point it useful. However, your recent example seems to be running fine on Python 3.3. You did not include any version info in your example output. Python 3.3.0 (default, Mar 11 2013, 00:32:12) [GCC 4.7.2] on linux Type "help", "copyright", "credits" or "license" for more information. >>> "öo̧"[1:] 'o̧' >>> I haven't run across any situations where Python 3.3 is doing wrong, which is why I am asking for some examples.
- lelf 13y ago
  3.3.2. No, it is not. Use 'o\u0308o\u0327'
  - wbond 13y ago
    Oh, I see the issue here. You are expecting the string class to function via graphemes rather than characters. It should be possible to implement grapheme support since character support is there, but I imagine the reverse is not true. A little googling turned this up. https://mail.python.org/pipermail/python-ideas/2013-July/021916.html https://mail.python.org/pipermail/python-ideas/2013-July/021...
    - tekacs 13y ago
      TL;DR of parent comment here for those skimming: x = 'o\u0308o\u0327' len(x) == 4 x == "öo̧" x[:2] == "ö" x[2:] == "o̧" x[0] == x[2] == "o"
wbond 13y ago
You got me curious about grapheme matching in Python with regex. It looks like it is not in the stdlib yet with 3.3. However, it you install https://pypi.python.org/pypi/regex https://pypi.python.org/pypi/regex and then replace: import re with import regex as re Then if you want to get into using graphemes slicing, you could use something like: import regex as re decomposed_str = 'o\u0308o\u0327' graphemes = re.findall('(\\X)', decomposed_str) sub_graphemes = grapheme[1:] decomposed_substr = ''.join(sub_graphemes)
e12e 13y ago
But what is that sequence (I know the unicode sequence is listed below -- but is it some wierd edge-case)? — because if I manually compose/type those (and a few other characters) everything seems to work fine: [edit: Python 3.2.3] [edit: [GCC 4.7.2] on linux2] >>> 'öo̧'[1:] #copy-paste 'o̧' >>> 'öo̧'[::-1] # "reverse" also breaks '̧oö' #But for Japanese: >>> '日本語'[1:] '本語' >>> '日本語'[:-1] '日本' >>> '日本語'[-1:] '語' >>> '日本語'[::-1] '語本日' # And Norwegian >>> 'æåø'[::-1] 'øåæ' # And a few "French" characters (in this case # manually typed as alt+~+e, etc >>> 'ẽêèe'[::-1] 'eèêẽ' # And crucially for your example, typed as # alt+"+o >>> 'öo'[::-1] 'oö' So is your initial example some kind of unicode-without-bom(b) or something? [edit2: I gather, that working with "pre-composed" characters work, and working with "de-composed" ones break. Which, while expected, is a little sad, I agree.]