3 ms·

> Python 3 gets so much of this right What does it gets right????? It's all broken as nearly everything else! It's sad 99% comments there are “oh see, I can r

by lelf 13y ago

> Python 3 gets so much of this right What does it gets right????? It's all broken as nearly everything else! It's sad 99% comments there are “oh see, I can run some examples from page just fine. So everything's all right, I've got full Unicode!” The reality is there's 1-2 languages that are trying to make it correct from the beginning (perl6, I'm looking at you). It's 2013 and if language can compose bytes to code points everyone declares a win, sticks "full unicode support" label to it and continues to str[5:9]. ”But I've got UnicodeUtils!” — it won't help. People just don't want or cannot write it correctly. Word is not [a-z]. Not [[:alpha:]] either. And not [insert regex here]. You cannot reverse by reversing codepoint list. And you cannot reverse by reversing grapheme list. And string length is hard to compute and then it doesn't any sense. And indexing string doesn't make any sense and it's far away from O(1)

wbond 13y ago
Can you provide some examples of Python 3 getting strings wrong? Between strings being native unicode code points (you have to encode to bytes to get UTF-8) and unicodedata for normalization and decomposition (http://docs.python.org/3.3/library/unicodedata.html http://docs.python.org/3.3/library/unicodedata.html) I've found Python 3 pretty robust. Python 3.3 also uses appropriate Unicode data for regular expressions, as mentioned on http://docs.python.org/3.3/howto/regex.html http://docs.python.org/3.3/howto/regex.html. If you want to compare strings you should really normalize them first, which is where unicodedata comes in. In my programming situations it would be wrong to conflate different decomposition of the same unicode string. Why is this? Because other software you interact with uses encodings and the UTF-8 encoding of two different decompositions if different. I've run into this with UTF-8 filenames on OS X when working with Subversion.
- lelf 13y ago
  Did you read the comment you're replying to at all? You can start at “It's sad 99% comments”. PS: Python 3.3.2 (default, Nov 27 2013, 20:04:48) [GCC 4.2.1 Compatible Apple LLVM 5.0 (clang-500.2.79)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> 'öo̧'[1:] '̈o̧' And sorry, those new regexes don't even support \X (grapheme matching) Edit: python version
  - wbond 13y ago
    Yes, I did, and you did not provide a single example. You just said "“oh see, I can run some examples from page just fine. So everything's all right, I've got full Unicode!". Taking the time to actually prove your point it useful. However, your recent example seems to be running fine on Python 3.3. You did not include any version info in your example output. Python 3.3.0 (default, Mar 11 2013, 00:32:12) [GCC 4.7.2] on linux Type "help", "copyright", "credits" or "license" for more information. >>> "öo̧"[1:] 'o̧' >>> I haven't run across any situations where Python 3.3 is doing wrong, which is why I am asking for some examples.
    - lelf 13y ago
      3.3.2. No, it is not. Use 'o\u0308o\u0327'
      - wbond 13y ago
        Oh, I see the issue here. You are expecting the string class to function via graphemes rather than characters. It should be possible to implement grapheme support since character support is there, but I imagine the reverse is not true. A little googling turned this up. https://mail.python.org/pipermail/python-ideas/2013-July/021916.html https://mail.python.org/pipermail/python-ideas/2013-July/021...
        tekacs 13y ago
        TL;DR of parent comment here for those skimming: x = 'o\u0308o\u0327' len(x) == 4 x == "öo̧" x[:2] == "ö" x[2:] == "o̧" x[0] == x[2] == "o"
  - wbond 13y ago
    You got me curious about grapheme matching in Python with regex. It looks like it is not in the stdlib yet with 3.3. However, it you install https://pypi.python.org/pypi/regex https://pypi.python.org/pypi/regex and then replace: import re with import regex as re Then if you want to get into using graphemes slicing, you could use something like: import regex as re decomposed_str = 'o\u0308o\u0327' graphemes = re.findall('(\\X)', decomposed_str) sub_graphemes = grapheme[1:] decomposed_substr = ''.join(sub_graphemes)

by lelf 13y ago

wbond 13y ago

Can you provide some examples of Python 3 getting strings wrong? Between strings being native unicode code points (you have to encode to bytes to get UTF-8) and unicodedata for normalization and decomposition (http://docs.python.org/3.3/library/unicodedata.html http://docs.python.org/3.3/library/unicodedata.html) I've found Python 3 pretty robust. Python 3.3 also uses appropriate Unicode data for regular expressions, as mentioned on http://docs.python.org/3.3/howto/regex.html http://docs.python.org/3.3/howto/regex.html. If you want to compare strings you should really normalize them first, which is where unicodedata comes in. In my programming situations it would be wrong to conflate different decomposition of the same unicode string. Why is this? Because other software you interact with uses encodings and the UTF-8 encoding of two different decompositions if different. I've run into this with UTF-8 filenames on OS X when working with Subversion.

lelf 13y ago
Did you read the comment you're replying to at all? You can start at “It's sad 99% comments”. PS: Python 3.3.2 (default, Nov 27 2013, 20:04:48) [GCC 4.2.1 Compatible Apple LLVM 5.0 (clang-500.2.79)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> 'öo̧'[1:] '̈o̧' And sorry, those new regexes don't even support \X (grapheme matching) Edit: python version
- wbond 13y ago
  Yes, I did, and you did not provide a single example. You just said "“oh see, I can run some examples from page just fine. So everything's all right, I've got full Unicode!". Taking the time to actually prove your point it useful. However, your recent example seems to be running fine on Python 3.3. You did not include any version info in your example output. Python 3.3.0 (default, Mar 11 2013, 00:32:12) [GCC 4.7.2] on linux Type "help", "copyright", "credits" or "license" for more information. >>> "öo̧"[1:] 'o̧' >>> I haven't run across any situations where Python 3.3 is doing wrong, which is why I am asking for some examples.
  - lelf 13y ago
    3.3.2. No, it is not. Use 'o\u0308o\u0327'
    - wbond 13y ago
      Oh, I see the issue here. You are expecting the string class to function via graphemes rather than characters. It should be possible to implement grapheme support since character support is there, but I imagine the reverse is not true. A little googling turned this up. https://mail.python.org/pipermail/python-ideas/2013-July/021916.html https://mail.python.org/pipermail/python-ideas/2013-July/021...
      - tekacs 13y ago
        TL;DR of parent comment here for those skimming: x = 'o\u0308o\u0327' len(x) == 4 x == "öo̧" x[:2] == "ö" x[2:] == "o̧" x[0] == x[2] == "o"
- wbond 13y ago
  You got me curious about grapheme matching in Python with regex. It looks like it is not in the stdlib yet with 3.3. However, it you install https://pypi.python.org/pypi/regex https://pypi.python.org/pypi/regex and then replace: import re with import regex as re Then if you want to get into using graphemes slicing, you could use something like: import regex as re decomposed_str = 'o\u0308o\u0327' graphemes = re.findall('(\\X)', decomposed_str) sub_graphemes = grapheme[1:] decomposed_substr = ''.join(sub_graphemes)