8 ms·

# -- coding: utf8 -- print "Hi, my name is Łukasz Langa." print "Hi, my name is Łukasz Langa."[::-1] This is a bad example. Reversing Unicode strings

# -*- coding: utf8 -*- print "Hi, my name is Łukasz Langa." print "Hi, my name is Łukasz Langa."[::-1] This is a bad example. Reversing Unicode strings isn't an operation that is often needed in practice, and it shouldn't be done codepoint-wise anyway, because some characters have to be represented with multiple codepoints: Python 3.5.2 (default, Nov 7 2016, 11:31:36) [GCC 6.2.1 20160830] on linux Type "help", "copyright", "credits" or "license" for more information. >>> 'x́q'[::-1] 'q́x' Doing it properly is bit of a grapheme clusterfuck: <http://unicode.org/reports/tr29/#Grapheme_Cluster_Boundaries> http://unicode.org/reports/tr29/#Grapheme_Cluster_Boundaries....

jventura 10y ago
A little bit offtopic, but I'm not having the same result as you do.. Python 3.5.2 (default, Nov 9 2016, 20:19:22) [GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.42.1)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> 'ǵp'[::-1] 'pǵ'
- pwdisswordfish 10y ago
  Ugh, I should have double-checked that neither glyph has a pre-composed code point. I switched the example to 'x́q'. (Writing it as 'g\u0301p' would also work.)
ambivalence 10y ago
See, this is a good example because in Python 2 the string is just bytes. Ł is 0xC5 0x81. Reversing those bytes just corrupts the content. This isn't exclusive to this one operation. Let me give you another example: >>> print "Hi, my name is Łukasz Langa.".lower() hi, my name is Łukasz langa. Whereas on Python 3: >>> "Hi, my name is Łukasz Langa.".lower() 'hi, my name is łukasz langa.' Or even more basically, Python 2: >>> len("Hi, my name is Łukasz Langa.") 29 Python 3: >>> len("Hi, my name is Łukasz Langa.") 28 Sure, there's hard things about code points and graphemes that Python 3 also doesn't touch, like the example you gave (originally: characters outside of the BMP using surrogate pairs, later: using combining characters), or equality (where normalization might be needed to compare two strings that look alike), or sorting (where collation rules mean the same code points might sort differently depending on the language used). But there's tools to deal with those tricky situations and the default is much more robust. And sure, you can use the `unicode()` type in Python 2 to get the correct results but the problem is that most APIs will work with both types and silently do the wrong thing every now and then. Those include UnicodeDecodeErrors and UnicodeEncodeErrors that you get sometimes. In Python 3, without the magic type promotion, you'll get a sane TypeError every time. Disclaimer: the name used in the example is mine.
- pwdisswordfish 10y ago
  It's only 'more robust' as long as you don't care about languages in which it isn't. The example is bad, because reversing code points also corrupts the content, but in a different and a somewhat harder to detect way. A better method of reversing text is to split it by grapheme clusters and reverse that. (It may still fail to work when someone manages to sneak a bidi control character in the string, and I am quite confident there may be other issues here that I cannot anticipate right now.) Python 3 offers no advantage here over Python 2. Reversing code points only appears to work, until you feed it stress-marked Cyrillic or Korean with individually encoded jamo; which is quite similar to the charge levelled against Python 2 strings: 'it appears to work, until you feed it non-ASCII input'. The problem isn't solved; the failures are just swept over to language environments in which Western programmers often don't bother to test their programs. Expecting len() to count characters in a string, as you seem to be doing, is also wrong. >>> len('а́') 2 The correct way to count characters is, again, to count grapheme clusters. However, counting characters is not an operation that useful in practice either. If you want to check a string against a database limit, those are usually specified in bytes (or sometimes indeed in code points). If you wish to know how much a piece of text takes on screen, you should ask a font-rendering library to compute it for you; unless you're rendering to a tty, in which case fine, count grapheme clusters. Apart from which syntax gets you codepoint versus byte length, there's no difference here between Python 2 and Python 3 here. Case conversion is also locale-dependent. In Turkish and Azeri, the uppercase form of i is not I, but İ. In Lithuanian, the lowercase form of Ì should be i̇̀, which has both a dot and a grave accent above. str.upper() and str.lower() fail to take that into account, and don't even warn about it in the documentation. Again, it almost works... until you're a Turk and it doesn't. Is it really all that better than Python 2? Manipulating human-readable strings is hard. Operating on code points instead of bytes makes it easier, but doesn't solve it completely. Handwaving about 'tools to deal with those tricky situations' is not an answer; I am quite sure some of those tools exist for Python 2 as well.
ubernostrum 10y ago
I've written a bit about this in context of companies which use a palindrome detector as a phone-screen question. My current implementation does normalization and uses Unicode character categories to strip out characters which should be ignored, but also comes with a free half-hour lecture about why A) this is a hard problem, B) no programming language I know can easily solve it, and C) this is a bad phone-screen question and indicative of bad process at the company.
- ubernostrum 10y ago
  And I wrote this, and then decided to poke at it again, and discovered: https://pypi.python.org/pypi/regex https://pypi.python.org/pypi/regex Which implements the '\X' grapheme-matching operator, and does in fact allow Python to identify palindromes in Unicode (first normalize and strip out any categories of characters to ignore, then findall(r'\X') returns a list of graphemes in the input, which can be compared to itself in reverse to determine palindromicity).
kosma 10y ago
A substring operation (say, name[:20]) is much more common and fails in similar ways.

# -*- coding: utf8 -*- print "Hi, my name is Łukasz Langa." print "Hi, my name is Łukasz Langa."[::-1] This is a bad example. Reversing Unicode strings

jventura 10y ago

A little bit offtopic, but I'm not having the same result as you do.. Python 3.5.2 (default, Nov 9 2016, 20:19:22) [GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.42.1)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> 'ǵp'[::-1] 'pǵ'

pwdisswordfish 10y ago
Ugh, I should have double-checked that neither glyph has a pre-composed code point. I switched the example to 'x́q'. (Writing it as 'g\u0301p' would also work.)

ambivalence 10y ago

See, this is a good example because in Python 2 the string is just bytes. Ł is 0xC5 0x81. Reversing those bytes just corrupts the content. This isn't exclusive to this one operation. Let me give you another example: >>> print "Hi, my name is Łukasz Langa.".lower() hi, my name is Łukasz langa. Whereas on Python 3: >>> "Hi, my name is Łukasz Langa.".lower() 'hi, my name is łukasz langa.' Or even more basically, Python 2: >>> len("Hi, my name is Łukasz Langa.") 29 Python 3: >>> len("Hi, my name is Łukasz Langa.") 28 Sure, there's hard things about code points and graphemes that Python 3 also doesn't touch, like the example you gave (originally: characters outside of the BMP using surrogate pairs, later: using combining characters), or equality (where normalization might be needed to compare two strings that look alike), or sorting (where collation rules mean the same code points might sort differently depending on the language used). But there's tools to deal with those tricky situations and the default is much more robust. And sure, you can use the `unicode()` type in Python 2 to get the correct results but the problem is that most APIs will work with both types and silently do the wrong thing every now and then. Those include UnicodeDecodeErrors and UnicodeEncodeErrors that you get sometimes. In Python 3, without the magic type promotion, you'll get a sane TypeError every time. Disclaimer: the name used in the example is mine.

pwdisswordfish 10y ago
It's only 'more robust' as long as you don't care about languages in which it isn't. The example is bad, because reversing code points also corrupts the content, but in a different and a somewhat harder to detect way. A better method of reversing text is to split it by grapheme clusters and reverse that. (It may still fail to work when someone manages to sneak a bidi control character in the string, and I am quite confident there may be other issues here that I cannot anticipate right now.) Python 3 offers no advantage here over Python 2. Reversing code points only appears to work, until you feed it stress-marked Cyrillic or Korean with individually encoded jamo; which is quite similar to the charge levelled against Python 2 strings: 'it appears to work, until you feed it non-ASCII input'. The problem isn't solved; the failures are just swept over to language environments in which Western programmers often don't bother to test their programs. Expecting len() to count characters in a string, as you seem to be doing, is also wrong. >>> len('а́') 2 The correct way to count characters is, again, to count grapheme clusters. However, counting characters is not an operation that useful in practice either. If you want to check a string against a database limit, those are usually specified in bytes (or sometimes indeed in code points). If you wish to know how much a piece of text takes on screen, you should ask a font-rendering library to compute it for you; unless you're rendering to a tty, in which case fine, count grapheme clusters. Apart from which syntax gets you codepoint versus byte length, there's no difference here between Python 2 and Python 3 here. Case conversion is also locale-dependent. In Turkish and Azeri, the uppercase form of i is not I, but İ. In Lithuanian, the lowercase form of Ì should be i̇̀, which has both a dot and a grave accent above. str.upper() and str.lower() fail to take that into account, and don't even warn about it in the documentation. Again, it almost works... until you're a Turk and it doesn't. Is it really all that better than Python 2? Manipulating human-readable strings is hard. Operating on code points instead of bytes makes it easier, but doesn't solve it completely. Handwaving about 'tools to deal with those tricky situations' is not an answer; I am quite sure some of those tools exist for Python 2 as well.

ubernostrum 10y ago

I've written a bit about this in context of companies which use a palindrome detector as a phone-screen question. My current implementation does normalization and uses Unicode character categories to strip out characters which should be ignored, but also comes with a free half-hour lecture about why A) this is a hard problem, B) no programming language I know can easily solve it, and C) this is a bad phone-screen question and indicative of bad process at the company.

ubernostrum 10y ago
And I wrote this, and then decided to poke at it again, and discovered: https://pypi.python.org/pypi/regex https://pypi.python.org/pypi/regex Which implements the '\X' grapheme-matching operator, and does in fact allow Python to identify palindromes in Unicode (first normalize and strip out any categories of characters to ignore, then findall(r'\X') returns a list of graphemes in the input, which can be compared to itself in reverse to determine palindromicity).

kosma 10y ago

A substring operation (say, name[:20]) is much more common and fails in similar ways.