3 ms·

See, this is a good example because in Python 2 the string is just bytes. Ł is 0xC5 0x81. Reversing those bytes just corrupts the content. This isn't exclusive

See, this is a good example because in Python 2 the string is just bytes. Ł is 0xC5 0x81. Reversing those bytes just corrupts the content. This isn't exclusive to this one operation. Let me give you another example: >>> print "Hi, my name is Łukasz Langa.".lower() hi, my name is Łukasz langa. Whereas on Python 3: >>> "Hi, my name is Łukasz Langa.".lower() 'hi, my name is łukasz langa.' Or even more basically, Python 2: >>> len("Hi, my name is Łukasz Langa.") 29 Python 3: >>> len("Hi, my name is Łukasz Langa.") 28 Sure, there's hard things about code points and graphemes that Python 3 also doesn't touch, like the example you gave (originally: characters outside of the BMP using surrogate pairs, later: using combining characters), or equality (where normalization might be needed to compare two strings that look alike), or sorting (where collation rules mean the same code points might sort differently depending on the language used). But there's tools to deal with those tricky situations and the default is much more robust. And sure, you can use the `unicode()` type in Python 2 to get the correct results but the problem is that most APIs will work with both types and silently do the wrong thing every now and then. Those include UnicodeDecodeErrors and UnicodeEncodeErrors that you get sometimes. In Python 3, without the magic type promotion, you'll get a sane TypeError every time. Disclaimer: the name used in the example is mine.

pwdisswordfish 10y ago
It's only 'more robust' as long as you don't care about languages in which it isn't. The example is bad, because reversing code points also corrupts the content, but in a different and a somewhat harder to detect way. A better method of reversing text is to split it by grapheme clusters and reverse that. (It may still fail to work when someone manages to sneak a bidi control character in the string, and I am quite confident there may be other issues here that I cannot anticipate right now.) Python 3 offers no advantage here over Python 2. Reversing code points only appears to work, until you feed it stress-marked Cyrillic or Korean with individually encoded jamo; which is quite similar to the charge levelled against Python 2 strings: 'it appears to work, until you feed it non-ASCII input'. The problem isn't solved; the failures are just swept over to language environments in which Western programmers often don't bother to test their programs. Expecting len() to count characters in a string, as you seem to be doing, is also wrong. >>> len('а́') 2 The correct way to count characters is, again, to count grapheme clusters. However, counting characters is not an operation that useful in practice either. If you want to check a string against a database limit, those are usually specified in bytes (or sometimes indeed in code points). If you wish to know how much a piece of text takes on screen, you should ask a font-rendering library to compute it for you; unless you're rendering to a tty, in which case fine, count grapheme clusters. Apart from which syntax gets you codepoint versus byte length, there's no difference here between Python 2 and Python 3 here. Case conversion is also locale-dependent. In Turkish and Azeri, the uppercase form of i is not I, but İ. In Lithuanian, the lowercase form of Ì should be i̇̀, which has both a dot and a grave accent above. str.upper() and str.lower() fail to take that into account, and don't even warn about it in the documentation. Again, it almost works... until you're a Turk and it doesn't. Is it really all that better than Python 2? Manipulating human-readable strings is hard. Operating on code points instead of bytes makes it easier, but doesn't solve it completely. Handwaving about 'tools to deal with those tricky situations' is not an answer; I am quite sure some of those tools exist for Python 2 as well.

pwdisswordfish 10y ago

It's only 'more robust' as long as you don't care about languages in which it isn't. The example is bad, because reversing code points also corrupts the content, but in a different and a somewhat harder to detect way. A better method of reversing text is to split it by grapheme clusters and reverse that. (It may still fail to work when someone manages to sneak a bidi control character in the string, and I am quite confident there may be other issues here that I cannot anticipate right now.) Python 3 offers no advantage here over Python 2. Reversing code points only appears to work, until you feed it stress-marked Cyrillic or Korean with individually encoded jamo; which is quite similar to the charge levelled against Python 2 strings: 'it appears to work, until you feed it non-ASCII input'. The problem isn't solved; the failures are just swept over to language environments in which Western programmers often don't bother to test their programs. Expecting len() to count characters in a string, as you seem to be doing, is also wrong. >>> len('а́') 2 The correct way to count characters is, again, to count grapheme clusters. However, counting characters is not an operation that useful in practice either. If you want to check a string against a database limit, those are usually specified in bytes (or sometimes indeed in code points). If you wish to know how much a piece of text takes on screen, you should ask a font-rendering library to compute it for you; unless you're rendering to a tty, in which case fine, count grapheme clusters. Apart from which syntax gets you codepoint versus byte length, there's no difference here between Python 2 and Python 3 here. Case conversion is also locale-dependent. In Turkish and Azeri, the uppercase form of i is not I, but İ. In Lithuanian, the lowercase form of Ì should be i̇̀, which has both a dot and a grave accent above. str.upper() and str.lower() fail to take that into account, and don't even warn about it in the documentation. Again, it almost works... until you're a Turk and it doesn't. Is it really all that better than Python 2? Manipulating human-readable strings is hard. Operating on code points instead of bytes makes it easier, but doesn't solve it completely. Handwaving about 'tools to deal with those tricky situations' is not an answer; I am quite sure some of those tools exist for Python 2 as well.