7 ms·

Python 3 gets so much of this right. It's one of the things I really loved about python 3 as it allows for correct string handling in most cases (see below). N

by pilif 13y ago

Python 3 gets so much of this right. It's one of the things I really loved about python 3 as it allows for correct string handling in most cases (see below). Note that this is only really true with Python 3.3 and later as in earlier versions stuff would start breaking for characters outside of the BMP (which is where JS is still stuck at, btw) unless you had a wide build which was using a lot of memory for strings (4 bytes per character) In general, internally using unicode and converting to and from bytes when doing i/o is the right way to go. But: Due to http://en.wikipedia.org/wiki/Han_unification http://en.wikipedia.org/wiki/Han_unification being locked into Unicode with a language might not be feasible for everybody - especially in Asian regions, Unicode isn't yet as widely spread and you still need to deal with regional encodings, mainly because even with the huge character set of Unicode, we still can't reliably write in every language. Ruby 1.9 and later helps here by having many, many string types (as many as it knows encodings), which can't be assigned to each other without conversion. This allows you to still have an internal character set for your application and doing encoding/decoding at i/o time, but you're not stuck with unicode if that's not feasible for your use-case. People hate this though because it seems to interfere with their otherwise perfectly fine workflow ("why can't I assign this "string" I got from a user to this string variable here??"), but it's actually preventing data corruption (once strings of multiple encodings are mixed up, it's often impossible to un-mix them, if they have the same characer width). I don't know how good the library support for the various Unicode encodings is in Ruby though. According to the article, there still is trouble with correctly doing case transformations and reversing them. Which brings me to another point: Some of the stuff you do with strings isn't just dependent on string encoding, but also locale. Uppercasing rules for example depend on locale, so you need to keep that into account too. And, of course, deal with cases when you don't know the locale the string was in (encoding is hard enough and most of the cases undetectable - but locales - next to impossible). I laugh at people who constantly tell me that this isn't hard and that "it's just strings".

exDM69 13y ago
> Python 3 gets so much of this right. It's one of the things I really loved about python 3 as it allows for correct string handling in most cases (see below). One of the biggest things that I feel Python gets right with the string type is that strings are immutable. It makes a lot of things easier. It really makes sense to have a good string type for small strings, stored in unicode. Immutability makes everything simpler. The string type is not a good fit for handling large amounts of text. There are trade offs for efficiency that have to be made to create a handy string type. It really makes sense to have a separate "bytes" type or some kind of StringBuffer for doing big text operations.
- wsc981 13y ago
  Isn't the string type immutable in many (most?) other languages as well? In Objective-C the default is an immutable string (though optionally one can create mutable strings as well). Lua also uses immutable strings. In Java and C# I think the situation is the same, since if you want to use high performance string manipulation, you'll generally resort some form of StringBuilder helper class.
  - meepmorp 13y ago
    I believe strings are mutable in Ruby.
    - reflectiv 13y ago
      They are... s = "hello" s << " world" s # hello world
      - dbaupp 13y ago
        Is that allocating a new buffer, leaving the "hello" string to be collected by the GC?
        tedunangst 13y ago
        No, it expands the existing buffer. (leaving " world" to be collected). Note that the following is different and more like what you're thinking. a = a + " world"
        jeorgun 13y ago
        No, it's operating in place: def append_world(str) str << " world" end a = "hello" append_world(a) a #=> "hello world"
  - SideburnsOfDoom 13y ago
    Correct, C# and .Net have an immutable string class and a mutable StringBuilder helper class.
fauigerzigerk 13y ago
>In general, internally using unicode and converting to and from bytes when doing i/o is the right way to go. I'm not sure what "internally using unicode" means. Pyhon's internal representation of strings has changed a lot. It hasn't even been stable in Python 3. Now they are apparently using an internal representation that varies depending on the "widest" character stored. The only solution that isn't driving me insane is to use UTF-8 everywhere. The Python 3 unicode situation is actually the main reason why I'm not using Python much these days.
- pilif 13y ago
  In Python 3, you don't care about what they use internally. You don't need to. If you want to work with strings, you work with strings. If you want to work with bytes, you work with bytes. If you want to convert bytes into strings (maybe because it's user input that you want to work with), then you tell Python what encoding these bytes are in and you have it create a string for you. You don't care what Python uses internally, because their string API is correct and correctly works on characters. That noël example of the original article consists of 4 characters in Python 3 which is exactly what you want. I know that just using UTF-8 everywhere would be cool, but that's not how the world works for various reasons. One is that UTF-8 is a variable length encoding which has some performance issues for some operations (like getting the length of the string. Or finding the n-th character). UTF-8 also isn't widely used by current operating systems (Mac OS and Windows use UCS-2). It's also not what's used by way too many legacy systems still around. So as long as the data you work with likely isn't in UTF-8, the encoding and decoding steps will be needed if you want to be correct. Otherwise, you risk mixing strings in different encodings together which is an irrecoverable error (aside of using heuristics based on the language of the content).
  - panzi 13y ago
    Agreed with most of it except: "because their string API is correct" Apparently they have a bug in their UTF-7 parser that can lead to invalid unicode strings. Don't know if it's already fixed.
    - 13y ago
mathias 13y ago
> stuff would start breaking for characters outside of the BMP (which is where JS is still stuck at, btw) ECMAScript 6 fixes that, mostly. See http://mathiasbynens.be/notes/javascript-unicode http://mathiasbynens.be/notes/javascript-unicode for details.
lelf 13y ago
> Python 3 gets so much of this right What does it gets right????? It's all broken as nearly everything else! It's sad 99% comments there are “oh see, I can run some examples from page just fine. So everything's all right, I've got full Unicode!” The reality is there's 1-2 languages that are trying to make it correct from the beginning (perl6, I'm looking at you). It's 2013 and if language can compose bytes to code points everyone declares a win, sticks "full unicode support" label to it and continues to str[5:9]. ”But I've got UnicodeUtils!” — it won't help. People just don't want or cannot write it correctly. Word is not [a-z]. Not [[:alpha:]] either. And not [insert regex here]. You cannot reverse by reversing codepoint list. And you cannot reverse by reversing grapheme list. And string length is hard to compute and then it doesn't any sense. And indexing string doesn't make any sense and it's far away from O(1)
- wbond 13y ago
  Can you provide some examples of Python 3 getting strings wrong? Between strings being native unicode code points (you have to encode to bytes to get UTF-8) and unicodedata for normalization and decomposition (http://docs.python.org/3.3/library/unicodedata.html http://docs.python.org/3.3/library/unicodedata.html) I've found Python 3 pretty robust. Python 3.3 also uses appropriate Unicode data for regular expressions, as mentioned on http://docs.python.org/3.3/howto/regex.html http://docs.python.org/3.3/howto/regex.html. If you want to compare strings you should really normalize them first, which is where unicodedata comes in. In my programming situations it would be wrong to conflate different decomposition of the same unicode string. Why is this? Because other software you interact with uses encodings and the UTF-8 encoding of two different decompositions if different. I've run into this with UTF-8 filenames on OS X when working with Subversion.
  - lelf 13y ago
    Did you read the comment you're replying to at all? You can start at “It's sad 99% comments”. PS: Python 3.3.2 (default, Nov 27 2013, 20:04:48) [GCC 4.2.1 Compatible Apple LLVM 5.0 (clang-500.2.79)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> 'öo̧'[1:] '̈o̧' And sorry, those new regexes don't even support \X (grapheme matching) Edit: python version

by pilif 13y ago

exDM69 13y ago

> Python 3 gets so much of this right. It's one of the things I really loved about python 3 as it allows for correct string handling in most cases (see below). One of the biggest things that I feel Python gets right with the string type is that strings are immutable. It makes a lot of things easier. It really makes sense to have a good string type for small strings, stored in unicode. Immutability makes everything simpler. The string type is not a good fit for handling large amounts of text. There are trade offs for efficiency that have to be made to create a handy string type. It really makes sense to have a separate "bytes" type or some kind of StringBuffer for doing big text operations.

wsc981 13y ago
Isn't the string type immutable in many (most?) other languages as well? In Objective-C the default is an immutable string (though optionally one can create mutable strings as well). Lua also uses immutable strings. In Java and C# I think the situation is the same, since if you want to use high performance string manipulation, you'll generally resort some form of StringBuilder helper class.
- meepmorp 13y ago
  I believe strings are mutable in Ruby.
  - reflectiv 13y ago
    They are... s = "hello" s << " world" s # hello world
    - dbaupp 13y ago
      Is that allocating a new buffer, leaving the "hello" string to be collected by the GC?
      - tedunangst 13y ago
        No, it expands the existing buffer. (leaving " world" to be collected). Note that the following is different and more like what you're thinking. a = a + " world"
      - jeorgun 13y ago
        No, it's operating in place: def append_world(str) str << " world" end a = "hello" append_world(a) a #=> "hello world"
- SideburnsOfDoom 13y ago
  Correct, C# and .Net have an immutable string class and a mutable StringBuilder helper class.

fauigerzigerk 13y ago

>In general, internally using unicode and converting to and from bytes when doing i/o is the right way to go. I'm not sure what "internally using unicode" means. Pyhon's internal representation of strings has changed a lot. It hasn't even been stable in Python 3. Now they are apparently using an internal representation that varies depending on the "widest" character stored. The only solution that isn't driving me insane is to use UTF-8 everywhere. The Python 3 unicode situation is actually the main reason why I'm not using Python much these days.

pilif 13y ago
In Python 3, you don't care about what they use internally. You don't need to. If you want to work with strings, you work with strings. If you want to work with bytes, you work with bytes. If you want to convert bytes into strings (maybe because it's user input that you want to work with), then you tell Python what encoding these bytes are in and you have it create a string for you. You don't care what Python uses internally, because their string API is correct and correctly works on characters. That noël example of the original article consists of 4 characters in Python 3 which is exactly what you want. I know that just using UTF-8 everywhere would be cool, but that's not how the world works for various reasons. One is that UTF-8 is a variable length encoding which has some performance issues for some operations (like getting the length of the string. Or finding the n-th character). UTF-8 also isn't widely used by current operating systems (Mac OS and Windows use UCS-2). It's also not what's used by way too many legacy systems still around. So as long as the data you work with likely isn't in UTF-8, the encoding and decoding steps will be needed if you want to be correct. Otherwise, you risk mixing strings in different encodings together which is an irrecoverable error (aside of using heuristics based on the language of the content).
- panzi 13y ago
  Agreed with most of it except: "because their string API is correct" Apparently they have a bug in their UTF-7 parser that can lead to invalid unicode strings. Don't know if it's already fixed.
  - 13y ago

mathias 13y ago

> stuff would start breaking for characters outside of the BMP (which is where JS is still stuck at, btw) ECMAScript 6 fixes that, mostly. See http://mathiasbynens.be/notes/javascript-unicode http://mathiasbynens.be/notes/javascript-unicode for details.

lelf 13y ago

> Python 3 gets so much of this right What does it gets right????? It's all broken as nearly everything else! It's sad 99% comments there are “oh see, I can run some examples from page just fine. So everything's all right, I've got full Unicode!” The reality is there's 1-2 languages that are trying to make it correct from the beginning (perl6, I'm looking at you). It's 2013 and if language can compose bytes to code points everyone declares a win, sticks "full unicode support" label to it and continues to str[5:9]. ”But I've got UnicodeUtils!” — it won't help. People just don't want or cannot write it correctly. Word is not [a-z]. Not [[:alpha:]] either. And not [insert regex here]. You cannot reverse by reversing codepoint list. And you cannot reverse by reversing grapheme list. And string length is hard to compute and then it doesn't any sense. And indexing string doesn't make any sense and it's far away from O(1)

wbond 13y ago
Can you provide some examples of Python 3 getting strings wrong? Between strings being native unicode code points (you have to encode to bytes to get UTF-8) and unicodedata for normalization and decomposition (http://docs.python.org/3.3/library/unicodedata.html http://docs.python.org/3.3/library/unicodedata.html) I've found Python 3 pretty robust. Python 3.3 also uses appropriate Unicode data for regular expressions, as mentioned on http://docs.python.org/3.3/howto/regex.html http://docs.python.org/3.3/howto/regex.html. If you want to compare strings you should really normalize them first, which is where unicodedata comes in. In my programming situations it would be wrong to conflate different decomposition of the same unicode string. Why is this? Because other software you interact with uses encodings and the UTF-8 encoding of two different decompositions if different. I've run into this with UTF-8 filenames on OS X when working with Subversion.
- lelf 13y ago
  Did you read the comment you're replying to at all? You can start at “It's sad 99% comments”. PS: Python 3.3.2 (default, Nov 27 2013, 20:04:48) [GCC 4.2.1 Compatible Apple LLVM 5.0 (clang-500.2.79)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> 'öo̧'[1:] '̈o̧' And sorry, those new regexes don't even support \X (grapheme matching) Edit: python version