3 ms·

A Python 3 string is not an array of "bytes + encoding". A Python 3 string is a sequence of Unicode code points, and you neither know nor are supposed to care w

A Python 3 string is not an array of "bytes + encoding". A Python 3 string is a sequence of Unicode code points, and you neither know nor are supposed to care what the underlying bytes in memory are. All operations on strings in Python 3 are in terms of code points: iterating yields the characters corresponding to the code points, indexing yields the character corresponding to the code point at the index, and length reports the number of code points. If you want bytes + encoding in Python 3, you want the 'bytes' type, not the 'str' type.

lloeki 9y ago
This was a shortcut way of summing things up. Indeed I don't care what it is behind the scenes, it's a string object and that's what matters vs being raw bytes that I can liberally interpret or fix. "A Python 3 string is a sequence of Unicode code points" is just what I mean, those are just bytes in memory that are interpreted as Unicode code points by Python, and it's opaquely exposed to the developer as a string.
- bhaak 9y ago
  But that's quite a different thing. In Python, you only got Unicode strings. In Ruby you got strings that are arrays of bytes that are lazily interpreted as string with the set encoding. In that regard. Ruby strings are more powerful than Python strings as you can handle different encodings. In Python, you have to work around that with the bytes objects to handle non Unicode encodings.
  - lloeki 9y ago
    > Ruby strings are more powerful than Python strings as you can handle different encodings. In Python, you have to work around that with the bytes objects to handle non Unicode encodings. True, but that's another debate entirely. The counterpoint is that Python's stance makes it extremely powerful as a consumer of strings has a lot of guarantees about the string he's receiving, which is great for defensive programming.
  - ubernostrum 9y ago
    As a Python person, I'd say it's not so much about "power" as it is about correctness: equating "sequence of bytes in an encoding" with "string" is the source of a lot of (potentially hard-to-find) bugs and annoyances. So the approach of saying that a string is a sequence of Unicode codepoints, and that there's a separate type to represent sequences of bytes (plus codecs to handle translating between the two types), is a better way to do things. And I think this is borne out by the sheer number of articles that have to be written to remind people who use "strings are bytes in an encoding" languages of just how many assumptions they end up making that turn out to be wrong (most of which boil down to expecting bytes-in-an-encoding types to behave the way Python's string really do behave, as sequences of codepoints rather than as sequences of bytes where there may or may not be one-to-one-correspondence between bytes and codepoints).

lloeki 9y ago

This was a shortcut way of summing things up. Indeed I don't care what it is behind the scenes, it's a string object and that's what matters vs being raw bytes that I can liberally interpret or fix. "A Python 3 string is a sequence of Unicode code points" is just what I mean, those are just bytes in memory that are interpreted as Unicode code points by Python, and it's opaquely exposed to the developer as a string.

bhaak 9y ago
But that's quite a different thing. In Python, you only got Unicode strings. In Ruby you got strings that are arrays of bytes that are lazily interpreted as string with the set encoding. In that regard. Ruby strings are more powerful than Python strings as you can handle different encodings. In Python, you have to work around that with the bytes objects to handle non Unicode encodings.
- lloeki 9y ago
  > Ruby strings are more powerful than Python strings as you can handle different encodings. In Python, you have to work around that with the bytes objects to handle non Unicode encodings. True, but that's another debate entirely. The counterpoint is that Python's stance makes it extremely powerful as a consumer of strings has a lot of guarantees about the string he's receiving, which is great for defensive programming.
- ubernostrum 9y ago
  As a Python person, I'd say it's not so much about "power" as it is about correctness: equating "sequence of bytes in an encoding" with "string" is the source of a lot of (potentially hard-to-find) bugs and annoyances. So the approach of saying that a string is a sequence of Unicode codepoints, and that there's a separate type to represent sequences of bytes (plus codecs to handle translating between the two types), is a better way to do things. And I think this is borne out by the sheer number of articles that have to be written to remind people who use "strings are bytes in an encoding" languages of just how many assumptions they end up making that turn out to be wrong (most of which boil down to expecting bytes-in-an-encoding types to behave the way Python's string really do behave, as sequences of codepoints rather than as sequences of bytes where there may or may not be one-to-one-correspondence between bytes and codepoints).