8 ms·

Hm, I never saw this as ambiguous at all, except for a few weird encodings that Python has as "convenience" method. Here's how you remember it: "Unicode" is no

Hm, I never saw this as ambiguous at all, except for a few weird encodings that Python has as "convenience" method. Here's how you remember it: "Unicode" is not an encoding. It never was, it never will be. Of course, the data must be encoded in memory somehow, but in Python 3, you cannot be sure what encoding that is because it's not really exposed to the user. From what I understand, there are different encodings that string objects will use, transparently, in order to save memory! You always "encode" something into bytes, and "decode" bytes back into something. There should be exactly two functions, because the functions have different types: "encode" is str -> bytes, "decode" is bytes -> str. Explicit is better than implicit. output = input.decode(self.coding) With Python 3, I instantly know that "input" is bytes and "output" is str.

hahainternet 11y ago
In [1]: len("नि") Out[1]: 2 So close, but it's not really a string or bytes, but a codepoint array. You can even iterate it: In [2]: for c in "नि": ...: print("Character {}".format(c)) ...: Character न Character ि
- Veedrac 11y ago
  Although true, the aim of the transition was to enforce encoding-correctness. Operations on strs should never break the encoding itself[1], but preserving the semantic meaning of the text itself is basically impossible for arbitrary data. [1] Sadly surrogate escapes break this. Rust's division into String:OsString:Vec<u8> makes that aspect a lot cleaner.
- dietrichepp 11y ago
  I'm getting tired of this argument, we had it in the Rust community as well and I was tired of it there, too. I've seen the same argument over and over again, and every argument had the same hole that I see in your post: What, exactly, is a "character"? Please answer me that, and then you can write a proposal for how the API should work, exactly, and why this works with Indic and German and Korean and everything else. Strings have to be arrays. It is inevitable. We are on Von Neumann machines with L1 cache lines that are something like 64 bytes wide, so making strings into something other than arrays is a complete non-starter. Because strings are arrays, we use array indexes to slice strings. The indexes are integers, because that just makes sense on a Von Neumann machine. So your complaint is that you can slice a string in "bad" ways. What makes it bad? You're trying to prevent people from slicing up grapheme clusters, which is a noble goal. But in practice, we get the indexes from methods like str.find() or re.match(), and we treat them as more or less opaque, so we don't end up slicing grapheme clusters very often in practice. Grapheme cluster slicing requires a big Unicode table in memory anyway, so in the rare case that you need it, you can pay the high performance cost for using it. In the meantime, formats like JSON and XML are defined in terms of code points, so using code point arrays eliminates a class of bugs where you could accidentally make malformed XML or JSON, which would then get completely REJECTED by the receiving side, causing your Jabber client to quit or your web browser to show a bunch of mojibake. And let me ask you this: what do I get when I write: x = "a" y = "\u0301" Is the resulting len(x + y) == 1? But len(x) == 1 and len(y) == 1? Can you tell me what the correct behavior is? Might you end up with bugs in programs because len(x + y) != len(x) + len(y)? Or do you introduce extra code points into the string when you concatenate them? Please, tell me what you actually think is correct behavior. It is far, far more useful than pointing out that something is "wrong" on purely semantic arguments.
  - anon1385 11y ago
    The correct answer is that strings shouldn't have length methods. There is no such thing as the 'length' of a string because as you pointed out there is no clear definition of what a 'character' is. https://www.mikeash.com/pyblog/friday-qa-2015-11-06-why-is-swifts-string-api-so-hard.html https://www.mikeash.com/pyblog/friday-qa-2015-11-06-why-is-s...
    - the_mitsuhiko 11y ago
      To add to that: O(1) indexing into strings outside of the ASCII range is completely pointless and makes absolutely no sense. A language should disallow that and not encourage it.
      - Veedrac 11y ago
        I disagree with it being pointless. Being able to get indices into a string for fast slicing is useful. Consider Regex where references get stored to slices in the array. What you can drop is the idea of the index being an integer with any particular semantic meaning. Rust uses byte indexes with an assertion that the index lies on a codepoint boundary, for instance.
        the_mitsuhiko 11y ago
        > Being able to get indices into a string for fast slicing is useful. Outside of the ASCII range? I doubt it. That most likely means you are doing something fundamentally wrong.
        heinrich5991 11y ago
        I think something like a lexer doesn't need to know about graphemes, it just needs to know which codepoints are XID_start and XID_continue for identifiers and hardcoded codepoints for everything else. EDIT: Or a JSON parser, etc.
twoodfin 11y ago
Indeed, the Python 3 Unicode string object is fascinatingly clever. Code worth reading: https://github.com/python/cpython/blob/master/Objects/unicodeobject.c https://github.com/python/cpython/blob/master/Objects/unicod...
- the_mitsuhiko 11y ago
  > Indeed, the Python 3 Unicode string object is fascinatingly clever. Code worth reading: Clever is relative. The Python 3 unicode string objects makes very little sense given real world situations. It wastes enormous amount of memory in many common scenarios for no reason at all other than to facilitate O(1) access to charpoints which is completely inappropriate for text processing anyways. Worse than that: it can blow up by another 50% the moment someone encodings it to utf-8 as the encoded version is cached around. The object is too clever for its own good.
  - klodolph 11y ago
    You're right that O(1) access is pointless, and UTF-8 is fine. But there are space optimizations on the Python string object which make it so it doesn't waste that much space. Basically, it dynamically selects LATIN-1, UCS-2, or UTF-32 (or you could call them UCS-1, UCS-2, UCS-4, but there isn't really any such thing as UCS-1, and UCS-4 is an obsolete name for UTF-32). >>> import sys >>> def charsize(x): ... return sys.getsizeof(x + x) - sys.getsizeof(x) ... >>> charsize(b'A') 1 >>> charsize('A') 1 >>> charsize('\xff') 1 >>> charsize('\uffff') 2 >>> charsize('\U0010FFFF') 4
    - the_mitsuhiko 11y ago
      > But there are space optimizations on the Python string object which make it so it doesn't waste that much space Except it does not help. When you render out a template it starts out as latin1, then the first time it hits a unicode char the whole thing re-encodes into ucs2 and then when it sees an emoji it re-encodes to ucs4. In practical terms you will see UCS2 and UCS4 all the time.
    - 11y ago
agumonkey 11y ago
It is ambiguous. When you still don't get encodings. As surprising as this can be, I click faster on monads than encodings (maybe it doesn't tease my brain enough for me to retain it).
cgrubb 11y ago
> You always "encode" something into bytes, and "decode" bytes back into something. There should be exactly two functions, because the functions have different types: "encode" is str -> bytes, "decode" is bytes -> str. Explicit is better than implicit. I'm not sure "encoding" tells us that the output is bytes. Surely information can be encoded in other ways. Names like .to_bytes() and .from_bytes() might have saved a lot of trips to the documentation.
- paulgayham 11y ago
  Encode is STR to Bytes? WTF? You 'encode' a string, already in UTF-8, to raw bytes? WTF?!
- mark-r 11y ago
  I would use .to_str() instead of .from_bytes(), since the action is from the perspective of the object you're calling it on.
- viraptor 11y ago
  > Surely information can be encoded in other ways. Information - yes. Strings... I'm not sure. What other representations are you thinking about?
  - msellout 11y ago
    urlencode(mapping) --> querystring
njharman 11y ago
ok, but what I read from a file is that bytes or "something". When save some "things" I read from utf16 file and want to save them to Postgresql who's client encoding is set to utf8, what do I do?
est 11y ago
> "Unicode" is not an encoding. It's important to point out that "Unicode" in Windows is an encoding. It means specifically UTF-16 LE, many people like me learned the concept "Unicode" on Windows got confused to hell when learning Python.
- mark-r 11y ago
  Microsoft - I don't know whether to hate on them or pity them. They were an early adopter of Unicode, back when there was only one encoding that probably didn't even have a name yet (UCS-2). It's easy to see how they would unleash that confusing nomenclature on the world, but they should have made an attempt to clear it up when other encodings became available, and they didn't.
aidenn0 11y ago
It is not true that unicode never was an encoding. It was originally a fixed-width 16-bit encoding[1]. This was arguably no longer true in the early '90s and definitely not true by 1996. 1: http://www.unicode.org/history/unicode88.pdf http://www.unicode.org/history/unicode88.pdf

hahainternet 11y ago

In [1]: len("नि") Out[1]: 2 So close, but it's not really a string or bytes, but a codepoint array. You can even iterate it: In [2]: for c in "नि": ...: print("Character {}".format(c)) ...: Character न Character ि

Veedrac 11y ago
Although true, the aim of the transition was to enforce encoding-correctness. Operations on strs should never break the encoding itself[1], but preserving the semantic meaning of the text itself is basically impossible for arbitrary data. [1] Sadly surrogate escapes break this. Rust's division into String:OsString:Vec<u8> makes that aspect a lot cleaner.
dietrichepp 11y ago
I'm getting tired of this argument, we had it in the Rust community as well and I was tired of it there, too. I've seen the same argument over and over again, and every argument had the same hole that I see in your post: What, exactly, is a "character"? Please answer me that, and then you can write a proposal for how the API should work, exactly, and why this works with Indic and German and Korean and everything else. Strings have to be arrays. It is inevitable. We are on Von Neumann machines with L1 cache lines that are something like 64 bytes wide, so making strings into something other than arrays is a complete non-starter. Because strings are arrays, we use array indexes to slice strings. The indexes are integers, because that just makes sense on a Von Neumann machine. So your complaint is that you can slice a string in "bad" ways. What makes it bad? You're trying to prevent people from slicing up grapheme clusters, which is a noble goal. But in practice, we get the indexes from methods like str.find() or re.match(), and we treat them as more or less opaque, so we don't end up slicing grapheme clusters very often in practice. Grapheme cluster slicing requires a big Unicode table in memory anyway, so in the rare case that you need it, you can pay the high performance cost for using it. In the meantime, formats like JSON and XML are defined in terms of code points, so using code point arrays eliminates a class of bugs where you could accidentally make malformed XML or JSON, which would then get completely REJECTED by the receiving side, causing your Jabber client to quit or your web browser to show a bunch of mojibake. And let me ask you this: what do I get when I write: x = "a" y = "\u0301" Is the resulting len(x + y) == 1? But len(x) == 1 and len(y) == 1? Can you tell me what the correct behavior is? Might you end up with bugs in programs because len(x + y) != len(x) + len(y)? Or do you introduce extra code points into the string when you concatenate them? Please, tell me what you actually think is correct behavior. It is far, far more useful than pointing out that something is "wrong" on purely semantic arguments.
- anon1385 11y ago
  The correct answer is that strings shouldn't have length methods. There is no such thing as the 'length' of a string because as you pointed out there is no clear definition of what a 'character' is. https://www.mikeash.com/pyblog/friday-qa-2015-11-06-why-is-swifts-string-api-so-hard.html https://www.mikeash.com/pyblog/friday-qa-2015-11-06-why-is-s...
  - the_mitsuhiko 11y ago
    To add to that: O(1) indexing into strings outside of the ASCII range is completely pointless and makes absolutely no sense. A language should disallow that and not encourage it.
    - Veedrac 11y ago
      I disagree with it being pointless. Being able to get indices into a string for fast slicing is useful. Consider Regex where references get stored to slices in the array. What you can drop is the idea of the index being an integer with any particular semantic meaning. Rust uses byte indexes with an assertion that the index lies on a codepoint boundary, for instance.
      - the_mitsuhiko 11y ago
        > Being able to get indices into a string for fast slicing is useful. Outside of the ASCII range? I doubt it. That most likely means you are doing something fundamentally wrong.
        heinrich5991 11y ago
        I think something like a lexer doesn't need to know about graphemes, it just needs to know which codepoints are XID_start and XID_continue for identifiers and hardcoded codepoints for everything else. EDIT: Or a JSON parser, etc.

twoodfin 11y ago

Indeed, the Python 3 Unicode string object is fascinatingly clever. Code worth reading: https://github.com/python/cpython/blob/master/Objects/unicodeobject.c https://github.com/python/cpython/blob/master/Objects/unicod...

the_mitsuhiko 11y ago
> Indeed, the Python 3 Unicode string object is fascinatingly clever. Code worth reading: Clever is relative. The Python 3 unicode string objects makes very little sense given real world situations. It wastes enormous amount of memory in many common scenarios for no reason at all other than to facilitate O(1) access to charpoints which is completely inappropriate for text processing anyways. Worse than that: it can blow up by another 50% the moment someone encodings it to utf-8 as the encoded version is cached around. The object is too clever for its own good.
- klodolph 11y ago
  You're right that O(1) access is pointless, and UTF-8 is fine. But there are space optimizations on the Python string object which make it so it doesn't waste that much space. Basically, it dynamically selects LATIN-1, UCS-2, or UTF-32 (or you could call them UCS-1, UCS-2, UCS-4, but there isn't really any such thing as UCS-1, and UCS-4 is an obsolete name for UTF-32). >>> import sys >>> def charsize(x): ... return sys.getsizeof(x + x) - sys.getsizeof(x) ... >>> charsize(b'A') 1 >>> charsize('A') 1 >>> charsize('\xff') 1 >>> charsize('\uffff') 2 >>> charsize('\U0010FFFF') 4
  - the_mitsuhiko 11y ago
    > But there are space optimizations on the Python string object which make it so it doesn't waste that much space Except it does not help. When you render out a template it starts out as latin1, then the first time it hits a unicode char the whole thing re-encodes into ucs2 and then when it sees an emoji it re-encodes to ucs4. In practical terms you will see UCS2 and UCS4 all the time.
  - 11y ago

agumonkey 11y ago

It is ambiguous. When you still don't get encodings. As surprising as this can be, I click faster on monads than encodings (maybe it doesn't tease my brain enough for me to retain it).

cgrubb 11y ago

> You always "encode" something into bytes, and "decode" bytes back into something. There should be exactly two functions, because the functions have different types: "encode" is str -> bytes, "decode" is bytes -> str. Explicit is better than implicit. I'm not sure "encoding" tells us that the output is bytes. Surely information can be encoded in other ways. Names like .to_bytes() and .from_bytes() might have saved a lot of trips to the documentation.

paulgayham 11y ago
Encode is STR to Bytes? WTF? You 'encode' a string, already in UTF-8, to raw bytes? WTF?!
mark-r 11y ago
I would use .to_str() instead of .from_bytes(), since the action is from the perspective of the object you're calling it on.
viraptor 11y ago
> Surely information can be encoded in other ways. Information - yes. Strings... I'm not sure. What other representations are you thinking about?
- msellout 11y ago
  urlencode(mapping) --> querystring

njharman 11y ago

ok, but what I read from a file is that bytes or "something". When save some "things" I read from utf16 file and want to save them to Postgresql who's client encoding is set to utf8, what do I do?

est 11y ago

> "Unicode" is not an encoding. It's important to point out that "Unicode" in Windows is an encoding. It means specifically UTF-16 LE, many people like me learned the concept "Unicode" on Windows got confused to hell when learning Python.

mark-r 11y ago
Microsoft - I don't know whether to hate on them or pity them. They were an early adopter of Unicode, back when there was only one encoding that probably didn't even have a name yet (UCS-2). It's easy to see how they would unleash that confusing nomenclature on the world, but they should have made an attempt to clear it up when other encodings became available, and they didn't.

aidenn0 11y ago

It is not true that unicode never was an encoding. It was originally a fixed-width 16-bit encoding[1]. This was arguably no longer true in the early '90s and definitely not true by 1996. 1: http://www.unicode.org/history/unicode88.pdf http://www.unicode.org/history/unicode88.pdf