4 ms·

Of course. What you were expecting was 6 right? You learnt ASCII. Then you learnt about escaping to get around limitations in ASCII. Then you learnt that Unico

Of course. What you were expecting was 6 right? You learnt ASCII. Then you learnt about escaping to get around limitations in ASCII. Then you learnt that Unicode got around non-Latin issues with ASCII. Each step has added cognitive load. It seems surprising to me that the next step wouldn't.

klodolph 11y ago
The surprise comes with a cost, and the cost is in bugs. The point of Python 3 is to eliminate some bugs in Python 2, and forcing developers to redesign their code around a completely new string API--no matter if that new API is more semantically sound--will come with its own cost of introduced bugs and more painful conversions, which goes against the purpose of Python 3. I'm not saying that the string API in Python is perfection, but it had a specific role to fill in the larger Python ecosystem, and it does that very well. Anyway... I would say that len(x) == 0 should be the same as saying that x is empty. According to the Unicode standard, the number of grapheme clusters in "\u0301" is 1, anyway! And on most systems, the string will display as an acute accent in its own space, and you can place the cursor before or after it. Not like the set of valid "cursor positions" in a string isn't a locale-dependent value, anyway. Grapheme clusters can be thought of to be locale-independent but they are NOT the same as cursor positions.
- lucozade 11y ago
  Not to sound too factious but the decision to redesign the string API was a big reason why Python 3 has been a little on the slow cooked side. In for a penny and all that. If I recall correctly the modifier on it's own isn't correct. There is a base that should go before it to produce a valid character. I don't recall the base unfortunately and I don't have the spec to hand.
  - klodolph 11y ago
    What do you mean by "correct"? The Unicode standard talks about well-formed UTF-8/16/32, but in general, any sequence of code points is permissible. By derivation, any sequence of code points can appear in JSON strings, and any sequence of code points excluding control characters can appear in XML. I personally have read the Unicode spec, and used it to implement some of the algorithms presented. This includes the grapheme cluster breaking algorithm... the case of combining characters at the beginning of a line is, in fact, covered by the spec, and cause a break both before and afterwards. In general, there are other cases where grapheme cluster breaks are not preserved by string concatenation, and you often want a locale-specific version anyway.
    - lucozade 11y ago
      > What do you mean by "correct"? I mean this from the standard: "Nonspacing combining marks used by the Unicode Standard may be exhibited in apparent isolation by applying them to U+00A0 no-break space. " You can have valid Unicode that doesn't do this. But it's not intended to represent the character as some kind of textual thing. It's intended so that you can build up Unicode strings sensibly. Same idea as allowing mis-matched surrogates.

klodolph 11y ago

The surprise comes with a cost, and the cost is in bugs. The point of Python 3 is to eliminate some bugs in Python 2, and forcing developers to redesign their code around a completely new string API--no matter if that new API is more semantically sound--will come with its own cost of introduced bugs and more painful conversions, which goes against the purpose of Python 3. I'm not saying that the string API in Python is perfection, but it had a specific role to fill in the larger Python ecosystem, and it does that very well. Anyway... I would say that len(x) == 0 should be the same as saying that x is empty. According to the Unicode standard, the number of grapheme clusters in "\u0301" is 1, anyway! And on most systems, the string will display as an acute accent in its own space, and you can place the cursor before or after it. Not like the set of valid "cursor positions" in a string isn't a locale-dependent value, anyway. Grapheme clusters can be thought of to be locale-independent but they are NOT the same as cursor positions.

lucozade 11y ago
Not to sound too factious but the decision to redesign the string API was a big reason why Python 3 has been a little on the slow cooked side. In for a penny and all that. If I recall correctly the modifier on it's own isn't correct. There is a base that should go before it to produce a valid character. I don't recall the base unfortunately and I don't have the spec to hand.
- klodolph 11y ago
  What do you mean by "correct"? The Unicode standard talks about well-formed UTF-8/16/32, but in general, any sequence of code points is permissible. By derivation, any sequence of code points can appear in JSON strings, and any sequence of code points excluding control characters can appear in XML. I personally have read the Unicode spec, and used it to implement some of the algorithms presented. This includes the grapheme cluster breaking algorithm... the case of combining characters at the beginning of a line is, in fact, covered by the spec, and cause a break both before and afterwards. In general, there are other cases where grapheme cluster breaks are not preserved by string concatenation, and you often want a locale-specific version anyway.
  - lucozade 11y ago
    > What do you mean by "correct"? I mean this from the standard: "Nonspacing combining marks used by the Unicode Standard may be exhibited in apparent isolation by applying them to U+00A0 no-break space. " You can have valid Unicode that doesn't do this. But it's not intended to represent the character as some kind of textual thing. It's intended so that you can build up Unicode strings sensibly. Same idea as allowing mis-matched surrogates.