6 ms·

> Python 2 has ASCII str() types, separate unicode(), but no byte type. Actually Py2's str class is a double-duty str/byte class such that some methods behave

> Python 2 has ASCII str() types, separate unicode(), but no byte type. Actually Py2's str class is a double-duty str/byte class such that some methods behave like str and others like byte. py2>> len('¡no') #string len=3 #encoded UTF-8 byte array len=4 4 #always gives bytes.len not str.len (see http://stackoverflow.com/questions/5471158/typeerror-str-does-not-support-the-buffer-interface/34688434#34688434 http://stackoverflow.com/questions/5471158/typeerror-str-doe...)

kzrdude 9y ago
No need to get too much into the meaning of the length of a str. It's neat to know the definition but neither choice gives you a good “string length” for real text. Case in point: len("é") == 2, so even with unicode strings we don't have length as a character count. As an english speaking person, I can clearly point at é and call it one character, but Python 2 or Python 3 doesn't agree. Rather, their str `len` doesn't agree with that, and that's fine. It's not a character count!
- dom0 9y ago
  That one is maybe not the best example, since len(unicodedata.normalize('NFC', "é")) == 1 Though I don't know any "normal" character that requires composition ad-hoc that could serve as a better example.
  - kzrdude 9y ago
    I think it's a fine example since the decomposed version clearly exists and can arrive as data into your program in a number of ways. That said, the grapheme cluster note will have examples of extended notions of characters that can't be represented by an equivalent single codepoint. There are some korean and indic examples and also emoji http://unicode.org/reports/tr29/ http://unicode.org/reports/tr29/ Oh and something that stirs the heart of us hackers: \r\n is a single grapheme!
    - greglindahl 9y ago
      This (different forms and no automatic normalization) is a "feature" of Unicode. It would be wrong of Python to try to cover it up.
      - kzrdude 9y ago
        Yes, so it's wrong for anyone to pretend len() or even indexing of a str is about characters, it's about codepoints of the string.
        wruza 9y ago
        IMO, len("") should raise OperationHasNoCommonSenseTodayException, just to not confuse unicode newbies. Modern text is not an array, it is a format, a complex format. Zero-width, double-width monotypes, combining, direction marks, normalization, etc. Almost no one wants to know about codepoint details when working with "just input strings", and those who want may use special namespaces for that. There is no point in making len() an alias for unicode.volatile_number_of_distinct_code_points(). visually_empty(s) is okay, len(s) is probably not.
        mrout 9y ago
        They're all valid measurements. The length of a string could be measured in bytes (which doesn't require you to know the encoding), code points (which doesn't require huge Unicode grapheme clustering tables), grapheme clusters (which doesn't require a font) or even pixels (which does require a font).
        int_19h 9y ago
        The best thing a language (or library) can do is to not bless any single one of these as the default. Strings shouldn't have a length at all. They should provide properties/methods/accessors like byte_count, codepoint_count, grapheme_count etc. Make the user of the API think every time they're asking for a length of the string - which one do they actually need? Which one is the best for whatever they're trying to do?
  - user5994461 9y ago
    ß é à ç ñ How about these?
- derefr 9y ago
  "Character count" (i.e. grapheme cluster count) only exists once you apply a font—because of ligature rules and some other stuff. Do you want your programming-language stdlib String class to require a font metrics library? (This isn't a rhetorical question; you can go either way. Objective C cares about grapheme clusters, and so needs to know about fonts.)
  - mrout 9y ago
    I disagree. A grapheme cluster and a glyph are different things. The former is not dependent on the font. 'fi' is two grapheme clusters even if the font renders it as a single glyph. 'é' is a single grapheme cluster even if the font renders it as two glyphs: 'e ́'
    - derefr 9y ago
      Consider the https://en.wikipedia.org/wiki/Regional_Indicator_Symbol https://en.wikipedia.org/wiki/Regional_Indicator_Symbol s. In every way that matters, these "act as" one 'character': you can't set your cursor position to be between them; backspacing one should delete both; the flag takes up one terminal col (even if it is rendered as two) such that "\033[1Dx" (cursor-left 1, print "x") will overwrite the whole flag, etc. But it's the font that controls those semantics—because it's the font that knows what flags do or do not exist. For any pair of RIS codepoints that doesn't form a flag (in the opinion of a given font), they behave like two separate characters. Thus: one grapheme cluster, or two, depending on the font. And another example—this one much less "idiomatic", but not specifically decried by the Unicode committee: http://kudakurage.com/ligature_symbols/ http://kudakurage.com/ligature_symbols/ That's a font, making entirely-arbitrary clusters out of codepoints. As far as I am aware, it's fully within its rights as a font to do so. There's nothing in the Unicode standard saying that the code-points ['f', 'i', 'l', 'e'], put in a row, can't combine to form a single grapheme cluster. They don't have combining behavior themselves, but—unlike, say, things in the Unicode "Separator" class—they don't have any property that says they don't combine with anything.
      - mrout 9y ago
Animats 9y ago
Python 3's "bytes" type can't quite decide whether it's an array of small integers or a string. > a = bytes([97,98,99]) > print(a) b'abc' > a[0] 97 You can use regular expressions on a "bytes" object, as well as most of the string operations, such as ".center()" The "bytes" type in Python 3 still does double duty as an ASCII str/byte class.
- jjawssd 9y ago
  Python 3: In this case print(a) calls a.__str__() which happens to default to calling a.__repr__() because a doesn't have an __str__ attribute. a[0] is equivalent to a.__getitem__(0) That's because the __repr__ of a bytes object instance formats the result into a human readable representation of the data which should ideally be identical to the syntax needed to create the instance, whereas the __getitem__ method returns a "real" and machine usable representation. Try it... >>> a b'abc' >>> a.__str__() "b'abc'" >>> a.__repr__() "b'abc'" >>> a.__getitem__(0) 97

kzrdude 9y ago

No need to get too much into the meaning of the length of a str. It's neat to know the definition but neither choice gives you a good “string length” for real text. Case in point: len("é") == 2, so even with unicode strings we don't have length as a character count. As an english speaking person, I can clearly point at é and call it one character, but Python 2 or Python 3 doesn't agree. Rather, their str `len` doesn't agree with that, and that's fine. It's not a character count!

dom0 9y ago
That one is maybe not the best example, since len(unicodedata.normalize('NFC', "é")) == 1 Though I don't know any "normal" character that requires composition ad-hoc that could serve as a better example.
- kzrdude 9y ago
  I think it's a fine example since the decomposed version clearly exists and can arrive as data into your program in a number of ways. That said, the grapheme cluster note will have examples of extended notions of characters that can't be represented by an equivalent single codepoint. There are some korean and indic examples and also emoji http://unicode.org/reports/tr29/ http://unicode.org/reports/tr29/ Oh and something that stirs the heart of us hackers: \r\n is a single grapheme!
  - greglindahl 9y ago
    This (different forms and no automatic normalization) is a "feature" of Unicode. It would be wrong of Python to try to cover it up.
    - kzrdude 9y ago
      Yes, so it's wrong for anyone to pretend len() or even indexing of a str is about characters, it's about codepoints of the string.
      - wruza 9y ago
        IMO, len("") should raise OperationHasNoCommonSenseTodayException, just to not confuse unicode newbies. Modern text is not an array, it is a format, a complex format. Zero-width, double-width monotypes, combining, direction marks, normalization, etc. Almost no one wants to know about codepoint details when working with "just input strings", and those who want may use special namespaces for that. There is no point in making len() an alias for unicode.volatile_number_of_distinct_code_points(). visually_empty(s) is okay, len(s) is probably not.
      - mrout 9y ago
        They're all valid measurements. The length of a string could be measured in bytes (which doesn't require you to know the encoding), code points (which doesn't require huge Unicode grapheme clustering tables), grapheme clusters (which doesn't require a font) or even pixels (which does require a font).
        int_19h 9y ago
        The best thing a language (or library) can do is to not bless any single one of these as the default. Strings shouldn't have a length at all. They should provide properties/methods/accessors like byte_count, codepoint_count, grapheme_count etc. Make the user of the API think every time they're asking for a length of the string - which one do they actually need? Which one is the best for whatever they're trying to do?
- user5994461 9y ago
  ß é à ç ñ How about these?
derefr 9y ago
"Character count" (i.e. grapheme cluster count) only exists once you apply a font—because of ligature rules and some other stuff. Do you want your programming-language stdlib String class to require a font metrics library? (This isn't a rhetorical question; you can go either way. Objective C cares about grapheme clusters, and so needs to know about fonts.)
- mrout 9y ago
  I disagree. A grapheme cluster and a glyph are different things. The former is not dependent on the font. 'fi' is two grapheme clusters even if the font renders it as a single glyph. 'é' is a single grapheme cluster even if the font renders it as two glyphs: 'e ́'
  - derefr 9y ago
    Consider the https://en.wikipedia.org/wiki/Regional_Indicator_Symbol https://en.wikipedia.org/wiki/Regional_Indicator_Symbol s. In every way that matters, these "act as" one 'character': you can't set your cursor position to be between them; backspacing one should delete both; the flag takes up one terminal col (even if it is rendered as two) such that "\033[1Dx" (cursor-left 1, print "x") will overwrite the whole flag, etc. But it's the font that controls those semantics—because it's the font that knows what flags do or do not exist. For any pair of RIS codepoints that doesn't form a flag (in the opinion of a given font), they behave like two separate characters. Thus: one grapheme cluster, or two, depending on the font. And another example—this one much less "idiomatic", but not specifically decried by the Unicode committee: http://kudakurage.com/ligature_symbols/ http://kudakurage.com/ligature_symbols/ That's a font, making entirely-arbitrary clusters out of codepoints. As far as I am aware, it's fully within its rights as a font to do so. There's nothing in the Unicode standard saying that the code-points ['f', 'i', 'l', 'e'], put in a row, can't combine to form a single grapheme cluster. They don't have combining behavior themselves, but—unlike, say, things in the Unicode "Separator" class—they don't have any property that says they don't combine with anything.
    - mrout 9y ago

Animats 9y ago

Python 3's "bytes" type can't quite decide whether it's an array of small integers or a string. > a = bytes([97,98,99]) > print(a) b'abc' > a[0] 97 You can use regular expressions on a "bytes" object, as well as most of the string operations, such as ".center()" The "bytes" type in Python 3 still does double duty as an ASCII str/byte class.

jjawssd 9y ago
Python 3: In this case print(a) calls a.__str__() which happens to default to calling a.__repr__() because a doesn't have an __str__ attribute. a[0] is equivalent to a.__getitem__(0) That's because the __repr__ of a bytes object instance formats the result into a human readable representation of the data which should ideally be identical to the syntax needed to create the instance, whereas the __getitem__ method returns a "real" and machine usable representation. Try it... >>> a b'abc' >>> a.__str__() "b'abc'" >>> a.__repr__() "b'abc'" >>> a.__getitem__(0) 97