4 ms·

I'm not disagreeing but the situation is a bit more complex. For example ISO 10646 ("Unicode") has multi-code point sequences denoting still a single character

I'm not disagreeing but the situation is a bit more complex. For example ISO 10646 ("Unicode") has multi-code point sequences denoting still a single character (such as variation sequences, which are actually used in standard HTML entities). Moreover, there are languages/scripts challenging the notion that a particular byte-unit corresponds to a single character. Treating strings as byte sequences (with optional UTF-8 interpretation/checking, uppercase/lowercase conversion if the concept even applies, trim functions based on all Unicode spaces not just those characters in US-ASCII etc.) is entirely a defensible choice for a programming language.

nneonneo 10y ago
I was sloppy in using the word "character" - strings are really sequences of codepoints. I've edited my post to reflect this. Of course, multiple codepoints may be needed to construct a single character, and that's by design.
- tannhaeuser 10y ago
  But has Python 3 Unicode-aware string-length rather than byte-length functions (+ substring, case-folding, character classification functions etc.) for dealing with this? If it hasn't then nothing really is achieved for the general case by switching to 4-byte or any other fixed-byte unit, including the case of dealing with HTML 5 text. See also perlgeek's question.
  - jnbiche 10y ago
    Yes, Python 3 does have Unicode-aware functions, as demonstrated in the post (via slices). Go and Rust also have very good Unicode support, in addition to Swift and (apparently) Perl 6 as mentioned above.
perlgeek 10y ago
> For example ISO 10646 ("Unicode") has multi-code point sequences denoting still a single character I only know two programming languages that deal correctly with that: Swift and Perl 6. If you know more, please tell me.
- steveklabnik 10y ago
  Rust separates this out; the standard library gives you bytes and Unicode Scalar Values. Grapheme cluster stuff is in a package on Cargo, maintained by the Servo team.
  - bquinlan 10y ago
    You can iterate over Grapheme clusters using the standard library: https://doc.rust-lang.org/1.3.0/std/str/struct.Graphemes.html https://doc.rust-lang.org/1.3.0/std/str/struct.Graphemes.htm...
    - steveklabnik 10y ago
      Those are unstable docs for 1.3; they don't exist in today's Rust.
  - jacobush 10y ago
    Sounds like Rust follows in the tradition of C. Like you now always hear about hos C has no string type, just a bunch of char, will we hear in the future that, alas, Rust has no native support for grapheme clustering?
    - steveklabnik 10y ago
      The issue is, there's multiple ways to do the clustering, in my understanding. We picked the most core thing to be our most core thing. Text is hard.
      - jacobush 10y ago
        Oh I get it didn't want to come across as flippant. Indeed text is hard.
  - perlgeek 10y ago
    If you rely on libraries, there are ICU libraries for C and Java too, see for example http://userguide.icu-project.org/boundaryanalysis http://userguide.icu-project.org/boundaryanalysis

nneonneo 10y ago

I was sloppy in using the word "character" - strings are really sequences of codepoints. I've edited my post to reflect this. Of course, multiple codepoints may be needed to construct a single character, and that's by design.

tannhaeuser 10y ago
But has Python 3 Unicode-aware string-length rather than byte-length functions (+ substring, case-folding, character classification functions etc.) for dealing with this? If it hasn't then nothing really is achieved for the general case by switching to 4-byte or any other fixed-byte unit, including the case of dealing with HTML 5 text. See also perlgeek's question.
- jnbiche 10y ago
  Yes, Python 3 does have Unicode-aware functions, as demonstrated in the post (via slices). Go and Rust also have very good Unicode support, in addition to Swift and (apparently) Perl 6 as mentioned above.

perlgeek 10y ago

> For example ISO 10646 ("Unicode") has multi-code point sequences denoting still a single character I only know two programming languages that deal correctly with that: Swift and Perl 6. If you know more, please tell me.

steveklabnik 10y ago
Rust separates this out; the standard library gives you bytes and Unicode Scalar Values. Grapheme cluster stuff is in a package on Cargo, maintained by the Servo team.
- bquinlan 10y ago
  You can iterate over Grapheme clusters using the standard library: https://doc.rust-lang.org/1.3.0/std/str/struct.Graphemes.html https://doc.rust-lang.org/1.3.0/std/str/struct.Graphemes.htm...
  - steveklabnik 10y ago
    Those are unstable docs for 1.3; they don't exist in today's Rust.
- jacobush 10y ago
  Sounds like Rust follows in the tradition of C. Like you now always hear about hos C has no string type, just a bunch of char, will we hear in the future that, alas, Rust has no native support for grapheme clustering?
  - steveklabnik 10y ago
    The issue is, there's multiple ways to do the clustering, in my understanding. We picked the most core thing to be our most core thing. Text is hard.
    - jacobush 10y ago
      Oh I get it didn't want to come across as flippant. Indeed text is hard.
- perlgeek 10y ago
  If you rely on libraries, there are ICU libraries for C and Java too, see for example http://userguide.icu-project.org/boundaryanalysis http://userguide.icu-project.org/boundaryanalysis