4 ms·
Python 3 has arguably one of the best built-in string implementations around. In Python 2, "unicode" was a type whose codepoint width depended on the interpret
by nneonneo 10y ago
Python 3 has arguably one of the best built-in string implementations around.
In Python 2, "unicode" was a type whose codepoint width depended on the interpreter build - 2 bytes on "narrow" builds and 4 bytes on "wide" builds. Since most builds were "narrow", in practice non-BMP codepoints were a real challenge to use.
Furthermore, most languages with a "wide" string type use a fixed 2 bytes per codepoint (Java, C#, C++, ...), which wastes space when you're dealing with ASCII a lot, and is a pain to work with if you have non-BMP codepoints, since now indexing requires scanning the string from start to end.
In Python 3, with PEP 393, strings now have a flexible internal representation which can use 1, 2, or 4 bytes per codepoint depending on the largest codepoint in the string. This saves both space and processing time (in most common situations) since common string operations like indexing and slicing are constant time. This representation also allows it to scale from ASCII to astral codepoints with ease.
Python 3 is worth the switch. Correct, strict separation of 'bytes' (bucket of octets) and 'str' (sequence of codepoints) is really the only way to preserve sanity and interoperability with today's encoding-rife reality.
- tannhaeuser 10y agoI'm not disagreeing but the situation is a bit more complex. For example ISO 10646 ("Unicode") has multi-code point sequences denoting still a single character (such as variation sequences, which are actually used in standard HTML entities). Moreover, there are languages/scripts challenging the notion that a particular byte-unit corresponds to a single character. Treating strings as byte sequences (with optional UTF-8 interpretation/checking, uppercase/lowercase conversion if the concept even applies, trim functions based on all Unicode spaces not just those characters in US-ASCII etc.) is entirely a defensible choice for a programming language.
- nneonneo 10y agoI was sloppy in using the word "character" - strings are really sequences of codepoints. I've edited my post to reflect this. Of course, multiple codepoints may be needed to construct a single character, and that's by design.
- tannhaeuser 10y agoBut has Python 3 Unicode-aware string-length rather than byte-length functions (+ substring, case-folding, character classification functions etc.) for dealing with this? If it hasn't then nothing really is achieved for the general case by switching to 4-byte or any other fixed-byte unit, including the case of dealing with HTML 5 text. See also perlgeek's question.
- jnbiche 10y agoYes, Python 3 does have Unicode-aware functions, as demonstrated in the post (via slices). Go and Rust also have very good Unicode support, in addition to Swift and (apparently) Perl 6 as mentioned above.
- perlgeek 10y ago> For example ISO 10646 ("Unicode") has multi-code point sequences denoting still a single character I only know two programming languages that deal correctly with that: Swift and Perl 6. If you know more, please tell me.
- steveklabnik 10y agoRust separates this out; the standard library gives you bytes and Unicode Scalar Values. Grapheme cluster stuff is in a package on Cargo, maintained by the Servo team.
- bquinlan 10y agoYou can iterate over Grapheme clusters using the standard library: https://doc.rust-lang.org/1.3.0/std/str/struct.Graphemes.html https://doc.rust-lang.org/1.3.0/std/str/struct.Graphemes.htm...
- steveklabnik 10y agoThose are unstable docs for 1.3; they don't exist in today's Rust.
- jacobush 10y agoSounds like Rust follows in the tradition of C. Like you now always hear about hos C has no string type, just a bunch of char, will we hear in the future that, alas, Rust has no native support for grapheme clustering?
- steveklabnik 10y agoThe issue is, there's multiple ways to do the clustering, in my understanding. We picked the most core thing to be our most core thing. Text is hard.
- jacobush 10y ago
- nradov 10y agoJava hasn't used 2 bytes per code point since many releases ago. It does unfortunately use 2 bytes per char, but the standard library makes it reasonably easy to deal with cases where you need 2 chars for a code point.
- Manishearth 10y agoI would say Swift has the best implementation. It actually conceptually treats strings as a group of grapheme clusters (the Character type is a grapheme cluster). This is more in line with how humans conceptually segment strings, and how most application logic should segment a string. > since now indexing requires scanning the string from start to end. This is not a good thing. O(1) code point indexing is a completely useless operation. Code points have no intrinsic meaning. They do not map to any natural language concept. They are just a tool to encode strings. The only time code points matter are when writing algorithms defined by the unicode spec, like casefolding. These operations usually require iteration anyway. If you're regularly indexing and slicing strings by code points, your code is likely broken when it comes across decomposed accents in European languages, most Asian languages, and emoji. FWIW you can have O(1) slice indexing in UTF8 as well; you just need to use byte position as your indexing type (and handle the error case when it's not aligned). Rust does this. It also has methods for working with char indices (and a .char_indices() iterator that lets you juggle the two), but byte indexing works. Generally these indices come from other parts of the application so you can design it to not have errors.
- Manishearth 10y agoI end up having to explain this often, and have been meaning to write about this for a long time, and I finally did: http://manishearth.github.io/blog/2017/01/14/stop-ascribing-meaning-to-unicode-code-points/ http://manishearth.github.io/blog/2017/01/14/stop-ascribing-...