17 ms·

How Python does Unicode

165 points by scw 9y ago

carapace 9y ago
Unicode is a horrible scam, the worst thing to happen to digital language representation. This is all just so much turd polishing. (Also, that explanation of UTF-8 is crap. UTF-8 is beautiful quite apart from its utility, but you'd hardly know it from the article.) I've said it before: Unicode is a conflation of a good idea and an impossible idea. The good idea is a standard mapping from numbers to little pictures. That's all ASCII was. The impossible idea is a digital code for every way humans write. It's a form of digital cultural imperialism. Unicode Consortium et. al. are absurdly arrogant.
- simonh 9y ago
  Critical rants that don't suggest a better alternative, or describe what a better alternative might look like even in outline, are rarely informative or persuasive.
  - carapace 9y ago
    Step One: Admit there's a problem. I heard, "Tell me more about what you think would be better." Here goes: For written languages that are well-served by a simple sequence of symbols (English, etc.) there is no problem: a catalog of the mappings from numbers to pictures is fine is all that is required. Put them in a sequence (anoint UTF-8 as the One True Encoding) and you're good-to-go. For languages that are NOT well-served by this simple abstraction the first thing to do (assuming you have the requisite breadth and depth of linguistic knowledge) is to figure out simple formal systems that do abstract the languages in question. Then determine equivalence classes and standardize the formal systems. Let the structure of the language abstraction be a "first-class" entity that has reference implementations. Instead of adding weird modifiers and other dynamic behavior to the code, let them be actual simple DSLs whose output is the proper graphics. Human languages are like a superset of what computers can represent. Here's the Unicode Standard[1] on Arabic: > The basic set of Arabic letters is well defined. Each letter receives only one Unicode character value in the basic Arabic block, no matter how many different contextual appearances it may exhibit in text. Each Arabic letter in the Unicode Standard may be said to represent the inherent semantic identity of the letter. A word is spelled as a sequence of these letters. The representative glyph shown in the Unicode character chart for an Arabic letter is usually the form of the letter when standing by itself. It is simply used to distinguish and identify the character in the code charts and does not restrict the glyphs used to represent it. They baldly admit that Unicode is not good for drawing Arabic. I find the phrase "the inherent semantic identity of the letter" to be particularly rich. It's nearly mysticism. If it is inconvenient to try to represent a language in terms of a sequence of symbols, then let's represent it as a (simple) program that renders the language correctly, which allows us to shoehorn non-linear behavior into a sequence of symbols. If you think about it, this is already what Unicode is doing with modifiers and such. If you read further in the Unicode Standard doc I quoted above you'll see that they basically do create a kind of DSL for dealing with Arabic. I'm saying: make it explicit. Don't try to pretend that Unicode is one big standard for human languages. Admit that the "space" of writing systems is way bigger and more involved than Latin et. al. Study the problem of representing writing in a computer as a first-class issue. Publish reference implementations of code that can handle each kind of writing system along with the catalog of numbered pictures. From the Unicode Standard again: > The Arabic script is cursive, even in its printed form. As a result, the same letter may be written in different forms depending on how it joins with its neighbors. Vow-els and various other marks may be written as combining marks called tashkil, which are applied to consonantal base letters. In normal writing, however, these marks are omitted. Computer systems that are adapted to English are not going to work for Arabic. I'd love to use a language simpler than PostScript to draw Arabic! Unicode strings are not that language. Consider the "Base-4 fractions in Telugu" https://blog.plover.com/math/telugu.html https://blog.plover.com/math/telugu.html The fact that we have a way to represent the graphics ౦౼౽౾౸౹౺౻ is great! But any software that wants to use them properly will require some code to translate to and from numbers in the computer to Telugu sequences of those graphics. Let that be part of "Unicode" and I'll shut up. In the meantime, I feel like it's a huge scam and a kind of cultural imperialism from us hacker types to the folks who are late to the party and for whom ASCII++ isn't going to really cut it. To sum up: I think the thing that replaces Unicode for dealing with human languages in digital form should: A.) Be created by linguists with help from computer folks, not by computer folks with some nagging from linguists (apologies to the linguist/computer folk who actually did the stuff.) B.) We should clearly state the problems first: What are the ways that human language are written down? C.) Write specific DSLs for each kind of writing. Publish reference implementations. I think that's it. Are you informed? Persuaded even? Entertained at least? ;-) [1] http://www.unicode.org/versions/Unicode9.0.0/ch09.pdf http://www.unicode.org/versions/Unicode9.0.0/ch09.pdf
    - ubershmekel 9y ago
      > Unicode is a horrible scam, the worst thing to happen to digital language representation. > They baldly admit that Unicode is not good for drawing Arabic > Consider the "Base-4 fractions in Telugu" [...] any software that wants to use them properly will require some code to translate to and from numbers in the computer to Telugu sequences of those graphics. Written language is hard to represent, encode and draw. You admit that Unicode&utf8 got 2 out of 3 right yet you call it a scam. Your complaint is a scam and a horrible trollism.
      - carapace 9y ago
        When I'm trolling you'll know it. I have a point, I believe it's a good point, and I'm making it. For languages that can be represented as a sequence of little pictures Unicode is a little better than ASCII. For the rest, it's a scam: We tell people that we have a way of dealing with human languages in computers but it's half-baked, born in ignorance, and all the grotty details are papered over, but you can write PIZZA SLICE or POOP now, so fuck it, ship it. Represent: "Astral Plane"? What does that have to do with a standard catalog mapping numbers to pictures? I feel Unicode messes that up. Encoding: UTF-8 is near perfect, 'nuff said. Ken Thompson and Rob Pike doing their thing. Drawing: Doesn't even begin to touch it really. Unicode is a nasty little black hole that's sucking up time and other resources and not really solving the problem.
        mixmastamyk 9y ago
        I think that's too negative. Is Unicode perfect? Of course not, but it's the best we've got for now. Just as Morse, Baudot, or ASCII were the best approximations at one point in time. It's a hard problem and will take decades to for the right solutions/implementations to present themselves. Surely one day there will be an improved successor to Unicode. Things are a lot better than they were even ten years ago, however.
        carapace 9y ago
        Yeah, sorry, I was pretty cranky last night. Please see my reply to simonh in this thread a few minutes ago. (I'm basically agreeing with you.)
- ubernostrum 9y ago
  that explanation of UTF-8 is crap. UTF-8 is beautiful quite apart from its utility, but you'd hardly know it from the article My goal was not to judge UTF-8 aesthetically, but to explain how it works and point out that it's a variable-width encoding which emphasizes its compatibility with ASCII for strings containing only code points <= U+007F. Unicode Consortium et. al. are absurdly arrogant. I would agree that Unicode as it exists today involves some historical and historic bad decisions. But again, staying off value judgments with respect to Unicode itself since the point of the article was to explain how Python now handles it internally.
  - carapace 9y ago
    Oh, Hi there. Apologies for being cranky. You did a great job explaining how Python now handles Unicode! To me it was strange reading about UTF-32 first and then getting to UTF-8 from that context. It seemed to obscure the coolth and beauty of the format. Overall a great article, sorry again for being so negative.
    - ubernostrum 9y ago
      That section was written for people who know little to nothing about Unicode and the ways Unicode can be encoded to bytes. So it starts with the obvious approach -- just spit out a sequence of bytes whose integer values are the code points, which is near enough as makes no difference to how UTF-32 works -- then introduces variable-width encoding through the history of UCS-2 and UTF-16, then gets to UTF-8 and what motivated it. The advantages/disadvantages of the various encodings is something that could eat up several pieces just as long as the entire post, and for fun I'd probably throw in weird stuff like the attempt to do EBCDIC-compatible UTF instead of ASCII-compatible, etc.
      - WorldMaker 9y ago
        Someone should write up EBCDIC-based UTF as an RFC. I'm sure that there's at least one COBOL programmer out there that has been waiting for that for decades. ETA: Mostly a joke, but it would also fit right in with things like WTF-8 (https://simonsapin.github.io/wtf-8/ https://simonsapin.github.io/wtf-8/)
Avernar 9y ago
I'm not a fan of how Python 3 stores Unicode strings internally. In my opinion they should have went with UTF-8. The extra scanning and conversion puts more preassure on the processor and caches under load. I agree that Python 2's Unicode handling is broken. That's why I just stored UTF-8 in a normal string and avoided the whole mess. The only thing I have to do is validate any input from the outside world is really UTF-8.
- ubernostrum 9y ago
  Since the high-level API is supposed to let you treat a string as a sequence of code points, a correct implementation (which Python didn't have until 3.3!) would've imposed the overhead of conversion to something resembling a fixed-width encoding whenever a programmer invoked certain operations. And the vast majority of strings in real-world Python contain only code points also present in latin-1, which means they can be stored in one byte per code point with this approach. And for strings which can't be stored in one byte per code point, you were similarly going to pay the price sooner or later.
  - Avernar 9y ago
    > Since the high-level API is supposed to let you treat a string as a sequence of code points, I disagree with that premise. It should operate on grapheme clusters. Operating on code points falls into the same trap as operating on bytes. > a correct implementation (which Python didn't have until 3.3!) would've imposed the overhead of conversion to something resembling a fixed-width encoding whenever a programmer invoked certain operations. Those operations should have been removed. Indexing is the big one that needs fixed width internal representation for speed. Code could have been rewritten to not require indexing. But mechanical translation from Python 2 to 3 was a goal and because of that they couldn't radically change the unicode API for the better. > And the vast majority of strings in real-world Python contain only code points also present in latin-1, which means they can be stored in one byte per code point with this approach. And for strings which can't be stored in one byte per code point, you were similarly going to pay the price sooner or later. You're going to pay the price for 4 byte per codepoint strings quite often. A single emoji will blow up a latin-1 string to 4 times the size.
dguaraglia 9y ago
My favorite story about Python's handling of Unicode was when one of my coworkers did a hotfix for our Python website, wrote tests, confirmed everything worked as expected... but right before committing and pushing to production wrote a comment like: # Apparently we expect the field to be in this format ¯\_(ツ)_/¯ Right above the code he'd just fixed. Of course, the moment we pushed the update it brought production down, because the Python interpreter doesn't understand Unicode in source files unless you specify which encoding you are using. After that, "¯\_(ツ)_/¯" became a synonym for his name on our HipChat server, heh.
- ubernostrum 9y ago
  This would be the case in Python 2, where source code files are assumed to be ASCII-encoded unless there's an encoding comment at the top of the file. In Python 3, source code files are assumed to be UTF-8.
  - dguaraglia 9y ago
    Correct, this was a codebase that still had some Pylons (gasp! Not even Pyramid, but legit Pylons) code.
  - ninkendo 9y ago
    Interesting that Python 2 couldn't fix that in a hotfix/point release... UTF-8 is backwards compatible with ASCII so it shouldn't break anything if source started being interpreted as UTF8. I'd be curious to see what their reasoning is.
    - dguaraglia 9y ago
      I would imagine Python's approach to introducing new language features had a lot to do with it. Having to go through the PEP system takes some time, and changes like these tend to be reserved for minor-version releases. All in all, I love the PEP system, it's such an open concept and I've been surprised by the amount of quality proposals that get implemented. Wish Go had something like it.
    - 9y ago
camgunz 9y ago
UCS-4 is essentially never the right choice. It wastes space and thus messes up your cache. UCS-2 can be the right choice if the language you're encoding uses a lot of non-Latin glyphs (i.e. East Asian languages) but suffers from the same problem as UCS-2. UTF-8 is a good default: for most strings it's very compact, and for strings with a lot of multibyte codepoints it doesn't compare too unfavorably with UTF-16. Python 3 tried to have its cake and eat it too by choosing the most compact encoding depending on the string, but in practice this wastes a lot of space. You'll double (or heaven forbid quadruple) your string size because of a single codepoint, and these codepoints are almost always a small percentage of the string. That's actually why UTF-16 and UTF-8 exist. It would have been better for strings to default to UTF-8, and to add an optional encoding so the programmer can specify what kind of encoding to use. As it is now, in order to use (for example) UTF-16 strings in Python you have to keep them around as bytes, decode them to a string, perform string operations, and reencode them to bytes again. Any benefit you get from using UTF-16 vanishes the moment you need to operate on it like a string, in other words. I get that the idea was to maintain indexing via codepoint, but (again) in practice that's not great: usually you want to index via grapheme -- if you want to index at all. A better solution is to allow programmers to specify string encoding and default it to UTF-8. From that, there's a clear path to everything you'd want to do.
- jstimpfle 9y ago
  > I get that the idea was to maintain indexing via codepoint, but (again) in practice that's not great: usually you want to index via grapheme -- if you want to index at all. I definitely need indexes, and I don't really care about graphemes. I actually have only a vague idea what that is. I write parsers typically by using a global string and lots of indices. The important thing for me is to be able to extract characters and slices at given positions, and to be able to say "parse error at line X character Y" where X and Y are helpful to the user most of the time. I would be absolutely fine with working in UTF-8 bytes only (and that would be faster I guess), but there would be a more pressing need to recompute character positions (as a code point or grapheme index) from byte offsets at times. There are more abstract parsing methods where parser subroutines are implemented in a position agnostic way, but I'm very happy with my simple method. If everything works on graphemes instead of code points (as I think does Perl6) I will be happy to use that, but it's not so important from a practical standpoint.
Animats 9y ago
Python took the obvious approach - they already had UTF-16 and UTF-32 builds, so this was just making that mechanism dynamic. Go and Rust expose UTF-8 at the byte level. This is something of a headache and may result in invalid string slices. It basically punts the problem back to the user. Here's an alternative: Use UTF-8 as the internal representation, but don't expose it to the user. If you're iterating over a string one rune or one grapheme at a time, the UTF-8 substructure is hidden from the user. Only if the user uses an explicit numeric subscript do you need to know a rune's position in string. When a request by subscript comes in, scan the string and build an index of rune subscript->byte position. This is expensive, but no worse than UTF-32 in space usage or expansion to UTF-32 in time. Optimizations: - Requests for s[0] to s[N], and s[-1] to s[-N], for small N, should be handled by working forwards or backwards through the UTF-8. (Yes, you can back up by rune in UTF-8. That's one of the neat features of the representation.) - Lookup functions such as "index" should return an opaque type which represents the position into that string. If such an object is used as a subscript, there's no need to build the index by rune. If you coerce this opaque type into an integer, the index table has to be built. Adding or subtracting small integers from this opaque type should be supported by working backwards or forwards in the string. - Regular expression processing has to be UTF-8 aware. It shouldn't need an index by rune. This would maintain Python's existing semantics while reducing memory consumption. Some performance measurement tool that finds all the places where an index by rune has to be built is useful. It's rare that you really need this, but sometimes you do.
- hsivonen 9y ago
  > Go and Rust expose UTF-8 at the byte level. This is something of a headache and may result in invalid string slices. It basically punts the problem back to the user. In Go, yes. In Rust, no. UTF-8 in Go is garbage in, garbage out. Rust, however, won't let you materialize an invalid &str without "unsafe".
  - hackits 9y ago
    > Go and Rust expose UTF-8 at the byte level. Or you can take the C++/C approach and have a character 1 byte, 2 bytes, or a multi-byte. It's a pain in the ass to constantly in C/C++ having to interface between two libraries that one decided to use char and another w_char!
deleted 9y ago
[deleted]
baby 9y ago
Question: if you read a file is there an algorithm that will make sure it is parsing the right encoding?
- pmyteh 9y ago
  Not reliably, no. You can detect if it's an invalid string according to the encoding you're currently using (value > 127 for ASCII, invalid surrogate pair for UTF-16) but there are lots of byte sequences that produce valid (but semantically meaningless) output in multiple encodings. To choose between them programmatically requires your algorithm to understand the meaning of the string as well as be able to decode it, which might be possible in limited domains, but is a very hard problem in general.
yarrel 9y ago
If this was phrased as a question it would be a trick one.
- btym 9y ago
  "How Python does Unicode: Poorly."
  - ubernostrum 9y ago
    I'd be curious to hear why you think that.
    - deleted 9y ago
      [deleted]
  - scrollaway 9y ago
    Python 2 maybe. Python 3 does Unicode wonderfully well; I miss it whenever I'm working with other languages.
    - Avernar 9y ago
      All Python 3 did was put a hard barrier between bytes and strings. That's it. Mising is all the grapheme handling that languages that do Unicode strings right have.
hprotagonist 9y ago
http://bit.ly/unipain http://bit.ly/unipain is my go-to reference whenever i get tripped up on what's going on with unicode in python. it is significantly more sane in python 3.3+.
mark-r 9y ago
I've always been curious on how this change in 3.3 impacts the C/C++ interface. I don't really know where to look it up, and since I haven't yet had to code a C++ library for Python I've had no burning need to answer the question.
- masklinn 9y ago
  https://www.python.org/dev/peps/pep-0393/ https://www.python.org/dev/peps/pep-0393/ has details down to C API changes related to the FSR implementation.
- ubernostrum 9y ago
  The Python C API grew some new functions and constants which are aware of what's going on and can tell you what encoding a particular Unicode object is using, read from/write to it, etc. The pre-3.3 APIs have a lot of deprecations in favor of the new API. If you want to use new API on a Unicode string created via old API, you have to use the new PyUnicode_READY() on it first.
wrs 9y ago
This was a pretty gutsy move on Python's part. The presence of a single emoji in an English string will blow up memory usage for the whole string by 4x. And because graphemes aren't 1:1 to code points, the O(1) indexing and length operations you bought with that trade-off will still confuse people who don't understand Unicode.
- ubernostrum 9y ago
  As I said in the article, I think the overhead of adding yet more weirdness in the form of quirks of the internal encoding (which could vary according to how the Python interpreter was compiled!) is a bad thing to do on top of how much people seem to struggle mentally just to get Unicode all on its own. Though I also think the struggle is mostly due to people being stuck in an everything-is-like-ASCII mindset, and though I didn't get into that, it's one big reason why I think UTF-8 is generally the wrong way to expose Unicode to a programmer, since it lets them think they can keep that cherished "one byte == one character" assumption right up until something breaks at 2AM on a weekend. Personally I'd like everyone to just actually learn at least the things about Unicode that I went into here (such as why "one code point == one character" is a wrong assumption), and I think that'd alleviate a lot of the pain. I also avoided talking much about normalization, because too many people hear about it and decide they can just normalize to NFKC and go back to assuming code point/character equivalence post-normalization.
  - ekidd 9y ago
    > it's one big reason why I think UTF-8 is generally the wrong way to expose Unicode to a programmer, since it lets them think they can keep that cherished "one byte == one character" assumption right up until something breaks at 2AM on a weekend. Unfortunately, as long as you believe that you can index into a Unicode string, your code is going to break. The only question is how soon. I actually like UTF-8 because it will break very quickly, and force the programmer to do the right thing. The first time you hit é or € or ️an emoji, you'll have a multibyte character, and you'll need to deal with it. All the other options will also break, but later on: - If you use UTF-16, then é and € will work, but emoji will still result in surrogate pairs. - If you use a 4-byte representation, then you'll be able to treat most emoji as single characters. But then somebody will build é from two separate code points as "e + U+0301 COMBINING ACUTE ACCENT", or you'll run into a flag or skin color emoji, and once again, you're back at square zero. You can't really index Unicode characters like ASCII strings. Written language is just too weird for that. But if you use UTF-8 (with a good API), then you'll be forced to accept that "str[3]" is hopeless very quickly. It helps a lot if your language has separate types for "byte" and "Unicode codepoint", however, so you can't accidentally treat a single byte as a character.

165 points by scw 9y ago

Unicode is a horrible scam, the worst thing to happen to digital language representation. This is all just so much turd polishing. (Also, that explanation of UTF-8 is crap. UTF-8 is beautiful quite apart from its utility, but you'd hardly know it from the article.) I've said it before: Unicode is a conflation of a good idea and an impossible idea. The good idea is a standard mapping from numbers to little pictures. That's all ASCII was. The impossible idea is a digital code for every way humans write. It's a form of digital cultural imperialism. Unicode Consortium et. al. are absurdly arrogant.

simonh 9y ago
Critical rants that don't suggest a better alternative, or describe what a better alternative might look like even in outline, are rarely informative or persuasive.
- carapace 9y ago
  Step One: Admit there's a problem. I heard, "Tell me more about what you think would be better." Here goes: For written languages that are well-served by a simple sequence of symbols (English, etc.) there is no problem: a catalog of the mappings from numbers to pictures is fine is all that is required. Put them in a sequence (anoint UTF-8 as the One True Encoding) and you're good-to-go. For languages that are NOT well-served by this simple abstraction the first thing to do (assuming you have the requisite breadth and depth of linguistic knowledge) is to figure out simple formal systems that do abstract the languages in question. Then determine equivalence classes and standardize the formal systems. Let the structure of the language abstraction be a "first-class" entity that has reference implementations. Instead of adding weird modifiers and other dynamic behavior to the code, let them be actual simple DSLs whose output is the proper graphics. Human languages are like a superset of what computers can represent. Here's the Unicode Standard[1] on Arabic: > The basic set of Arabic letters is well defined. Each letter receives only one Unicode character value in the basic Arabic block, no matter how many different contextual appearances it may exhibit in text. Each Arabic letter in the Unicode Standard may be said to represent the inherent semantic identity of the letter. A word is spelled as a sequence of these letters. The representative glyph shown in the Unicode character chart for an Arabic letter is usually the form of the letter when standing by itself. It is simply used to distinguish and identify the character in the code charts and does not restrict the glyphs used to represent it. They baldly admit that Unicode is not good for drawing Arabic. I find the phrase "the inherent semantic identity of the letter" to be particularly rich. It's nearly mysticism. If it is inconvenient to try to represent a language in terms of a sequence of symbols, then let's represent it as a (simple) program that renders the language correctly, which allows us to shoehorn non-linear behavior into a sequence of symbols. If you think about it, this is already what Unicode is doing with modifiers and such. If you read further in the Unicode Standard doc I quoted above you'll see that they basically do create a kind of DSL for dealing with Arabic. I'm saying: make it explicit. Don't try to pretend that Unicode is one big standard for human languages. Admit that the "space" of writing systems is way bigger and more involved than Latin et. al. Study the problem of representing writing in a computer as a first-class issue. Publish reference implementations of code that can handle each kind of writing system along with the catalog of numbered pictures. From the Unicode Standard again: > The Arabic script is cursive, even in its printed form. As a result, the same letter may be written in different forms depending on how it joins with its neighbors. Vow-els and various other marks may be written as combining marks called tashkil, which are applied to consonantal base letters. In normal writing, however, these marks are omitted. Computer systems that are adapted to English are not going to work for Arabic. I'd love to use a language simpler than PostScript to draw Arabic! Unicode strings are not that language. Consider the "Base-4 fractions in Telugu" https://blog.plover.com/math/telugu.html https://blog.plover.com/math/telugu.html The fact that we have a way to represent the graphics ౦౼౽౾౸౹౺౻ is great! But any software that wants to use them properly will require some code to translate to and from numbers in the computer to Telugu sequences of those graphics. Let that be part of "Unicode" and I'll shut up. In the meantime, I feel like it's a huge scam and a kind of cultural imperialism from us hacker types to the folks who are late to the party and for whom ASCII++ isn't going to really cut it. To sum up: I think the thing that replaces Unicode for dealing with human languages in digital form should: A.) Be created by linguists with help from computer folks, not by computer folks with some nagging from linguists (apologies to the linguist/computer folk who actually did the stuff.) B.) We should clearly state the problems first: What are the ways that human language are written down? C.) Write specific DSLs for each kind of writing. Publish reference implementations. I think that's it. Are you informed? Persuaded even? Entertained at least? ;-) [1] http://www.unicode.org/versions/Unicode9.0.0/ch09.pdf http://www.unicode.org/versions/Unicode9.0.0/ch09.pdf
  - ubershmekel 9y ago
    > Unicode is a horrible scam, the worst thing to happen to digital language representation. > They baldly admit that Unicode is not good for drawing Arabic > Consider the "Base-4 fractions in Telugu" [...] any software that wants to use them properly will require some code to translate to and from numbers in the computer to Telugu sequences of those graphics. Written language is hard to represent, encode and draw. You admit that Unicode&utf8 got 2 out of 3 right yet you call it a scam. Your complaint is a scam and a horrible trollism.
    - carapace 9y ago
      When I'm trolling you'll know it. I have a point, I believe it's a good point, and I'm making it. For languages that can be represented as a sequence of little pictures Unicode is a little better than ASCII. For the rest, it's a scam: We tell people that we have a way of dealing with human languages in computers but it's half-baked, born in ignorance, and all the grotty details are papered over, but you can write PIZZA SLICE or POOP now, so fuck it, ship it. Represent: "Astral Plane"? What does that have to do with a standard catalog mapping numbers to pictures? I feel Unicode messes that up. Encoding: UTF-8 is near perfect, 'nuff said. Ken Thompson and Rob Pike doing their thing. Drawing: Doesn't even begin to touch it really. Unicode is a nasty little black hole that's sucking up time and other resources and not really solving the problem.
      - mixmastamyk 9y ago
        I think that's too negative. Is Unicode perfect? Of course not, but it's the best we've got for now. Just as Morse, Baudot, or ASCII were the best approximations at one point in time. It's a hard problem and will take decades to for the right solutions/implementations to present themselves. Surely one day there will be an improved successor to Unicode. Things are a lot better than they were even ten years ago, however.
        carapace 9y ago
        Yeah, sorry, I was pretty cranky last night. Please see my reply to simonh in this thread a few minutes ago. (I'm basically agreeing with you.)
ubernostrum 9y ago
that explanation of UTF-8 is crap. UTF-8 is beautiful quite apart from its utility, but you'd hardly know it from the article My goal was not to judge UTF-8 aesthetically, but to explain how it works and point out that it's a variable-width encoding which emphasizes its compatibility with ASCII for strings containing only code points <= U+007F. Unicode Consortium et. al. are absurdly arrogant. I would agree that Unicode as it exists today involves some historical and historic bad decisions. But again, staying off value judgments with respect to Unicode itself since the point of the article was to explain how Python now handles it internally.
- carapace 9y ago
  Oh, Hi there. Apologies for being cranky. You did a great job explaining how Python now handles Unicode! To me it was strange reading about UTF-32 first and then getting to UTF-8 from that context. It seemed to obscure the coolth and beauty of the format. Overall a great article, sorry again for being so negative.
  - ubernostrum 9y ago
    That section was written for people who know little to nothing about Unicode and the ways Unicode can be encoded to bytes. So it starts with the obvious approach -- just spit out a sequence of bytes whose integer values are the code points, which is near enough as makes no difference to how UTF-32 works -- then introduces variable-width encoding through the history of UCS-2 and UTF-16, then gets to UTF-8 and what motivated it. The advantages/disadvantages of the various encodings is something that could eat up several pieces just as long as the entire post, and for fun I'd probably throw in weird stuff like the attempt to do EBCDIC-compatible UTF instead of ASCII-compatible, etc.
    - WorldMaker 9y ago
      Someone should write up EBCDIC-based UTF as an RFC. I'm sure that there's at least one COBOL programmer out there that has been waiting for that for decades. ETA: Mostly a joke, but it would also fit right in with things like WTF-8 (https://simonsapin.github.io/wtf-8/ https://simonsapin.github.io/wtf-8/)

Avernar 9y ago

I'm not a fan of how Python 3 stores Unicode strings internally. In my opinion they should have went with UTF-8. The extra scanning and conversion puts more preassure on the processor and caches under load. I agree that Python 2's Unicode handling is broken. That's why I just stored UTF-8 in a normal string and avoided the whole mess. The only thing I have to do is validate any input from the outside world is really UTF-8.

ubernostrum 9y ago
Since the high-level API is supposed to let you treat a string as a sequence of code points, a correct implementation (which Python didn't have until 3.3!) would've imposed the overhead of conversion to something resembling a fixed-width encoding whenever a programmer invoked certain operations. And the vast majority of strings in real-world Python contain only code points also present in latin-1, which means they can be stored in one byte per code point with this approach. And for strings which can't be stored in one byte per code point, you were similarly going to pay the price sooner or later.
- Avernar 9y ago
  > Since the high-level API is supposed to let you treat a string as a sequence of code points, I disagree with that premise. It should operate on grapheme clusters. Operating on code points falls into the same trap as operating on bytes. > a correct implementation (which Python didn't have until 3.3!) would've imposed the overhead of conversion to something resembling a fixed-width encoding whenever a programmer invoked certain operations. Those operations should have been removed. Indexing is the big one that needs fixed width internal representation for speed. Code could have been rewritten to not require indexing. But mechanical translation from Python 2 to 3 was a goal and because of that they couldn't radically change the unicode API for the better. > And the vast majority of strings in real-world Python contain only code points also present in latin-1, which means they can be stored in one byte per code point with this approach. And for strings which can't be stored in one byte per code point, you were similarly going to pay the price sooner or later. You're going to pay the price for 4 byte per codepoint strings quite often. A single emoji will blow up a latin-1 string to 4 times the size.

dguaraglia 9y ago

My favorite story about Python's handling of Unicode was when one of my coworkers did a hotfix for our Python website, wrote tests, confirmed everything worked as expected... but right before committing and pushing to production wrote a comment like: # Apparently we expect the field to be in this format ¯\_(ツ)_/¯ Right above the code he'd just fixed. Of course, the moment we pushed the update it brought production down, because the Python interpreter doesn't understand Unicode in source files unless you specify which encoding you are using. After that, "¯\_(ツ)_/¯" became a synonym for his name on our HipChat server, heh.

ubernostrum 9y ago
This would be the case in Python 2, where source code files are assumed to be ASCII-encoded unless there's an encoding comment at the top of the file. In Python 3, source code files are assumed to be UTF-8.
- dguaraglia 9y ago
  Correct, this was a codebase that still had some Pylons (gasp! Not even Pyramid, but legit Pylons) code.
- ninkendo 9y ago
  Interesting that Python 2 couldn't fix that in a hotfix/point release... UTF-8 is backwards compatible with ASCII so it shouldn't break anything if source started being interpreted as UTF8. I'd be curious to see what their reasoning is.
  - dguaraglia 9y ago
    I would imagine Python's approach to introducing new language features had a lot to do with it. Having to go through the PEP system takes some time, and changes like these tend to be reserved for minor-version releases. All in all, I love the PEP system, it's such an open concept and I've been surprised by the amount of quality proposals that get implemented. Wish Go had something like it.
  - 9y ago

camgunz 9y ago

UCS-4 is essentially never the right choice. It wastes space and thus messes up your cache. UCS-2 can be the right choice if the language you're encoding uses a lot of non-Latin glyphs (i.e. East Asian languages) but suffers from the same problem as UCS-2. UTF-8 is a good default: for most strings it's very compact, and for strings with a lot of multibyte codepoints it doesn't compare too unfavorably with UTF-16. Python 3 tried to have its cake and eat it too by choosing the most compact encoding depending on the string, but in practice this wastes a lot of space. You'll double (or heaven forbid quadruple) your string size because of a single codepoint, and these codepoints are almost always a small percentage of the string. That's actually why UTF-16 and UTF-8 exist. It would have been better for strings to default to UTF-8, and to add an optional encoding so the programmer can specify what kind of encoding to use. As it is now, in order to use (for example) UTF-16 strings in Python you have to keep them around as bytes, decode them to a string, perform string operations, and reencode them to bytes again. Any benefit you get from using UTF-16 vanishes the moment you need to operate on it like a string, in other words. I get that the idea was to maintain indexing via codepoint, but (again) in practice that's not great: usually you want to index via grapheme -- if you want to index at all. A better solution is to allow programmers to specify string encoding and default it to UTF-8. From that, there's a clear path to everything you'd want to do.

jstimpfle 9y ago
> I get that the idea was to maintain indexing via codepoint, but (again) in practice that's not great: usually you want to index via grapheme -- if you want to index at all. I definitely need indexes, and I don't really care about graphemes. I actually have only a vague idea what that is. I write parsers typically by using a global string and lots of indices. The important thing for me is to be able to extract characters and slices at given positions, and to be able to say "parse error at line X character Y" where X and Y are helpful to the user most of the time. I would be absolutely fine with working in UTF-8 bytes only (and that would be faster I guess), but there would be a more pressing need to recompute character positions (as a code point or grapheme index) from byte offsets at times. There are more abstract parsing methods where parser subroutines are implemented in a position agnostic way, but I'm very happy with my simple method. If everything works on graphemes instead of code points (as I think does Perl6) I will be happy to use that, but it's not so important from a practical standpoint.

Animats 9y ago

Python took the obvious approach - they already had UTF-16 and UTF-32 builds, so this was just making that mechanism dynamic. Go and Rust expose UTF-8 at the byte level. This is something of a headache and may result in invalid string slices. It basically punts the problem back to the user. Here's an alternative: Use UTF-8 as the internal representation, but don't expose it to the user. If you're iterating over a string one rune or one grapheme at a time, the UTF-8 substructure is hidden from the user. Only if the user uses an explicit numeric subscript do you need to know a rune's position in string. When a request by subscript comes in, scan the string and build an index of rune subscript->byte position. This is expensive, but no worse than UTF-32 in space usage or expansion to UTF-32 in time. Optimizations: - Requests for s[0] to s[N], and s[-1] to s[-N], for small N, should be handled by working forwards or backwards through the UTF-8. (Yes, you can back up by rune in UTF-8. That's one of the neat features of the representation.) - Lookup functions such as "index" should return an opaque type which represents the position into that string. If such an object is used as a subscript, there's no need to build the index by rune. If you coerce this opaque type into an integer, the index table has to be built. Adding or subtracting small integers from this opaque type should be supported by working backwards or forwards in the string. - Regular expression processing has to be UTF-8 aware. It shouldn't need an index by rune. This would maintain Python's existing semantics while reducing memory consumption. Some performance measurement tool that finds all the places where an index by rune has to be built is useful. It's rare that you really need this, but sometimes you do.

hsivonen 9y ago
> Go and Rust expose UTF-8 at the byte level. This is something of a headache and may result in invalid string slices. It basically punts the problem back to the user. In Go, yes. In Rust, no. UTF-8 in Go is garbage in, garbage out. Rust, however, won't let you materialize an invalid &str without "unsafe".
- hackits 9y ago
  > Go and Rust expose UTF-8 at the byte level. Or you can take the C++/C approach and have a character 1 byte, 2 bytes, or a multi-byte. It's a pain in the ass to constantly in C/C++ having to interface between two libraries that one decided to use char and another w_char!

deleted 9y ago

[deleted]

baby 9y ago

Question: if you read a file is there an algorithm that will make sure it is parsing the right encoding?

pmyteh 9y ago
Not reliably, no. You can detect if it's an invalid string according to the encoding you're currently using (value > 127 for ASCII, invalid surrogate pair for UTF-16) but there are lots of byte sequences that produce valid (but semantically meaningless) output in multiple encodings. To choose between them programmatically requires your algorithm to understand the meaning of the string as well as be able to decode it, which might be possible in limited domains, but is a very hard problem in general.

yarrel 9y ago

If this was phrased as a question it would be a trick one.

btym 9y ago
"How Python does Unicode: Poorly."
- ubernostrum 9y ago
  I'd be curious to hear why you think that.
  - deleted 9y ago
    [deleted]
- scrollaway 9y ago
  Python 2 maybe. Python 3 does Unicode wonderfully well; I miss it whenever I'm working with other languages.
  - Avernar 9y ago
    All Python 3 did was put a hard barrier between bytes and strings. That's it. Mising is all the grapheme handling that languages that do Unicode strings right have.

hprotagonist 9y ago

http://bit.ly/unipain http://bit.ly/unipain is my go-to reference whenever i get tripped up on what's going on with unicode in python. it is significantly more sane in python 3.3+.

mark-r 9y ago

I've always been curious on how this change in 3.3 impacts the C/C++ interface. I don't really know where to look it up, and since I haven't yet had to code a C++ library for Python I've had no burning need to answer the question.

masklinn 9y ago
https://www.python.org/dev/peps/pep-0393/ https://www.python.org/dev/peps/pep-0393/ has details down to C API changes related to the FSR implementation.
ubernostrum 9y ago
The Python C API grew some new functions and constants which are aware of what's going on and can tell you what encoding a particular Unicode object is using, read from/write to it, etc. The pre-3.3 APIs have a lot of deprecations in favor of the new API. If you want to use new API on a Unicode string created via old API, you have to use the new PyUnicode_READY() on it first.

wrs 9y ago

This was a pretty gutsy move on Python's part. The presence of a single emoji in an English string will blow up memory usage for the whole string by 4x. And because graphemes aren't 1:1 to code points, the O(1) indexing and length operations you bought with that trade-off will still confuse people who don't understand Unicode.

ubernostrum 9y ago
As I said in the article, I think the overhead of adding yet more weirdness in the form of quirks of the internal encoding (which could vary according to how the Python interpreter was compiled!) is a bad thing to do on top of how much people seem to struggle mentally just to get Unicode all on its own. Though I also think the struggle is mostly due to people being stuck in an everything-is-like-ASCII mindset, and though I didn't get into that, it's one big reason why I think UTF-8 is generally the wrong way to expose Unicode to a programmer, since it lets them think they can keep that cherished "one byte == one character" assumption right up until something breaks at 2AM on a weekend. Personally I'd like everyone to just actually learn at least the things about Unicode that I went into here (such as why "one code point == one character" is a wrong assumption), and I think that'd alleviate a lot of the pain. I also avoided talking much about normalization, because too many people hear about it and decide they can just normalize to NFKC and go back to assuming code point/character equivalence post-normalization.
- ekidd 9y ago
  > it's one big reason why I think UTF-8 is generally the wrong way to expose Unicode to a programmer, since it lets them think they can keep that cherished "one byte == one character" assumption right up until something breaks at 2AM on a weekend. Unfortunately, as long as you believe that you can index into a Unicode string, your code is going to break. The only question is how soon. I actually like UTF-8 because it will break very quickly, and force the programmer to do the right thing. The first time you hit é or € or ️an emoji, you'll have a multibyte character, and you'll need to deal with it. All the other options will also break, but later on: - If you use UTF-16, then é and € will work, but emoji will still result in surrogate pairs. - If you use a 4-byte representation, then you'll be able to treat most emoji as single characters. But then somebody will build é from two separate code points as "e + U+0301 COMBINING ACUTE ACCENT", or you'll run into a flag or skin color emoji, and once again, you're back at square zero. You can't really index Unicode characters like ASCII strings. Written language is just too weird for that. But if you use UTF-8 (with a good API), then you'll be forced to accept that "str[3]" is hopeless very quickly. It helps a lot if your language has separate types for "byte" and "Unicode codepoint", however, so you can't accidentally treat a single byte as a character.