27 ms·
Unicode is a horrible scam, the worst thing to happen to digital language representation. This is all just so much turd polishing. (Also, that explanation of
by carapace 9y ago
Unicode is a horrible scam, the worst thing to happen to digital language representation. This is all just so much turd polishing.
(Also, that explanation of UTF-8 is crap. UTF-8 is beautiful quite apart from its utility, but you'd hardly know it from the article.)
I've said it before: Unicode is a conflation of a good idea and an impossible idea. The good idea is a standard mapping from numbers to little pictures. That's all ASCII was. The impossible idea is a digital code for every way humans write. It's a form of digital cultural imperialism.
Unicode Consortium et. al. are absurdly arrogant.
- simonh 9y agoCritical rants that don't suggest a better alternative, or describe what a better alternative might look like even in outline, are rarely informative or persuasive.
- carapace 9y agoStep One: Admit there's a problem. I heard, "Tell me more about what you think would be better." Here goes: For written languages that are well-served by a simple sequence of symbols (English, etc.) there is no problem: a catalog of the mappings from numbers to pictures is fine is all that is required. Put them in a sequence (anoint UTF-8 as the One True Encoding) and you're good-to-go. For languages that are NOT well-served by this simple abstraction the first thing to do (assuming you have the requisite breadth and depth of linguistic knowledge) is to figure out simple formal systems that do abstract the languages in question. Then determine equivalence classes and standardize the formal systems. Let the structure of the language abstraction be a "first-class" entity that has reference implementations. Instead of adding weird modifiers and other dynamic behavior to the code, let them be actual simple DSLs whose output is the proper graphics. Human languages are like a superset of what computers can represent. Here's the Unicode Standard[1] on Arabic: > The basic set of Arabic letters is well defined. Each letter receives only one Unicode character value in the basic Arabic block, no matter how many different contextual appearances it may exhibit in text. Each Arabic letter in the Unicode Standard may be said to represent the inherent semantic identity of the letter. A word is spelled as a sequence of these letters. The representative glyph shown in the Unicode character chart for an Arabic letter is usually the form of the letter when standing by itself. It is simply used to distinguish and identify the character in the code charts and does not restrict the glyphs used to represent it. They baldly admit that Unicode is not good for drawing Arabic. I find the phrase "the inherent semantic identity of the letter" to be particularly rich. It's nearly mysticism. If it is inconvenient to try to represent a language in terms of a sequence of symbols, then let's represent it as a (simple) program that renders the language correctly, which allows us to shoehorn non-linear behavior into a sequence of symbols. If you think about it, this is already what Unicode is doing with modifiers and such. If you read further in the Unicode Standard doc I quoted above you'll see that they basically do create a kind of DSL for dealing with Arabic. I'm saying: make it explicit. Don't try to pretend that Unicode is one big standard for human languages. Admit that the "space" of writing systems is way bigger and more involved than Latin et. al. Study the problem of representing writing in a computer as a first-class issue. Publish reference implementations of code that can handle each kind of writing system along with the catalog of numbered pictures. From the Unicode Standard again: > The Arabic script is cursive, even in its printed form. As a result, the same letter may be written in different forms depending on how it joins with its neighbors. Vow-els and various other marks may be written as combining marks called tashkil, which are applied to consonantal base letters. In normal writing, however, these marks are omitted. Computer systems that are adapted to English are not going to work for Arabic. I'd love to use a language simpler than PostScript to draw Arabic! Unicode strings are not that language. Consider the "Base-4 fractions in Telugu" https://blog.plover.com/math/telugu.html https://blog.plover.com/math/telugu.html The fact that we have a way to represent the graphics ౦౼౽౾౸౹౺౻ is great! But any software that wants to use them properly will require some code to translate to and from numbers in the computer to Telugu sequences of those graphics. Let that be part of "Unicode" and I'll shut up. In the meantime, I feel like it's a huge scam and a kind of cultural imperialism from us hacker types to the folks who are late to the party and for whom ASCII++ isn't going to really cut it. To sum up: I think the thing that replaces Unicode for dealing with human languages in digital form should: A.) Be created by linguists with help from computer folks, not by computer folks with some nagging from linguists (apologies to the linguist/computer folk who actually did the stuff.) B.) We should clearly state the problems first: What are the ways that human language are written down? C.) Write specific DSLs for each kind of writing. Publish reference implementations. I think that's it. Are you informed? Persuaded even? Entertained at least? ;-) [1] http://www.unicode.org/versions/Unicode9.0.0/ch09.pdf http://www.unicode.org/versions/Unicode9.0.0/ch09.pdf
- ubershmekel 9y ago> Unicode is a horrible scam, the worst thing to happen to digital language representation. > They baldly admit that Unicode is not good for drawing Arabic > Consider the "Base-4 fractions in Telugu" [...] any software that wants to use them properly will require some code to translate to and from numbers in the computer to Telugu sequences of those graphics. Written language is hard to represent, encode and draw. You admit that Unicode&utf8 got 2 out of 3 right yet you call it a scam. Your complaint is a scam and a horrible trollism.
- carapace 9y agoWhen I'm trolling you'll know it. I have a point, I believe it's a good point, and I'm making it. For languages that can be represented as a sequence of little pictures Unicode is a little better than ASCII. For the rest, it's a scam: We tell people that we have a way of dealing with human languages in computers but it's half-baked, born in ignorance, and all the grotty details are papered over, but you can write PIZZA SLICE or POOP now, so fuck it, ship it. Represent: "Astral Plane"? What does that have to do with a standard catalog mapping numbers to pictures? I feel Unicode messes that up. Encoding: UTF-8 is near perfect, 'nuff said. Ken Thompson and Rob Pike doing their thing. Drawing: Doesn't even begin to touch it really. Unicode is a nasty little black hole that's sucking up time and other resources and not really solving the problem.
- mixmastamyk 9y agoI think that's too negative. Is Unicode perfect? Of course not, but it's the best we've got for now. Just as Morse, Baudot, or ASCII were the best approximations at one point in time. It's a hard problem and will take decades to for the right solutions/implementations to present themselves. Surely one day there will be an improved successor to Unicode. Things are a lot better than they were even ten years ago, however.
- carapace 9y agoYeah, sorry, I was pretty cranky last night. Please see my reply to simonh in this thread a few minutes ago. (I'm basically agreeing with you.)
- ubernostrum 9y agothat explanation of UTF-8 is crap. UTF-8 is beautiful quite apart from its utility, but you'd hardly know it from the article My goal was not to judge UTF-8 aesthetically, but to explain how it works and point out that it's a variable-width encoding which emphasizes its compatibility with ASCII for strings containing only code points <= U+007F. Unicode Consortium et. al. are absurdly arrogant. I would agree that Unicode as it exists today involves some historical and historic bad decisions. But again, staying off value judgments with respect to Unicode itself since the point of the article was to explain how Python now handles it internally.
- carapace 9y agoOh, Hi there. Apologies for being cranky. You did a great job explaining how Python now handles Unicode! To me it was strange reading about UTF-32 first and then getting to UTF-8 from that context. It seemed to obscure the coolth and beauty of the format. Overall a great article, sorry again for being so negative.
- ubernostrum 9y agoThat section was written for people who know little to nothing about Unicode and the ways Unicode can be encoded to bytes. So it starts with the obvious approach -- just spit out a sequence of bytes whose integer values are the code points, which is near enough as makes no difference to how UTF-32 works -- then introduces variable-width encoding through the history of UCS-2 and UTF-16, then gets to UTF-8 and what motivated it. The advantages/disadvantages of the various encodings is something that could eat up several pieces just as long as the entire post, and for fun I'd probably throw in weird stuff like the attempt to do EBCDIC-compatible UTF instead of ASCII-compatible, etc.
- WorldMaker 9y agoSomeone should write up EBCDIC-based UTF as an RFC. I'm sure that there's at least one COBOL programmer out there that has been waiting for that for decades. ETA: Mostly a joke, but it would also fit right in with things like WTF-8 (https://simonsapin.github.io/wtf-8/ https://simonsapin.github.io/wtf-8/)