5 ms·
To be honest, I blame Unicode. Why allow different representations for the same character, and then provide a normalized form anyway, except it's not one normal
by nottorp 9y ago
To be honest, I blame Unicode. Why allow different representations for the same character, and then provide a normalized form anyway, except it's not one normalized form but several? Sounds like job security to me.
- Piskvorrr 9y agoBlame the encodings that came before it: Latin-1 and Windows-1252, and all the other ones. Hindsight is always 20/20, y'know.
- jrimbault 9y agoI'm just being sarcastic: should we also blame the computers of old for not being able to handle more than 255 characters ?
- Piskvorrr 9y agoNot really. In a barely-networked and severely resource-constrained world (kilobytes of memory, not gigabytes), this decision made sense. Shift happens.
- cryptonector 9y agoNo, just developers. Of course, the use of bytes for characters goes back a long time, to times when computers had small memories and disk (and other) storage capacities. And to even before then, to the days of telexes and typewriters. It's completely understandable. But UTF-8 is genius, which is why we use it. Incidentally, ASCII was actually a multi-byte codeset... since one could combine most lower-case characters with BS (backspace) and overstrike with apostrophe, backtick, tilde, comma (for cedilles), and double-quotes (for umlauts), or with the same char for bold, or with underscore for underline. nroff(1) still uses this for bold and underscore, no?
- thaumasiotes 9y agoI've always wondered what the concept of a "backspace" character was supposed to mean. (I tried printing it to erase a previously printed character, but that doesn't work.) I guess it's another relic of the idea that computer output goes to a printer rather than a display?
- Piskvorrr 9y agoIndeed it is. Or rather, output goes to a remote printer. Morse code - teletype (backspace appears here) - mainframe - ASCII - Unicode (you are here).
- cryptonector 9y agoYes! Terminal vendors had to explicitly support this in terminals, though maybe not just because of printers but because it's actually quite useful. Note that BS only produces overstrike when followed by certain characters (which we might term "combining" for the fun of it), while most will just change the character at that location. A tty spinner is just |BS/BS-BS\BS|BS... with some delay between each -- no overstrike there.
- nottorp 9y agoI don't remember those having multiple representations for the same character in the same encoding or normal forms...
- Piskvorrr 9y agoIndeed not: there is an almost infinite number of things that Latin-1 did not have; e.g. any characters beyond Western Europe. While keeping the 128-256 block compatible was a part of early Unicode (hence the "one-glyph" é), having composed characters was a Unicode primary design goal (hence e and the composing accent). A pure Unicode implementation would have been better, maybe; what we have instead is one that has gained mass adoption. https://en.wikipedia.org/wiki/Latin-1_Supplement_(Unicode_block) https://en.wikipedia.org/wiki/Latin-1_Supplement_(Unicode_bl...
- nottorp 9y agoWell, composed characters may or may not have been a good idea, although they're a pain for text search if you ask me. But why oh why did they have to specify FOUR normalization forms? Ref: http://www.unicode.org/reports/tr15/#Norm_Forms http://www.unicode.org/reports/tr15/#Norm_Forms
- Piskvorrr 9y ago¯\_(ツ)_/¯
- TazeTSchnitzel 9y agoBecause you need a normalisation form fit for your particular purpose.
- the_mitsuhiko 9y ago> But why oh why did they have to specify FOUR normalization forms? Ignoring compatibility normalizations there are really only two and neither of those two you can remove. So that's a bit of an odd question. If you want to remove the compatibility forms that's fair but nobody is force to use them. You can consider them "partially normalized" or "not normalized at all" for all intents and purposes
- userbinator 9y agoMore like a typical outcome of design-by-committee.
- the_mitsuhiko 9y ago> To be honest, I blame Unicode. Why allow different representations for the same character, and then provide a normalized form anyway, except it's not one normalized form but several? That's not really the issue here. Even if you can avoid normalization in some languages the problem will come back in others. For instance what are you going to do about invisible characters, control characters or more? Traditionally software attempted to just ignore the problems and let it blow up in other places. For some fun issues just try whitespace and bell characters in filenames and navigate around in your shell. Normalization is just one of many issues with filenames.
- TazeTSchnitzel 9y agoCompatibility and efficiency are two big reasons why you would want to have composed forms like e.g. the single codepoint for é: legacy encodings also have a single codepoint for it, and it's more efficient to represent it with one codepoint than with two. So, we need composed characters, but surely combining characters aren't needed, then? Well, no, because it's unreasonable to include a codepoint for every single combination, especially once you have multiple diacritics on a single character, for example. Also, compatibility factors in here too, because legacy encodings likewise have combining characters.
- Ono-Sendai 9y agoCombining characters seems like a pretty stupid thing to have in a character set/encoding. Does anyone know how many combined characters are actually needed?
- cryptonector 9y agoMore than there are pre-composed codepoint assignments for.
- cryptonector 9y agoAnd also: scripts evolve. People may well start combining various diacritical marks in ways not done before. Also, it can be useful to decompose in some cases. For example, it's useful for removing diacritical marks, which can be useful for fuzzy searches. Hangul is really phonetic, not syllabic, so there it's the reverse: decomposed forms are most useful, except when you need to think in syllables.
- cryptonector 9y agoIt's not Unicode's fault. Unicode took a terrible situation (lots of complex scripts) and did the best it could to give us a unified way of representing text in all scripts.