3 ms·

Blame the encodings that came before it: Latin-1 and Windows-1252, and all the other ones. Hindsight is always 20/20, y'know.

jrimbault 9y ago
I'm just being sarcastic: should we also blame the computers of old for not being able to handle more than 255 characters ?
- Piskvorrr 9y ago
  Not really. In a barely-networked and severely resource-constrained world (kilobytes of memory, not gigabytes), this decision made sense. Shift happens.
- cryptonector 9y ago
  No, just developers. Of course, the use of bytes for characters goes back a long time, to times when computers had small memories and disk (and other) storage capacities. And to even before then, to the days of telexes and typewriters. It's completely understandable. But UTF-8 is genius, which is why we use it. Incidentally, ASCII was actually a multi-byte codeset... since one could combine most lower-case characters with BS (backspace) and overstrike with apostrophe, backtick, tilde, comma (for cedilles), and double-quotes (for umlauts), or with the same char for bold, or with underscore for underline. nroff(1) still uses this for bold and underscore, no?
  - thaumasiotes 9y ago
    I've always wondered what the concept of a "backspace" character was supposed to mean. (I tried printing it to erase a previously printed character, but that doesn't work.) I guess it's another relic of the idea that computer output goes to a printer rather than a display?
    - Piskvorrr 9y ago
      Indeed it is. Or rather, output goes to a remote printer. Morse code - teletype (backspace appears here) - mainframe - ASCII - Unicode (you are here).
    - cryptonector 9y ago
      Yes! Terminal vendors had to explicitly support this in terminals, though maybe not just because of printers but because it's actually quite useful. Note that BS only produces overstrike when followed by certain characters (which we might term "combining" for the fun of it), while most will just change the character at that location. A tty spinner is just |BS/BS-BS\BS|BS... with some delay between each -- no overstrike there.
nottorp 9y ago
I don't remember those having multiple representations for the same character in the same encoding or normal forms...
- Piskvorrr 9y ago
  Indeed not: there is an almost infinite number of things that Latin-1 did not have; e.g. any characters beyond Western Europe. While keeping the 128-256 block compatible was a part of early Unicode (hence the "one-glyph" é), having composed characters was a Unicode primary design goal (hence e and the composing accent). A pure Unicode implementation would have been better, maybe; what we have instead is one that has gained mass adoption. https://en.wikipedia.org/wiki/Latin-1_Supplement_(Unicode_block) https://en.wikipedia.org/wiki/Latin-1_Supplement_(Unicode_bl...
  - nottorp 9y ago
    Well, composed characters may or may not have been a good idea, although they're a pain for text search if you ask me. But why oh why did they have to specify FOUR normalization forms? Ref: http://www.unicode.org/reports/tr15/#Norm_Forms http://www.unicode.org/reports/tr15/#Norm_Forms
    - Piskvorrr 9y ago
      ¯\_(ツ)_/¯
    - TazeTSchnitzel 9y ago
      Because you need a normalisation form fit for your particular purpose.
    - the_mitsuhiko 9y ago
      > But why oh why did they have to specify FOUR normalization forms? Ignoring compatibility normalizations there are really only two and neither of those two you can remove. So that's a bit of an odd question. If you want to remove the compatibility forms that's fair but nobody is force to use them. You can consider them "partially normalized" or "not normalized at all" for all intents and purposes

jrimbault 9y ago

I'm just being sarcastic: should we also blame the computers of old for not being able to handle more than 255 characters ?

Piskvorrr 9y ago
Not really. In a barely-networked and severely resource-constrained world (kilobytes of memory, not gigabytes), this decision made sense. Shift happens.
cryptonector 9y ago
No, just developers. Of course, the use of bytes for characters goes back a long time, to times when computers had small memories and disk (and other) storage capacities. And to even before then, to the days of telexes and typewriters. It's completely understandable. But UTF-8 is genius, which is why we use it. Incidentally, ASCII was actually a multi-byte codeset... since one could combine most lower-case characters with BS (backspace) and overstrike with apostrophe, backtick, tilde, comma (for cedilles), and double-quotes (for umlauts), or with the same char for bold, or with underscore for underline. nroff(1) still uses this for bold and underscore, no?
- thaumasiotes 9y ago
  I've always wondered what the concept of a "backspace" character was supposed to mean. (I tried printing it to erase a previously printed character, but that doesn't work.) I guess it's another relic of the idea that computer output goes to a printer rather than a display?
  - Piskvorrr 9y ago
    Indeed it is. Or rather, output goes to a remote printer. Morse code - teletype (backspace appears here) - mainframe - ASCII - Unicode (you are here).
  - cryptonector 9y ago
    Yes! Terminal vendors had to explicitly support this in terminals, though maybe not just because of printers but because it's actually quite useful. Note that BS only produces overstrike when followed by certain characters (which we might term "combining" for the fun of it), while most will just change the character at that location. A tty spinner is just |BS/BS-BS\BS|BS... with some delay between each -- no overstrike there.

nottorp 9y ago

I don't remember those having multiple representations for the same character in the same encoding or normal forms...

Piskvorrr 9y ago
Indeed not: there is an almost infinite number of things that Latin-1 did not have; e.g. any characters beyond Western Europe. While keeping the 128-256 block compatible was a part of early Unicode (hence the "one-glyph" é), having composed characters was a Unicode primary design goal (hence e and the composing accent). A pure Unicode implementation would have been better, maybe; what we have instead is one that has gained mass adoption. https://en.wikipedia.org/wiki/Latin-1_Supplement_(Unicode_block) https://en.wikipedia.org/wiki/Latin-1_Supplement_(Unicode_bl...
- nottorp 9y ago
  Well, composed characters may or may not have been a good idea, although they're a pain for text search if you ask me. But why oh why did they have to specify FOUR normalization forms? Ref: http://www.unicode.org/reports/tr15/#Norm_Forms http://www.unicode.org/reports/tr15/#Norm_Forms
  - Piskvorrr 9y ago
    ¯\_(ツ)_/¯
  - TazeTSchnitzel 9y ago
    Because you need a normalisation form fit for your particular purpose.
  - the_mitsuhiko 9y ago
    > But why oh why did they have to specify FOUR normalization forms? Ignoring compatibility normalizations there are really only two and neither of those two you can remove. So that's a bit of an odd question. If you want to remove the compatibility forms that's fair but nobody is force to use them. You can consider them "partially normalized" or "not normalized at all" for all intents and purposes