7 ms·

I mean, awesome for whom? It might be awesome for end users as you can type in or copy/paste things without caring which language you're using. But for programm

by euske 7y ago

I mean, awesome for whom? It might be awesome for end users as you can type in or copy/paste things without caring which language you're using. But for programmers, Unicode is a bloated monstrosity and a source of endless nightmare. Eventually, it's not going to be awesome for end users either because it will be plagued by a lot of (subtle) inconsistencies. Unicode looks a lot like a leaky abstraction to me (because of poor foresight), and it's getting worse each year.

jcranmer 7y ago
If you think Unicode is a "bloated monstrosity and a source of endless nightmare," what would you remove from Unicode? And if you're going to respond "emoji", I'll point out that removing emoji doesn't actually remove anything that makes text processing with Unicode difficult, just makes it more likely that people will assume that what works for English works for everybody. (Side note: it is not possible to accurately represent modern English text solely with ASCII, as English does contain several words with accented characters, such as façade and résumé).
- mr_toad 7y ago
  Most resumes I’ve seen don’t even bother with the accents. Most are written in Word on Windows, and I’d guess that most people don’t even know how to access the accented characters.
- jancsika 7y ago
  > such as façade and résumé That's simple: just url encode. Compare: www.façebook.com to www.fa%C3%A7ebook.com The second one is way easier to comprehend than the first.
  - bawolff 7y ago
    You mean www.xn--faebook-vxa.com of course :P
- userbinator 7y ago
  If you extend ASCII to CP1252, which is the most common encoding besides/before UTF-8 became common, then you do get those accented characters (and that's likely responsible for the popularity of '1252.) In fact, the first 256 characters of Unicode are almost identical to CP1252. I'm pretty sure that's not a coincidence.
  - pdonis 7y ago
    > the first 256 characters of Unicode are almost identical to CP1252. I'm pretty sure that's not a coincidence. That depends on whether you consider the fact that Windows CP 1252 is almost identical to Latin-1 (ISO-8859-1), which is exactly the first 256 characters of Unicode, to be a coincidence.
    - throw0101a 7y ago
      > This character encoding is a superset of ISO 8859-1 in terms of printable characters, but differs from the IANA's ISO-8859-1 by using displayable characters rather than control characters in the 80 to 9F (hex) range. * https://en.wikipedia.org/wiki/Windows-1252 https://en.wikipedia.org/wiki/Windows-1252
- ridiculous_fish 7y ago
  How about removing variation selectors? For example it's possible to turn an emoji back into text by appending a code point! They are very painful to implement and most don't get it right. See https://twitter.com/ridiculous_fish/status/1089421033793236993/photo/1 https://twitter.com/ridiculous_fish/status/10894210337932369...
- AlgorithmicTime 7y ago
  Those words are adequately represented in ASCII by 'facade' and 'resume'
- kps 7y ago
  ‘Remove’ is too strong, since Unicode is entrenched. But there are things that should have been done differently. For instance, combining characters and operators should have been placed before the base character rather than after, so that (a) it would be possible to know when you've reached the end of a character^W glyph^W grapheme cluster without reading ahead, and (b) dead keys would be identical to the corresponding characters. > façade and résumé ASCII (1967) allowed for them: c BS , or , BS c ↦ ç and e BS ' or ' BS e ↦ é. Encoding ç as 63 CC A7 is not manifestly better than encoding it as 63 08 2C.
  - jcranmer 7y ago
    > ASCII (1967) allowed for them: c BS , or , BS c ↦ ç and e BS ' or ' BS e ↦ é. Encoding ç as 63 CC A7 is not manifestly better than encoding it as 63 08 2C. Doesn't work for ñ, since the ASCII ~ is often typeset in the middle of the box instead of in a position to appear above an 'n' character. " is a pretty poor substitute for ◌̈ though, especially when you're trying to write ï as in naïve. And then there's the æ of archæology, which doesn't work with overwriting. I'll also point out that ç is U+00E7 in Unicode and C3 A7 in UTF-8, not 63 CC A6, since it's a precomposed character (and NFC form is usually understood to be the preferred way to normalize Unicode unless there's a reason to do something else).
    - kps 7y ago
      Tilde exists in ASCII because of its use as an accent. (In 1967 the non-diacritic interpretation was an overline.) The use in programming languages, and lowering to fit other mathematical operators, came later. There was never any requirement that ‘n BS ~’ have the same appearance as ‘n’ overprinted with ‘~’, although terminals capable of making the distinction didn't appear until the 70s. Precomposed characters aren't relevant to illustrating composition mechanisms.
- euske 7y ago
  Unicode doesn't solve the underlying complexity of human languages, as you noted. I think the main contribution of Unicode Consortium is that they brought all the nitty-gritty problems of human languages into one central repository and made them visible to everyone. That itself is an awesome effort, and I personally had a lot of benefit from it (my native language is Japanese). But that doesn't make Unicode as a standard "awesome". Maybe we should be thankful for how messy it is? That's more or less a view that I can agree with.
zzo38computer 7y ago
I think Unicode is terrible. Remove everything. Use ASCII and other character sets. Unicode is OK for searching for data using many different languages (if you omit much of the junk such as emoji and compatibility characters), although might not be best with that too. You can't effectively use one character set well for everything; different applications have different requirements. Unicode is equally bad for everything, rather than e.g. ASCII which is good for some stuff and not usable for some stuff, and other character set which is a similar thing. Many things you just can't do accurately with Unicode.
- flohofwoe 7y ago
  Well if you write an application for a 'non-technical' international audience, you'll have to support international text output. And representing text as one of the universal Unicode encodings is still much better than the codepage mess and region-specific multi-byte encodings like Shift-JIS we had before. UTF-8 is usually the best choice both for simple tools and 'user-facing applications' since it is backward-compatible with 7-bit ASCII (e.g. usually you don't need to change a thing in your code, at least if you just pass strings around). If you encounter a byte in an UTF-8 encoded string which has the topmost bit cleared, it's an ASCII character and definitely not part of a multi-byte sequence. If the topmost bit is set, the byte is part of a multi-byte-sequence, and such sequences must remain intact.
  - zzo38computer 7y ago
    UTF-8 isn't such a bad encoding (although it isn't ideal for fix pitch text; I invented a character set and encoding which would be better for fix pitch text). But I was not talking about the encoding; I was talking about the Unicode character set.
    - msla 7y ago
      > UTF-8 isn't such a bad encoding (although it isn't ideal for fix pitch text; I invented a character set and encoding which would be better for fix pitch text). This is utterly incoherent.
maxerickson 7y ago
25 or 50 examples of inconsistencies would help support your tone.
- btilly 7y ago
  Unicode was originally designed to fit in 16 bits, and this is memorialized in Java APIs that make it easy to mess up. The unicode character does not specify the glyph to draw. Han unification is the best known, but not only source, of this challenge. The glyph does not specify the unicode character. Precombined vs combining characters is a source of this challenge. The result is that a name can be entered into a database then unfindable due to a search. This feature has also been a source of security holes. See https://appcheck-ng.com/unicode-normalization-vulnerabilities-the-special-k-polyglot/ https://appcheck-ng.com/unicode-normalization-vulnerabilitie... for an explanation of how. You would think that you could avoid this through banning control and combining characters and not lose anything. Indeed at one point the authors of Go (who included the inventors of UTF-8) thought this. But there are whole languages (particularly from the Indian subcontinent) that cannot be written without combining characters. There are also lots and lots of invisible characters. This has been used to "fingerprint" text. (Each person gets a different invisible signature. The forwarded email includes the signature.) That's an interesting feature but complicates matching text documents even more. Need I go on? When I see Unicode, I know that there lie dragons that programmers don't necessarily expect.
  - maxerickson 7y ago
    One of your points is that an encoding designed to handle languages has support for more than one kind of white space. Given that languages use more than one kind of white space, this is sort of a necessity. Another one is that a standard designed to support all languages has a feature necessary for supporting some languages. Those aren't inconsistencies, so do feel free to go on.
    - btilly 7y ago
      One of your points is that an encoding designed to handle languages has support for more than one kind of white space. No. It is that there is more than one kind of invisible character. No language has invisible characters. Another one is that a standard designed to support all languages has a feature necessary for supporting some languages. Not sure what point you are misreading here. But that was not among my points.
bluetwo 7y ago
Without arguing the details, I have to agree with your statement because the article never really supported it's claim of being awesome.
notatoad 7y ago
>but for programmers I think that depends on what level of the stack you work at. I'm a programmer, but strictly at an end-user-facing level. I'm not implementing Unicode support, I'm using programming languages that already have Unicode support. And Unicode support is an absolute godsend. It's amazing to not have to think about any of that, and just treat it as a solved problem.

by euske 7y ago

jcranmer 7y ago

If you think Unicode is a "bloated monstrosity and a source of endless nightmare," what would you remove from Unicode? And if you're going to respond "emoji", I'll point out that removing emoji doesn't actually remove anything that makes text processing with Unicode difficult, just makes it more likely that people will assume that what works for English works for everybody. (Side note: it is not possible to accurately represent modern English text solely with ASCII, as English does contain several words with accented characters, such as façade and résumé).

mr_toad 7y ago
Most resumes I’ve seen don’t even bother with the accents. Most are written in Word on Windows, and I’d guess that most people don’t even know how to access the accented characters.
jancsika 7y ago
> such as façade and résumé That's simple: just url encode. Compare: www.façebook.com to www.fa%C3%A7ebook.com The second one is way easier to comprehend than the first.
- bawolff 7y ago
  You mean www.xn--faebook-vxa.com of course :P
userbinator 7y ago
If you extend ASCII to CP1252, which is the most common encoding besides/before UTF-8 became common, then you do get those accented characters (and that's likely responsible for the popularity of '1252.) In fact, the first 256 characters of Unicode are almost identical to CP1252. I'm pretty sure that's not a coincidence.
- pdonis 7y ago
  > the first 256 characters of Unicode are almost identical to CP1252. I'm pretty sure that's not a coincidence. That depends on whether you consider the fact that Windows CP 1252 is almost identical to Latin-1 (ISO-8859-1), which is exactly the first 256 characters of Unicode, to be a coincidence.
  - throw0101a 7y ago
    > This character encoding is a superset of ISO 8859-1 in terms of printable characters, but differs from the IANA's ISO-8859-1 by using displayable characters rather than control characters in the 80 to 9F (hex) range. * https://en.wikipedia.org/wiki/Windows-1252 https://en.wikipedia.org/wiki/Windows-1252
ridiculous_fish 7y ago
How about removing variation selectors? For example it's possible to turn an emoji back into text by appending a code point! They are very painful to implement and most don't get it right. See https://twitter.com/ridiculous_fish/status/1089421033793236993/photo/1 https://twitter.com/ridiculous_fish/status/10894210337932369...
AlgorithmicTime 7y ago
Those words are adequately represented in ASCII by 'facade' and 'resume'
kps 7y ago
‘Remove’ is too strong, since Unicode is entrenched. But there are things that should have been done differently. For instance, combining characters and operators should have been placed before the base character rather than after, so that (a) it would be possible to know when you've reached the end of a character^W glyph^W grapheme cluster without reading ahead, and (b) dead keys would be identical to the corresponding characters. > façade and résumé ASCII (1967) allowed for them: c BS , or , BS c ↦ ç and e BS ' or ' BS e ↦ é. Encoding ç as 63 CC A7 is not manifestly better than encoding it as 63 08 2C.
- jcranmer 7y ago
  > ASCII (1967) allowed for them: c BS , or , BS c ↦ ç and e BS ' or ' BS e ↦ é. Encoding ç as 63 CC A7 is not manifestly better than encoding it as 63 08 2C. Doesn't work for ñ, since the ASCII ~ is often typeset in the middle of the box instead of in a position to appear above an 'n' character. " is a pretty poor substitute for ◌̈ though, especially when you're trying to write ï as in naïve. And then there's the æ of archæology, which doesn't work with overwriting. I'll also point out that ç is U+00E7 in Unicode and C3 A7 in UTF-8, not 63 CC A6, since it's a precomposed character (and NFC form is usually understood to be the preferred way to normalize Unicode unless there's a reason to do something else).
  - kps 7y ago
    Tilde exists in ASCII because of its use as an accent. (In 1967 the non-diacritic interpretation was an overline.) The use in programming languages, and lowering to fit other mathematical operators, came later. There was never any requirement that ‘n BS ~’ have the same appearance as ‘n’ overprinted with ‘~’, although terminals capable of making the distinction didn't appear until the 70s. Precomposed characters aren't relevant to illustrating composition mechanisms.
euske 7y ago
Unicode doesn't solve the underlying complexity of human languages, as you noted. I think the main contribution of Unicode Consortium is that they brought all the nitty-gritty problems of human languages into one central repository and made them visible to everyone. That itself is an awesome effort, and I personally had a lot of benefit from it (my native language is Japanese). But that doesn't make Unicode as a standard "awesome". Maybe we should be thankful for how messy it is? That's more or less a view that I can agree with.

zzo38computer 7y ago

I think Unicode is terrible. Remove everything. Use ASCII and other character sets. Unicode is OK for searching for data using many different languages (if you omit much of the junk such as emoji and compatibility characters), although might not be best with that too. You can't effectively use one character set well for everything; different applications have different requirements. Unicode is equally bad for everything, rather than e.g. ASCII which is good for some stuff and not usable for some stuff, and other character set which is a similar thing. Many things you just can't do accurately with Unicode.

flohofwoe 7y ago
Well if you write an application for a 'non-technical' international audience, you'll have to support international text output. And representing text as one of the universal Unicode encodings is still much better than the codepage mess and region-specific multi-byte encodings like Shift-JIS we had before. UTF-8 is usually the best choice both for simple tools and 'user-facing applications' since it is backward-compatible with 7-bit ASCII (e.g. usually you don't need to change a thing in your code, at least if you just pass strings around). If you encounter a byte in an UTF-8 encoded string which has the topmost bit cleared, it's an ASCII character and definitely not part of a multi-byte sequence. If the topmost bit is set, the byte is part of a multi-byte-sequence, and such sequences must remain intact.
- zzo38computer 7y ago
  UTF-8 isn't such a bad encoding (although it isn't ideal for fix pitch text; I invented a character set and encoding which would be better for fix pitch text). But I was not talking about the encoding; I was talking about the Unicode character set.
  - msla 7y ago
    > UTF-8 isn't such a bad encoding (although it isn't ideal for fix pitch text; I invented a character set and encoding which would be better for fix pitch text). This is utterly incoherent.

maxerickson 7y ago

25 or 50 examples of inconsistencies would help support your tone.

btilly 7y ago
Unicode was originally designed to fit in 16 bits, and this is memorialized in Java APIs that make it easy to mess up. The unicode character does not specify the glyph to draw. Han unification is the best known, but not only source, of this challenge. The glyph does not specify the unicode character. Precombined vs combining characters is a source of this challenge. The result is that a name can be entered into a database then unfindable due to a search. This feature has also been a source of security holes. See https://appcheck-ng.com/unicode-normalization-vulnerabilities-the-special-k-polyglot/ https://appcheck-ng.com/unicode-normalization-vulnerabilitie... for an explanation of how. You would think that you could avoid this through banning control and combining characters and not lose anything. Indeed at one point the authors of Go (who included the inventors of UTF-8) thought this. But there are whole languages (particularly from the Indian subcontinent) that cannot be written without combining characters. There are also lots and lots of invisible characters. This has been used to "fingerprint" text. (Each person gets a different invisible signature. The forwarded email includes the signature.) That's an interesting feature but complicates matching text documents even more. Need I go on? When I see Unicode, I know that there lie dragons that programmers don't necessarily expect.
- maxerickson 7y ago
  One of your points is that an encoding designed to handle languages has support for more than one kind of white space. Given that languages use more than one kind of white space, this is sort of a necessity. Another one is that a standard designed to support all languages has a feature necessary for supporting some languages. Those aren't inconsistencies, so do feel free to go on.
  - btilly 7y ago
    One of your points is that an encoding designed to handle languages has support for more than one kind of white space. No. It is that there is more than one kind of invisible character. No language has invisible characters. Another one is that a standard designed to support all languages has a feature necessary for supporting some languages. Not sure what point you are misreading here. But that was not among my points.

bluetwo 7y ago

Without arguing the details, I have to agree with your statement because the article never really supported it's claim of being awesome.

notatoad 7y ago

>but for programmers I think that depends on what level of the stack you work at. I'm a programmer, but strictly at an end-user-facing level. I'm not implementing Unicode support, I'm using programming languages that already have Unicode support. And Unicode support is an absolute godsend. It's amazing to not have to think about any of that, and just treat it as a solved problem.