4 ms·

> It would be nice if there was a well defined 'simple' subset that handled the 80% case that could be a reasonable target for the average app to support fully

> It would be nice if there was a well defined 'simple' subset that handled the 80% case that could be a reasonable target for the average app to support fully. Isn't that ASCII?

mattnewport 8y ago
Well it ends up that if you don't want to go down the Unicode rabbit hole too far then yeah, your best bet is probably to stick with ASCII. As an example from my industry though, it would be nice if I could implement user name entry and display for a high score table in a simple game and support names in common European languages without needing to handle all the edge cases of e.g. mixed left to right and right to left text, combining characters, surrogates, etc. I'm far from a Unicode or a languages expert but I'm familiar with one language with right to left non Latin characters and aware of just enough Unicode madness to know I don't know enough to handle many edge cases properly. It would be nice if a regular developer like me could support something more than plain ASCII but less than the full insanity of Unicode to accommodate at least some non English users.
- saagarjha 8y ago
  You're basically throwing people with "inconvenient" character sets (i.e. everything that doesn't use something strongly resembling Latin characters) under the bus. Sure, you might be able to support Spanish, French, and German, but you're basically disregarding Japanese, Chinese, and Hindi when doing so (and possibly even ASCII, since you'd trade some symbols for accents).
  - mattnewport 8y ago
    That's not at all what I'm advocating. My point is that the extreme difficulty of fully supporting Unicode with all of its complex edge cases (like my examples of mixing left to right and right to left languages) means that the two most common outcomes are throwing up your hands and giving up supporting anything but English and just using ASCII or having broken, partial Unicode support. I'm suggesting it would be nice to have another option where you could provide some level of support for non English languages with something you have some hope of implementing correctly. Applications that correctly handle editing of mixed left to right and right to left text are rare for example but you could support Farsi speakers reasonably well in many applications without handling that scenario.
- bradleyjg 8y ago
  There’s Latin-1 which gets you full coverage on many European languages and almost complete coverage on several others (e.g. French uses œ, which it doesn’t have but French readers will be able to understand if oe is substituted). The problem with Latin-1 and other 1-byte options is that, unlike ASCII, they aren’t forward compatible with utf-8, which is the emerging de facto string exchange encoding. For a stand alone video game, maybe that doesn’t matter but for anything network enabled it can be a big issue.
tomcam 8y ago
No. Computers are in use by about 3 billion people right now. Only a minority of them use only ASCII characters in their day-to-day writing. Turns out the US and UK comprise only a small fraction of the world’s population.
- bradleyjg 8y ago
  But for many of the companies that the people on HN happen to work for the majority or even overwhelming majority of customers.
  - gmueckl 8y ago
    And therein lies a terrible misconception: the world does not speak English or German or Mandarin or French, but a horrible mix of all these languages. Eventually, almost any system will have to deal with that. Simple example and a current pet peeve of mine while staying in the US: my name is spelled with an ü. I will likely try to enter that in your web form, because it is part of my legal name. A lot of systems happen to "sanitize" that input when it is passed across some invisible internal boundaries and it becomes a u. Now that system has actually changed my name. ü and u are completely different letters. The proper conversion actually is the transcription ue - two letters! If that were - say - the address for a letter to Germany, it might well return to sender because there is noone with the (altered) name living at the given address.

> It would be nice if there was a well defined 'simple' subset that handled the 80% case that could be a reasonable target for the average app to support fully. Isn't that ASCII?

mattnewport 8y ago

Well it ends up that if you don't want to go down the Unicode rabbit hole too far then yeah, your best bet is probably to stick with ASCII. As an example from my industry though, it would be nice if I could implement user name entry and display for a high score table in a simple game and support names in common European languages without needing to handle all the edge cases of e.g. mixed left to right and right to left text, combining characters, surrogates, etc. I'm far from a Unicode or a languages expert but I'm familiar with one language with right to left non Latin characters and aware of just enough Unicode madness to know I don't know enough to handle many edge cases properly. It would be nice if a regular developer like me could support something more than plain ASCII but less than the full insanity of Unicode to accommodate at least some non English users.

saagarjha 8y ago
You're basically throwing people with "inconvenient" character sets (i.e. everything that doesn't use something strongly resembling Latin characters) under the bus. Sure, you might be able to support Spanish, French, and German, but you're basically disregarding Japanese, Chinese, and Hindi when doing so (and possibly even ASCII, since you'd trade some symbols for accents).
- mattnewport 8y ago
  That's not at all what I'm advocating. My point is that the extreme difficulty of fully supporting Unicode with all of its complex edge cases (like my examples of mixing left to right and right to left languages) means that the two most common outcomes are throwing up your hands and giving up supporting anything but English and just using ASCII or having broken, partial Unicode support. I'm suggesting it would be nice to have another option where you could provide some level of support for non English languages with something you have some hope of implementing correctly. Applications that correctly handle editing of mixed left to right and right to left text are rare for example but you could support Farsi speakers reasonably well in many applications without handling that scenario.
bradleyjg 8y ago
There’s Latin-1 which gets you full coverage on many European languages and almost complete coverage on several others (e.g. French uses œ, which it doesn’t have but French readers will be able to understand if oe is substituted). The problem with Latin-1 and other 1-byte options is that, unlike ASCII, they aren’t forward compatible with utf-8, which is the emerging de facto string exchange encoding. For a stand alone video game, maybe that doesn’t matter but for anything network enabled it can be a big issue.

tomcam 8y ago

No. Computers are in use by about 3 billion people right now. Only a minority of them use only ASCII characters in their day-to-day writing. Turns out the US and UK comprise only a small fraction of the world’s population.

bradleyjg 8y ago
But for many of the companies that the people on HN happen to work for the majority or even overwhelming majority of customers.
- gmueckl 8y ago
  And therein lies a terrible misconception: the world does not speak English or German or Mandarin or French, but a horrible mix of all these languages. Eventually, almost any system will have to deal with that. Simple example and a current pet peeve of mine while staying in the US: my name is spelled with an ü. I will likely try to enter that in your web form, because it is part of my legal name. A lot of systems happen to "sanitize" that input when it is passed across some invisible internal boundaries and it becomes a u. Now that system has actually changed my name. ü and u are completely different letters. The proper conversion actually is the transcription ue - two letters! If that were - say - the address for a letter to Germany, it might well return to sender because there is noone with the (altered) name living at the given address.