5 ms·

I'm personally incredibly annoyed by just the idea of "Unicode is hard, let's do ASCII", most of the world is non-ASCII, it's just annoying and sad to still see

I'm personally incredibly annoyed by just the idea of "Unicode is hard, let's do ASCII", most of the world is non-ASCII, it's just annoying and sad to still see systems that fail when people try to use their native languages. UTF-8 should be the default pretty much everywhere, there are quite easy ways to avoid homograph attacks, those attacks are a poor excuse to discriminate against non-anglosphere.

antris 7y ago
Yes, all user-facing text should be Unicode, always. But ASCII has its use cases as well, as the parent comment mentioned in programming. I am natively non-ASCII compatible, but I am glad to program in ASCII and English. The only things that should not be English in written code are domain-specific terms that do not have an official unambiguous English translation. That's the only case to be made for non-ASCII characters in programming that I can think of, but I think romanization can take care of it with most languages.
- ksec 7y ago
  This, I think people are arguing for ASCII in programming, not going back to ASCII in general.
  - squiggleblaz 7y ago
    But aside from APL and a few Haskell coders, who isn't programming in ascii? I mean, sometimes i chuck an emoji into a comment but I don't think that really counts. And how could programming in ascii have solved this bug? The problem is that the strings were compared without giving any thought to what "are these strings equal" is supposed to mean. The only general solution to this kind of bug - that is, to considering distinct emails to be identical - is to have a special function that can check if two emails are identical according to the spec. This function will be different than determining if two names are the same in an American database where dotless and dotful i are the same, and that function will also be different than a function determining if two names are the same in a Turkish database where dotless and dotful i are different. Although I don't know why you're trying to compare strings for identity, you're probably just after similarity anyway - how often do we convert strings to lowercase before we compare them. Generic string equality functions are always the source of bugs, since there's no useful, single uniformally applicable definition of string equality.
- elldoubleyew 7y ago
  >I think romanization can take care of it with most languages Except for the ones that it really can't. When you try and romanize standard written Chinese it becomes nigh impossible to read for most speakers, often it would be more confusing than an English translation.
simias 7y ago
I'm very much in favor of UTF-8 everywhere, I think the pros outweigh the cons, but: >there are quite easy ways to avoid homograph attacks I'd like to hear about those because as far as I can tell it's still very much an unsolved problem.
- jcranmer 7y ago
  The simplest way to avoid most of them is to ban script mixing, especially mixing Latin, Greek, Cyrillic, and Cherokee.
  - squiggleblaz 7y ago
    So in a secure environment I can't write words in IPA? IPA requires mixing Greek beta and theta with Latin letters (but there's a Latin/IPA phi just for fun). It also doesn't help ɡ vs g. It sounds like any genuine solution is either not simple or excludes legitimate use cases.
    - cdirkx 7y ago
      Yes, the Unicode Technical standard identifies the Latin, Greek and Cyrillic (most of IPA characters) scripts as confusable, and recommends against mixing them in identifiers. Use in general text is fine though.
- cdirkx 7y ago
  The Unicode Technical Standard [1] (different from Unicode Standard) recommends treating identifiers (filenames, variable or fuction names, email adresses, usernames, etc.) different from normal text. There is a special class of 'identifier characters' which already exludes a lot of sneaky characters like invisible punctuation and obscure scripts that are not in modern use. Additionally there are 5 additional restriction levels for identifiers depending on your specific situation: 1. ASCII only 2. single script 3. single script or Latin+{Japn, Hanb, Kore} 4. single script or Latin+{any, excluding Cyrillic, Greek} 5. Not containing any characters in the recommended blacklist of characters for use in secure contexts 6. No restrictions other than normal identifier restrictions For any specific strings that might be confused, there is an algorithm to compute the visual 'skeleton' of a string and match it against that of another string to test if they are confusable. [1] https://www.unicode.org/reports/tr39/ https://www.unicode.org/reports/tr39/
iforgotpassword 7y ago
> there are quite easy ways to avoid homograph attacks, those attacks are a poor excuse to discriminate against non-anglosphere. No. There are these kind of attacks every now and then to this day. Maybe of you're not following itsec they fly under your radar, but getting this right is exceptionally hard. And besides these kind of attack, every major os had multiple bugs just in the processing of Unicode that could at least be used for DoS attacks. So saying it's easy to avoid any sort of abuse of Unicode seems quite ridiculous. Go ahead and support it for messages, display names and whatnot, but for the love of god, limit the login name of users to ASCII. Don't assume that your Python/Go/JavaScript lib for Unicode handles sanitizing and canonicalization properly. It doesn't. And even if it has only a minor bug that doesn't lead to direct issues, the next update of the lib might fix the problem and now you have to deal with the fact that your db might contain data that was processed with the old faulty lib and now gets compared to the properly processed output of the new version. Just don't. Use it as opaque data for displaying, as GP said, but never as an identifier for anything.
knolax 7y ago
I swear software's biggest problem is that developers prioritize their own ergonomics over actually producing functional code. Imagine if auto engineers just decided "safety is hard, let's just make deathtraps". I know the current narrative is that the 737 Max failed because of MBAs but it's a really big indictment that out of all the complex and hard to engineer systems in an aircraft it was poor software that caused the crash.
- bregma 7y ago
  It was a bad corporate ecosystem that caused the crash. The people at fault were the accountants that tweaked lines un Excel until they got the numbers they wanted amd then pressed reality into the service of those numbers. Bad software didn't eliminate pilot training on the new systems. Bad software didn't shortcut the required FAA safety certification on the system. Bad software didn't reduce the number of redundant sensors that fed the system. Bad beancounters did all that. C-levels in suits who got million-dollar bonuses for killing 346 people to gain marginally increased quarterly results. Don't blame the software.
- jsjohnst 7y ago
  > really big indictment that out of all the complex and hard to engineer systems in an aircraft it was poor software that caused the crash While I frequently point out how bad we are as an industry at making fault tolerant code, this part is just flat wrong. The software portion of the 737, while definitely flawed in a catastrophic way, would not have came to be if the aeronautics engineers had done their job and designed a flight worthy plane without software hacks. Not to imply the aeronautics guys are the root cause either though, the 737 Max fiasco is a top to bottom completely failure of Boeing as a whole, virtually every department involved in the Max has a significant reason to share in the blame.
rsync 7y ago
"I'm personally incredibly annoyed by just the idea of "Unicode is hard, let's do ASCII", most of the world is non-ASCII, it's just annoying and sad to still see systems that fail when people try to use their native languages." You are correct that most of the world is non-ascii and I am enthusiastic about recognizing that diversity and I am willing to pay certain costs in return for the richness it provides. However, I will point out that certain systems have been deemed crucial and in need of deliberate (and brutal) dumbing-down. Specifically, I speak of the global Air Traffic Control system that is English only[1]. Tagalog/Flemish/Satsugu is hard. Let's (land airplanes with) English. [1] https://en.wikipedia.org/wiki/Aviation_English https://en.wikipedia.org/wiki/Aviation_English
Iv 7y ago
I want to be able to collaborate with Japanese, Chinese, Indians, Russian, Arabian, Malay programmers. Even if we somehow live in a dystopia where all the english-speaking world disappears in a magic puff, I think I will continue to shut down my native french and communicate with other fellow programmers in English ASCII. I wish more programmers understood the value of having a lingua franca is. UTF-8 should be used everywhere on the user-facing side, but under the hoods, you want unambiguous representation of strings and code. It is easy to avoid homograph attacks if you do have a non-unicode representation available to you.

antris 7y ago

Yes, all user-facing text should be Unicode, always. But ASCII has its use cases as well, as the parent comment mentioned in programming. I am natively non-ASCII compatible, but I am glad to program in ASCII and English. The only things that should not be English in written code are domain-specific terms that do not have an official unambiguous English translation. That's the only case to be made for non-ASCII characters in programming that I can think of, but I think romanization can take care of it with most languages.

ksec 7y ago
This, I think people are arguing for ASCII in programming, not going back to ASCII in general.
- squiggleblaz 7y ago
  But aside from APL and a few Haskell coders, who isn't programming in ascii? I mean, sometimes i chuck an emoji into a comment but I don't think that really counts. And how could programming in ascii have solved this bug? The problem is that the strings were compared without giving any thought to what "are these strings equal" is supposed to mean. The only general solution to this kind of bug - that is, to considering distinct emails to be identical - is to have a special function that can check if two emails are identical according to the spec. This function will be different than determining if two names are the same in an American database where dotless and dotful i are the same, and that function will also be different than a function determining if two names are the same in a Turkish database where dotless and dotful i are different. Although I don't know why you're trying to compare strings for identity, you're probably just after similarity anyway - how often do we convert strings to lowercase before we compare them. Generic string equality functions are always the source of bugs, since there's no useful, single uniformally applicable definition of string equality.
elldoubleyew 7y ago
>I think romanization can take care of it with most languages Except for the ones that it really can't. When you try and romanize standard written Chinese it becomes nigh impossible to read for most speakers, often it would be more confusing than an English translation.

simias 7y ago

I'm very much in favor of UTF-8 everywhere, I think the pros outweigh the cons, but: >there are quite easy ways to avoid homograph attacks I'd like to hear about those because as far as I can tell it's still very much an unsolved problem.

jcranmer 7y ago
The simplest way to avoid most of them is to ban script mixing, especially mixing Latin, Greek, Cyrillic, and Cherokee.
- squiggleblaz 7y ago
  So in a secure environment I can't write words in IPA? IPA requires mixing Greek beta and theta with Latin letters (but there's a Latin/IPA phi just for fun). It also doesn't help ɡ vs g. It sounds like any genuine solution is either not simple or excludes legitimate use cases.
  - cdirkx 7y ago
    Yes, the Unicode Technical standard identifies the Latin, Greek and Cyrillic (most of IPA characters) scripts as confusable, and recommends against mixing them in identifiers. Use in general text is fine though.
cdirkx 7y ago
The Unicode Technical Standard [1] (different from Unicode Standard) recommends treating identifiers (filenames, variable or fuction names, email adresses, usernames, etc.) different from normal text. There is a special class of 'identifier characters' which already exludes a lot of sneaky characters like invisible punctuation and obscure scripts that are not in modern use. Additionally there are 5 additional restriction levels for identifiers depending on your specific situation: 1. ASCII only 2. single script 3. single script or Latin+{Japn, Hanb, Kore} 4. single script or Latin+{any, excluding Cyrillic, Greek} 5. Not containing any characters in the recommended blacklist of characters for use in secure contexts 6. No restrictions other than normal identifier restrictions For any specific strings that might be confused, there is an algorithm to compute the visual 'skeleton' of a string and match it against that of another string to test if they are confusable. [1] https://www.unicode.org/reports/tr39/ https://www.unicode.org/reports/tr39/

iforgotpassword 7y ago

> there are quite easy ways to avoid homograph attacks, those attacks are a poor excuse to discriminate against non-anglosphere. No. There are these kind of attacks every now and then to this day. Maybe of you're not following itsec they fly under your radar, but getting this right is exceptionally hard. And besides these kind of attack, every major os had multiple bugs just in the processing of Unicode that could at least be used for DoS attacks. So saying it's easy to avoid any sort of abuse of Unicode seems quite ridiculous. Go ahead and support it for messages, display names and whatnot, but for the love of god, limit the login name of users to ASCII. Don't assume that your Python/Go/JavaScript lib for Unicode handles sanitizing and canonicalization properly. It doesn't. And even if it has only a minor bug that doesn't lead to direct issues, the next update of the lib might fix the problem and now you have to deal with the fact that your db might contain data that was processed with the old faulty lib and now gets compared to the properly processed output of the new version. Just don't. Use it as opaque data for displaying, as GP said, but never as an identifier for anything.

knolax 7y ago

I swear software's biggest problem is that developers prioritize their own ergonomics over actually producing functional code. Imagine if auto engineers just decided "safety is hard, let's just make deathtraps". I know the current narrative is that the 737 Max failed because of MBAs but it's a really big indictment that out of all the complex and hard to engineer systems in an aircraft it was poor software that caused the crash.

bregma 7y ago
It was a bad corporate ecosystem that caused the crash. The people at fault were the accountants that tweaked lines un Excel until they got the numbers they wanted amd then pressed reality into the service of those numbers. Bad software didn't eliminate pilot training on the new systems. Bad software didn't shortcut the required FAA safety certification on the system. Bad software didn't reduce the number of redundant sensors that fed the system. Bad beancounters did all that. C-levels in suits who got million-dollar bonuses for killing 346 people to gain marginally increased quarterly results. Don't blame the software.
jsjohnst 7y ago
> really big indictment that out of all the complex and hard to engineer systems in an aircraft it was poor software that caused the crash While I frequently point out how bad we are as an industry at making fault tolerant code, this part is just flat wrong. The software portion of the 737, while definitely flawed in a catastrophic way, would not have came to be if the aeronautics engineers had done their job and designed a flight worthy plane without software hacks. Not to imply the aeronautics guys are the root cause either though, the 737 Max fiasco is a top to bottom completely failure of Boeing as a whole, virtually every department involved in the Max has a significant reason to share in the blame.

rsync 7y ago

"I'm personally incredibly annoyed by just the idea of "Unicode is hard, let's do ASCII", most of the world is non-ASCII, it's just annoying and sad to still see systems that fail when people try to use their native languages." You are correct that most of the world is non-ascii and I am enthusiastic about recognizing that diversity and I am willing to pay certain costs in return for the richness it provides. However, I will point out that certain systems have been deemed crucial and in need of deliberate (and brutal) dumbing-down. Specifically, I speak of the global Air Traffic Control system that is English only[1]. Tagalog/Flemish/Satsugu is hard. Let's (land airplanes with) English. [1] https://en.wikipedia.org/wiki/Aviation_English https://en.wikipedia.org/wiki/Aviation_English

Iv 7y ago

I want to be able to collaborate with Japanese, Chinese, Indians, Russian, Arabian, Malay programmers. Even if we somehow live in a dystopia where all the english-speaking world disappears in a magic puff, I think I will continue to shut down my native french and communicate with other fellow programmers in English ASCII. I wish more programmers understood the value of having a lingua franca is. UTF-8 should be used everywhere on the user-facing side, but under the hoods, you want unambiguous representation of strings and code. It is easy to avoid homograph attacks if you do have a non-unicode representation available to you.