5 ms·

Listen guys, don't get me wrong. As someone with Ø in my name, and both Å and Ø in my address, don't get me started on poorly written systems which cannot handl

Listen guys, don't get me wrong. As someone with Ø in my name, and both Å and Ø in my address, don't get me started on poorly written systems which cannot handle unicode properly. I've seen my name and address mangled in shipping forms, in airline tickets (every time) and even in my marriage-papers since I married abroad. I literally have personal reasons for getting everyone, and I mean everyone, on the unicode bandwagon. That said... Maybe it's because I'm a child of the late 70s and early 80s and learned to program on computers which simply didn't have non-ASCII characters at all... But can't we all just sit down and admit that allowing non-ASCII characters in programming-language identifiers was a bad idea? Can't we in the next revision of EcmaScript (or Rust, or whatever) mandate ASCII-only identifiers when in strict mode or using modules or whatever? Having invisible characters represent executable code is not just a dumb a idea, it's so hazardous that you might call it borderline malicious. There has to be some way to undo this damage, without breaking compatibility with the code which is already out there, right?

Dagonfly 5y ago
Adding a variable decorator/annotation like @Unicode(german,french) would be a good stop-gap. You could only use ASCII characters unless you specified the script that you want to use. One could even set a max limit on how many scripts per variable. Because while I have used German characters in variables before (only if I'm referring to some law or spec), I never had a use case for more than 2 scripts within one variable.
- lifthrasiir 5y ago
  For your information the relevant Unicode specification is the Script_Extensions property [1]. (You can't easily filter by languages, so you should filter by scripts.) [1] https://www.unicode.org/reports/tr24/tr24-32.html#Script_Extensions https://www.unicode.org/reports/tr24/tr24-32.html#Script_Ext...
- est31 5y ago
  The multiple scripts per variable thing is implemented in Rust via a lint. For the explicit enabling of single scripts, I have suggested that for Rust, but sadly people preferred allowing all identifiers (while giving an option to only have ascii but I'd argue this is unfair for anyone who only wants to use a specific non-ascii language, why do they have to suddenly allow all languages in their code base?). There are also practical concerns, like who says what a language is, which characters it contains, how that language is called, etc? Someone has to maintain all these lists.
  - chrismorgan 5y ago
    > who says what a language is, which characters it contains, how that language is called, etc? The Unicode Consortium already maintains all of that data in the CLDR (Common Locale Data Registry).
- silvestrov 5y ago
  I think this is a good idea because once in a while you need to write non-ascii characters in names. This mostly comes up when implementing tax rules or government administrative divisions as some countries have names/concepts which have no good translation into English, so you are left with using the non-English name, which often contains non-ASCII characters.
rtoway 5y ago
Rust has a lint against this kind of attack + you can explicitly disable non-ASCII identifiers if you really want to
- est31 5y ago
  Ideally that lint would be on by default though. Most code doesn't use non-ASCII identifiers. It's not happened though because of uhm. political reasons.
  - drran 5y ago
    Most code made by English speakers contains English word and Latin characters, so other languages and alphabets must be abandoned, and their native speakers must imprisoned until they understand their mistakes.
    - toastal 5y ago
      Abugidas and logographies banished
    - drran 5y ago
      OK, OK, we can start with a warning in the compiler that use of any language except English is unsafe.
  - rtoway 5y ago
    The lint is on by default in the latest version of the compiler
jillesvangurp 5y ago
The issue with this is less that this is possible and more that a lot of javascript ends up in production without ever getting compiled, linted, type-checked, etc. Stuff like this is designed to bypass what little human oversight there is to prevent bad things from happening. What is actually visible also depends on what fonts you have installed on your system. So, it's less clear cut than you think. The problem is not so much that humans can't see this but that they are not looking very hard to begin with (otherwise, they'd be using the appropriate tools) and that we should rely less on them actively looking. Blind trust that things will be fine is the root problem here.
- josteink 5y ago
  > The problem is not so much that humans can't see this but that they are not looking very hard to begin with (otherwise, they'd be using the appropriate tools) and that we should rely less on them actively looking. And simply not allowing non-ASCII identifiers in the first place would be a move in that direction. Now you have one thing less to look for.
auggierose 5y ago
Yeah, let's just switch to Cosmopolitan Identifiers: https://obua.com/publications/cosmo-id/3/ https://obua.com/publications/cosmo-id/3/ :-) But yeah, it would break existing code, sorry.
badsectoracula 5y ago
You can only type ~27% of my name with just ASCII (and even then one letter will not be exactly)... and i agree with you. If anything i'd go a bit further and say that, sure, use Unicode in places where you can find arbitrary text like documents, messages, etc but anything that has to do with the 'guts' of the computer should stay away from Unicode (or at least treat it as data, like how filenames are treated on Linux). I disagree with the getting everyone on the Unicode bandwagon though, IMO Unicode has introduced a ton of problems exactly because it tries to be a ton of stuff at the same time. I don't know how exactly a better solution would be but i have a very hard time accepting that such a convoluted and error prone system is the best solution. IMO if decades later there are still issues with getting it right then there is something fundamentally wrong with the system itself and not with the applications and developers trying to work with it.
- gpderetta 5y ago
  An existing working solution, even if not perfect, patched, with lot of baggage and technical debt is infinitely better than a non-yet invented ideal, perfect solution. And even if the perfect solution existed right now, in a few decades it will be as filed with baggage as the current one. Sometime one has to realize that hard problems are hard.
  - badsectoracula 5y ago
    > Sometime one has to realize that hard problems are hard. And trying to have a single thing solve everything doesn't help at all. When a problem is hard you try to break it into subproblems. Unicode simply tries to do a lot of things at once.
AzzieElbab 5y ago
a lot of software used in shipping/logistics predates unicode
lifthrasiir 5y ago
> But can't we all just sit down and admit that allowing non-ASCII characters in programming-language identifiers was a bad idea? It's a bad idea only if all members in your team can easily produce and comprehend an ASCII-only code. > Having invisible characters represent executable code is not just a dumb a idea, it's so hazardous that you might call it borderline malicious. Not if those invisible characters do affect the rendering. Invisible formatting characters like ZWJ and ZWNJ are allowed because they are used in some scripts. The relevant Unicode specification [1] even provides a guideline to limit ZWJ and ZWNJ strictly to the context where they do affect the rendering. That said, the Hangul filler and half-width Hangul filler were mistakes. They are purely legacy characters and never have been used in practice, so I encourage new languages to exclude them from the default (X)ID_Start/Continue set (Unicode can't do that because of the compatibility, maybe they can introduce another pair of properties without those characters). [1] https://unicode.org/reports/tr31/#Layout_and_Format_Control_Characters https://unicode.org/reports/tr31/#Layout_and_Format_Control_...
- josteink 5y ago
  > The relevant Unicode specification [1] even provides a guideline to limit ZWJ and ZWNJ strictly to the context where they do affect the rendering. Which is exactly what I am suggesting by saying non-ASCII characters should be banned from being used as identifiers, not from being present in the code-file all together or in the form of strings, etc. If the formatting of your output in your applications (as seen by the user) depends on the names you've declared your variables with, then you are doing something horribly wrong.
  - lifthrasiir 5y ago
    You seem to think those formatting characters as something that should be in the higher-level protocol like HTML. They are not. They are used when two consecutive abstract characters can be combined in two or more different ways. And those different renderings frequently have different meanings. That's why they can't be simply removed when normalized; doing so will destroy the text.

Dagonfly 5y ago

Adding a variable decorator/annotation like @Unicode(german,french) would be a good stop-gap. You could only use ASCII characters unless you specified the script that you want to use. One could even set a max limit on how many scripts per variable. Because while I have used German characters in variables before (only if I'm referring to some law or spec), I never had a use case for more than 2 scripts within one variable.

lifthrasiir 5y ago
For your information the relevant Unicode specification is the Script_Extensions property [1]. (You can't easily filter by languages, so you should filter by scripts.) [1] https://www.unicode.org/reports/tr24/tr24-32.html#Script_Extensions https://www.unicode.org/reports/tr24/tr24-32.html#Script_Ext...
est31 5y ago
The multiple scripts per variable thing is implemented in Rust via a lint. For the explicit enabling of single scripts, I have suggested that for Rust, but sadly people preferred allowing all identifiers (while giving an option to only have ascii but I'd argue this is unfair for anyone who only wants to use a specific non-ascii language, why do they have to suddenly allow all languages in their code base?). There are also practical concerns, like who says what a language is, which characters it contains, how that language is called, etc? Someone has to maintain all these lists.
- chrismorgan 5y ago
  > who says what a language is, which characters it contains, how that language is called, etc? The Unicode Consortium already maintains all of that data in the CLDR (Common Locale Data Registry).
silvestrov 5y ago
I think this is a good idea because once in a while you need to write non-ascii characters in names. This mostly comes up when implementing tax rules or government administrative divisions as some countries have names/concepts which have no good translation into English, so you are left with using the non-English name, which often contains non-ASCII characters.

rtoway 5y ago

Rust has a lint against this kind of attack + you can explicitly disable non-ASCII identifiers if you really want to

est31 5y ago
Ideally that lint would be on by default though. Most code doesn't use non-ASCII identifiers. It's not happened though because of uhm. political reasons.
- drran 5y ago
  Most code made by English speakers contains English word and Latin characters, so other languages and alphabets must be abandoned, and their native speakers must imprisoned until they understand their mistakes.
  - toastal 5y ago
    Abugidas and logographies banished
  - drran 5y ago
    OK, OK, we can start with a warning in the compiler that use of any language except English is unsafe.
- rtoway 5y ago
  The lint is on by default in the latest version of the compiler

jillesvangurp 5y ago

The issue with this is less that this is possible and more that a lot of javascript ends up in production without ever getting compiled, linted, type-checked, etc. Stuff like this is designed to bypass what little human oversight there is to prevent bad things from happening. What is actually visible also depends on what fonts you have installed on your system. So, it's less clear cut than you think. The problem is not so much that humans can't see this but that they are not looking very hard to begin with (otherwise, they'd be using the appropriate tools) and that we should rely less on them actively looking. Blind trust that things will be fine is the root problem here.

josteink 5y ago
> The problem is not so much that humans can't see this but that they are not looking very hard to begin with (otherwise, they'd be using the appropriate tools) and that we should rely less on them actively looking. And simply not allowing non-ASCII identifiers in the first place would be a move in that direction. Now you have one thing less to look for.

auggierose 5y ago

Yeah, let's just switch to Cosmopolitan Identifiers: https://obua.com/publications/cosmo-id/3/ https://obua.com/publications/cosmo-id/3/ :-) But yeah, it would break existing code, sorry.

badsectoracula 5y ago

You can only type ~27% of my name with just ASCII (and even then one letter will not be exactly)... and i agree with you. If anything i'd go a bit further and say that, sure, use Unicode in places where you can find arbitrary text like documents, messages, etc but anything that has to do with the 'guts' of the computer should stay away from Unicode (or at least treat it as data, like how filenames are treated on Linux). I disagree with the getting everyone on the Unicode bandwagon though, IMO Unicode has introduced a ton of problems exactly because it tries to be a ton of stuff at the same time. I don't know how exactly a better solution would be but i have a very hard time accepting that such a convoluted and error prone system is the best solution. IMO if decades later there are still issues with getting it right then there is something fundamentally wrong with the system itself and not with the applications and developers trying to work with it.

gpderetta 5y ago
An existing working solution, even if not perfect, patched, with lot of baggage and technical debt is infinitely better than a non-yet invented ideal, perfect solution. And even if the perfect solution existed right now, in a few decades it will be as filed with baggage as the current one. Sometime one has to realize that hard problems are hard.
- badsectoracula 5y ago
  > Sometime one has to realize that hard problems are hard. And trying to have a single thing solve everything doesn't help at all. When a problem is hard you try to break it into subproblems. Unicode simply tries to do a lot of things at once.

AzzieElbab 5y ago

a lot of software used in shipping/logistics predates unicode

lifthrasiir 5y ago

> But can't we all just sit down and admit that allowing non-ASCII characters in programming-language identifiers was a bad idea? It's a bad idea only if all members in your team can easily produce and comprehend an ASCII-only code. > Having invisible characters represent executable code is not just a dumb a idea, it's so hazardous that you might call it borderline malicious. Not if those invisible characters do affect the rendering. Invisible formatting characters like ZWJ and ZWNJ are allowed because they are used in some scripts. The relevant Unicode specification [1] even provides a guideline to limit ZWJ and ZWNJ strictly to the context where they do affect the rendering. That said, the Hangul filler and half-width Hangul filler were mistakes. They are purely legacy characters and never have been used in practice, so I encourage new languages to exclude them from the default (X)ID_Start/Continue set (Unicode can't do that because of the compatibility, maybe they can introduce another pair of properties without those characters). [1] https://unicode.org/reports/tr31/#Layout_and_Format_Control_Characters https://unicode.org/reports/tr31/#Layout_and_Format_Control_...

josteink 5y ago
> The relevant Unicode specification [1] even provides a guideline to limit ZWJ and ZWNJ strictly to the context where they do affect the rendering. Which is exactly what I am suggesting by saying non-ASCII characters should be banned from being used as identifiers, not from being present in the code-file all together or in the form of strings, etc. If the formatting of your output in your applications (as seen by the user) depends on the names you've declared your variables with, then you are doing something horribly wrong.
- lifthrasiir 5y ago
  You seem to think those formatting characters as something that should be in the higher-level protocol like HTML. They are not. They are used when two consecutive abstract characters can be combined in two or more different ways. And those different renderings frequently have different meanings. That's why they can't be simply removed when normalized; doing so will destroy the text.