7 ms·

Unicode in identifiers is an exceptionally bad idea, with a miniscule upside and horrible downsides: 1) As the article demonstrates, this allows obfuscation co

by adwn 8y ago

Unicode in identifiers is an exceptionally bad idea, with a miniscule upside and horrible downsides: 1) As the article demonstrates, this allows obfuscation contests in easy mode. Strangely, the author claims: > Nor do I think most of these tricks would get by a code review or go unnoticed using something as simple as syntax coloring. How would syntax highlighting catch that? Those are valid identifiers! And yes, these "tricks" would almost certainly go unnoticed in a code review, at least when you don't assume that your coworker is maliciously inserting bugs. 2) Even in the absence of malicious intent, Unicode identifiers open the door to bugs and head-scratchers: different characters looking identical, or the same character encoded in different ways. 3) ASCII letters (a-z, A-Z), digits (0-9), and underscore are kind of the lowest common demoninator. When you allow Unicode identifiers, you don't "internationalize" your code, you're doing the exact opposite! If you're using ä, ö, ü, ß in your API, how is a Canadian supposed to easily use it?

boudin 8y ago
The one big upside is, for quite a good amount of people, being able to use their native language instead of being stuck with the latin alphabet When I was in france, all the software were written using french identifiers. It was much better this way since none of the code was meant to be shared outside of france and it made the code much more readable for people who were supposed to work on it. Why should it be possible only for latin based languages?
- adwn 8y ago
  All keywords are in English. The standard library is in English. All external library APIs, open source or otherwise, are in English. But your own functions and variables have to be different? Why? If your grasp on English is sufficient for using a programming language, than it is also sufficient for using English identifiers. And I'm saying that as a native German speaker. > it made the code much more readable "Much more readable"? It made the code base inconsistent with the rest of the language. Are you saying that they were lacking even basic English language skills?
  - hsivonen 8y ago
    It may make sense to use non-English words to name stuff that is about local legal concepts in enterprise software.
  - nemaar 8y ago
    I very much agree with you. Having some strange words in the middle of a large code that is mostly english is just, well, weird. I think inconsistency is a serious issue.
  - gumby 8y ago
    The language keywords are just keywords; you simply learn their meaning. As a native English speaker, though 'while' is a somewhat intuitively-named keyword, the 'for' keyword in c/c++ has to be learned. After a while you don't notice, any more than musicians have to learn Italian to play music (quick: what do 'fortissimo' and 'da capo ala fine' mean). I don't see any problem with source code that says something like 'auto maßtab = waage.m_maßtab();' 'auto' and '=' are simply language tokens.
    - adwn 8y ago
      > I don't see any problem with source code that says something like 'auto maßtab = waage.m_maßtab();' There's no potential for confusion if you allow ASCII letters and 'ß'. Confusion can arise, however, when you allow all Unicode letters.
  - ken 8y ago
    This isn't a very convincing argument to me. "We fail at source code localization in 4 other ways already" is not a good argument for "let's force a failure in a 5th way, too, even though that's the one place where there's no technical issue with it". To me, using English keywords everywhere is a historical accident, not a design goal. In next-generation systems like Subtext [1], it looks entirely possible to add localization to source code, too. Once you throw out the Lisp way of having one namespace for all symbols (or, OK, like 7 in CL), to be used as functions or values, and go to a system where you want to be able to statically type-check everything, why not fully commit to it and let users give local names to identifiers? Then we won't be limited to legacy C-style naming rules, or have to argue about how many space characters to add to make all their names line up on a high-resolution VT100 emulator, either. [1]: https://en.wikipedia.org/wiki/Subtext_(programming_language) https://en.wikipedia.org/wiki/Subtext_(programming_language)
- reaperducer 8y ago
  I once had to debug code where the code was in English, but the comments were all in Cyrillic. Oddly (to me, anyway), the software house that made the program was located in Africa.
  - bsenftner 8y ago
    Similar experience while working for Sony in Japan: all the code is in English, all the comments are in Kanji. Oddly, the variable names were always and only the variable names used in the code examples, as if they thought they were the only allowed variables.
- anoncake 8y ago
  The code of StarOffice probably wasn't meant to be shared outside Germany. I think LibreOffice recently finished translating everything to English.
- vbezhenar 8y ago
  In Russia it's considered very bad and novice style to use Russian words as identifiers (they usually spell it with latin letters anyway, constantly switching between latin and cyrillic layouts is very cumbersome). There's one exception: 1C platform, but that software is used internally for accounting and basically a DSL with heavy use of application domain terminology. Sometimes I'm tempted to use Russian identifiers for application domain terminology, because very often there's no adequate translation or I just don't know English enough to translate some terms. I like the following example: every man here has name combined from last name (Фамилия) which is used for all family members, first name (Имя), that's pretty standard. But there's also third name: Отчество, which is basically father's name with some suffix, e.g. father Иван will have child with third name Иванович. Now I've seen all kinds of translations for those terms, including plain wrong. Last name, family name for Фамилия, first name, name for имя, father name, second name, patronymic, patronymic name for отчество. I've seen bugs because one developer interpreted those terms differently from another. And that's just for name. There are plenty of other terms. Add to that that many developers are bad at English and at best will use first result from Google Translate. So may be national identifiers have its place.
  - kbenson 8y ago
    > every man here has name combined from last name (Фамилия) which is used for all family members, first name (Имя), that's pretty standard. Sounds like you get to skirt a few of the problems outlines in Falsehoods Programmers Believe About Names[1]. That's nice for programmers in Russia that deal with Russia only, if true. :) 1: https://www.kalzumeus.com/2010/06/17/falsehoods-programmers-believe-about-names/ https://www.kalzumeus.com/2010/06/17/falsehoods-programmers-...
    - drewmate 8y ago
      I think Russians do get to skirt a few of the problems... if they believe their systems are only used by native Russians with standard Russian names. Of course the whole point of the article you mentioned is that making such assumptions leads to odd behavior. I (an American) lived in Russia for a while. One time I was having a couch delivered to my apartment and had to give the store my full name. I was delighted to see my middle name show up on the receipt in the отчество (patronymic) field with the Russian patronymic suffix ович attached. I still got the couch no problem, but I got a laugh out of their attempt to shoehorn my (clearly not Russian) names into their naming paradigm.
- renox 8y ago
  Much better that's debatable: 1) you can't really be a programmer without good English knowledge 2) I'm French and worked on a software in French: eventually the source code had to be translated in English (a requirement from one customer): what a waste of time & money..
ubernostrum 8y ago
Unicode in identifiers is an exceptionally bad idea Unicode identifiers can be perfectly well-defined, and many languages have them. The general idea is an identifier can start with any character that has derived property XID_Start, and the remainder can be any characters that have derived property XID_Continue. different characters looking identical, or the same character encoded in different ways Follow the recommendation of UAX #31 and normalize identifiers to NFKC prior to comparing them. For example, the ligature variants in the article should not -- if recommendations are followed -- be different identifiers. Here's some Python (3): >>> import unicodedata >>> raw = ['vpnTrafficPort', 'vpnTraﬀicPort', 'vpnTrafﬁcPort', 'vpnTraﬃcPort'] >>> normalized = set(unicodedata.normalize('NFKC', s) for s in raw) >>> normalized {'vpnTrafficPort'} So there wouldn't be a "surprise" lurking in Python -- all four strings are legal identifiers, but all four of them are also the same identifier (because Python applies normalization). The real difficulty with Unicode identifiers is in places where you really can't avoid Unicode: user inputs. Those who don't read Unicode technical reports are doomed to suffer the moment they build, say, a user-account system that comes into contact with the real world. When you allow Unicode identifiers, you don't "internationalize" your code No, you let other people localize their code. We (the tech world) spent decades making everyone else learn English as a prerequisite to learning a programming language. Now we have the ability to lessen that burden and let the non-ASCII world (which does in fact include English!) spend less time learning English and more time writing code. We probably should do that, even if it seems icky to you.
- continuational 8y ago
  So instead of having to learn English as a prerequisite to understand code bases, you propose that we learn a dozen or more different natural languages instead? Can you imagine what it would be like if every library had to be released in N different translations? Sounds like an exceptionally bad idea. A huge step back for our field.
  - ubernostrum 8y ago
    If you take the position that every codebase everywhere must be available to and understood by every programmer everywhere, sure, you have to settle on a common language. Luckily, I don't think anybody takes that position, or at least takes it seriously. But there are plenty of monolingual dev teams out there whose shared language isn't English and isn't written in ASCII. Why should they be forced to write code in English if nobody who works with it is an English speaker? I already regularly see people post questions to Stack Overflow and mailing lists and IRC where names of classes and functions and variables are ASCII but not English (i.e., someone writes a class called "Utilisateur", not "User"). Why not let them just use their language fully?
chrisseaton 8y ago
> Unicode in identifiers is an exceptionally bad idea In Swift all identifiers are Unicode - basic letters are part of Unicode. The entire source file is Unicode all the way through. > how is a Canadian supposed to easily use it? Using their keyboard? Can't a Canadian type these letters in the exactly the same way as anyone else would?
- adwn 8y ago
  >> Unicode in identifiers is an exceptionally bad idea > In Swift all identifiers are Unicode - basic letters are part of Unicode. Yes, and it's an exceptionally bad idea to allow non-basic letters in identifiers, as the linked article demonstrates. >> how is a Canadian supposed to easily use it? > Using their keyboard? Can't a Canadian type these letters in the exactly the same way as anyone else would? No, they can't. ä, ö, ü, ß are part of the German keyboard layout QWERTZ, but they're not on a QWERTY keyboard.
  - chrisseaton 8y ago
    > No, they can't Anyone can type them using a modifier key. I have a UK keyboard and to get ß I hold option and press s. It's no different to using the shift key. If you've got no problem with capital letters you should have no problem with these letters.
    - adwn 8y ago
      But then you need to learn and remember how to type those letters. There's such a big difference between capital letters, which you need and use constantly, and badly discoverable key combinations, which you need and use infrequently, that your comparison is downright disingenuous. Sometimes I boot into a machine where the keyboard layout is set to English. Finding the right keys on my non-English keyboard to even just type "ls -l /" or similar, is a major pain in the ass. It would be the same for a Canadian having to type 'ß' on an irregular basis.
    - vbezhenar 8y ago
      How do I type it with US layout on Windows?
Someone 8y ago
I don’t think the fact that Unicode identifiers can be confusing is a big problem. If you trust some third party enough to download code they wrote and run it, the added risk that you confuse a few identifiers and run code you didn’t intend to run isn’t that big. If this happens in code you wrote, you fix it when you discover it or learn to live with it, just as when you find the same problems with ASCII identifiers. Those also can be confusing (Il0 vs I1O, color vs colour, misspellings, etc) Also, inconsistent transliteration to ASCII can introduce problems. Using native strings can prevent that.
KKKKkkkk1 8y ago
The importance of Unicode is in letting people write programs in foreign languages. I had a Pascal textbook written in the 80s in which all the variables had German names. That's no longer possible today. Shitty variable naming using ASCII, on the other hand, is still possible and is actually quite prevalent.
tworingsoft 8y ago
> How would syntax highlighting catch that? Those are valid identifiers! Author here, you're absolutely right. When I wrote this, I was thinking of some of the more obvious tricks like the alternate semicolon, parens or curlies. I'm gonna edit the article to qualify that, and add a note about maybe using a SwiftLint exclusion rule containing all these unicode characters. Thanks for the discussion! :)

by adwn 8y ago

boudin 8y ago

The one big upside is, for quite a good amount of people, being able to use their native language instead of being stuck with the latin alphabet When I was in france, all the software were written using french identifiers. It was much better this way since none of the code was meant to be shared outside of france and it made the code much more readable for people who were supposed to work on it. Why should it be possible only for latin based languages?

adwn 8y ago
All keywords are in English. The standard library is in English. All external library APIs, open source or otherwise, are in English. But your own functions and variables have to be different? Why? If your grasp on English is sufficient for using a programming language, than it is also sufficient for using English identifiers. And I'm saying that as a native German speaker. > it made the code much more readable "Much more readable"? It made the code base inconsistent with the rest of the language. Are you saying that they were lacking even basic English language skills?
- hsivonen 8y ago
  It may make sense to use non-English words to name stuff that is about local legal concepts in enterprise software.
- nemaar 8y ago
  I very much agree with you. Having some strange words in the middle of a large code that is mostly english is just, well, weird. I think inconsistency is a serious issue.
- gumby 8y ago
  The language keywords are just keywords; you simply learn their meaning. As a native English speaker, though 'while' is a somewhat intuitively-named keyword, the 'for' keyword in c/c++ has to be learned. After a while you don't notice, any more than musicians have to learn Italian to play music (quick: what do 'fortissimo' and 'da capo ala fine' mean). I don't see any problem with source code that says something like 'auto maßtab = waage.m_maßtab();' 'auto' and '=' are simply language tokens.
  - adwn 8y ago
    > I don't see any problem with source code that says something like 'auto maßtab = waage.m_maßtab();' There's no potential for confusion if you allow ASCII letters and 'ß'. Confusion can arise, however, when you allow all Unicode letters.
- ken 8y ago
  This isn't a very convincing argument to me. "We fail at source code localization in 4 other ways already" is not a good argument for "let's force a failure in a 5th way, too, even though that's the one place where there's no technical issue with it". To me, using English keywords everywhere is a historical accident, not a design goal. In next-generation systems like Subtext [1], it looks entirely possible to add localization to source code, too. Once you throw out the Lisp way of having one namespace for all symbols (or, OK, like 7 in CL), to be used as functions or values, and go to a system where you want to be able to statically type-check everything, why not fully commit to it and let users give local names to identifiers? Then we won't be limited to legacy C-style naming rules, or have to argue about how many space characters to add to make all their names line up on a high-resolution VT100 emulator, either. [1]: https://en.wikipedia.org/wiki/Subtext_(programming_language) https://en.wikipedia.org/wiki/Subtext_(programming_language)
reaperducer 8y ago
I once had to debug code where the code was in English, but the comments were all in Cyrillic. Oddly (to me, anyway), the software house that made the program was located in Africa.
- bsenftner 8y ago
  Similar experience while working for Sony in Japan: all the code is in English, all the comments are in Kanji. Oddly, the variable names were always and only the variable names used in the code examples, as if they thought they were the only allowed variables.
anoncake 8y ago
The code of StarOffice probably wasn't meant to be shared outside Germany. I think LibreOffice recently finished translating everything to English.
vbezhenar 8y ago
In Russia it's considered very bad and novice style to use Russian words as identifiers (they usually spell it with latin letters anyway, constantly switching between latin and cyrillic layouts is very cumbersome). There's one exception: 1C platform, but that software is used internally for accounting and basically a DSL with heavy use of application domain terminology. Sometimes I'm tempted to use Russian identifiers for application domain terminology, because very often there's no adequate translation or I just don't know English enough to translate some terms. I like the following example: every man here has name combined from last name (Фамилия) which is used for all family members, first name (Имя), that's pretty standard. But there's also third name: Отчество, which is basically father's name with some suffix, e.g. father Иван will have child with third name Иванович. Now I've seen all kinds of translations for those terms, including plain wrong. Last name, family name for Фамилия, first name, name for имя, father name, second name, patronymic, patronymic name for отчество. I've seen bugs because one developer interpreted those terms differently from another. And that's just for name. There are plenty of other terms. Add to that that many developers are bad at English and at best will use first result from Google Translate. So may be national identifiers have its place.
- kbenson 8y ago
  > every man here has name combined from last name (Фамилия) which is used for all family members, first name (Имя), that's pretty standard. Sounds like you get to skirt a few of the problems outlines in Falsehoods Programmers Believe About Names[1]. That's nice for programmers in Russia that deal with Russia only, if true. :) 1: https://www.kalzumeus.com/2010/06/17/falsehoods-programmers-believe-about-names/ https://www.kalzumeus.com/2010/06/17/falsehoods-programmers-...
  - drewmate 8y ago
    I think Russians do get to skirt a few of the problems... if they believe their systems are only used by native Russians with standard Russian names. Of course the whole point of the article you mentioned is that making such assumptions leads to odd behavior. I (an American) lived in Russia for a while. One time I was having a couch delivered to my apartment and had to give the store my full name. I was delighted to see my middle name show up on the receipt in the отчество (patronymic) field with the Russian patronymic suffix ович attached. I still got the couch no problem, but I got a laugh out of their attempt to shoehorn my (clearly not Russian) names into their naming paradigm.
renox 8y ago
Much better that's debatable: 1) you can't really be a programmer without good English knowledge 2) I'm French and worked on a software in French: eventually the source code had to be translated in English (a requirement from one customer): what a waste of time & money..

ubernostrum 8y ago

Unicode in identifiers is an exceptionally bad idea Unicode identifiers can be perfectly well-defined, and many languages have them. The general idea is an identifier can start with any character that has derived property XID_Start, and the remainder can be any characters that have derived property XID_Continue. different characters looking identical, or the same character encoded in different ways Follow the recommendation of UAX #31 and normalize identifiers to NFKC prior to comparing them. For example, the ligature variants in the article should not -- if recommendations are followed -- be different identifiers. Here's some Python (3): >>> import unicodedata >>> raw = ['vpnTrafficPort', 'vpnTraﬀicPort', 'vpnTrafﬁcPort', 'vpnTraﬃcPort'] >>> normalized = set(unicodedata.normalize('NFKC', s) for s in raw) >>> normalized {'vpnTrafficPort'} So there wouldn't be a "surprise" lurking in Python -- all four strings are legal identifiers, but all four of them are also the same identifier (because Python applies normalization). The real difficulty with Unicode identifiers is in places where you really can't avoid Unicode: user inputs. Those who don't read Unicode technical reports are doomed to suffer the moment they build, say, a user-account system that comes into contact with the real world. When you allow Unicode identifiers, you don't "internationalize" your code No, you let other people localize their code. We (the tech world) spent decades making everyone else learn English as a prerequisite to learning a programming language. Now we have the ability to lessen that burden and let the non-ASCII world (which does in fact include English!) spend less time learning English and more time writing code. We probably should do that, even if it seems icky to you.

continuational 8y ago
So instead of having to learn English as a prerequisite to understand code bases, you propose that we learn a dozen or more different natural languages instead? Can you imagine what it would be like if every library had to be released in N different translations? Sounds like an exceptionally bad idea. A huge step back for our field.
- ubernostrum 8y ago
  If you take the position that every codebase everywhere must be available to and understood by every programmer everywhere, sure, you have to settle on a common language. Luckily, I don't think anybody takes that position, or at least takes it seriously. But there are plenty of monolingual dev teams out there whose shared language isn't English and isn't written in ASCII. Why should they be forced to write code in English if nobody who works with it is an English speaker? I already regularly see people post questions to Stack Overflow and mailing lists and IRC where names of classes and functions and variables are ASCII but not English (i.e., someone writes a class called "Utilisateur", not "User"). Why not let them just use their language fully?

chrisseaton 8y ago

> Unicode in identifiers is an exceptionally bad idea In Swift all identifiers are Unicode - basic letters are part of Unicode. The entire source file is Unicode all the way through. > how is a Canadian supposed to easily use it? Using their keyboard? Can't a Canadian type these letters in the exactly the same way as anyone else would?

adwn 8y ago
>> Unicode in identifiers is an exceptionally bad idea > In Swift all identifiers are Unicode - basic letters are part of Unicode. Yes, and it's an exceptionally bad idea to allow non-basic letters in identifiers, as the linked article demonstrates. >> how is a Canadian supposed to easily use it? > Using their keyboard? Can't a Canadian type these letters in the exactly the same way as anyone else would? No, they can't. ä, ö, ü, ß are part of the German keyboard layout QWERTZ, but they're not on a QWERTY keyboard.
- chrisseaton 8y ago
  > No, they can't Anyone can type them using a modifier key. I have a UK keyboard and to get ß I hold option and press s. It's no different to using the shift key. If you've got no problem with capital letters you should have no problem with these letters.
  - adwn 8y ago
    But then you need to learn and remember how to type those letters. There's such a big difference between capital letters, which you need and use constantly, and badly discoverable key combinations, which you need and use infrequently, that your comparison is downright disingenuous. Sometimes I boot into a machine where the keyboard layout is set to English. Finding the right keys on my non-English keyboard to even just type "ls -l /" or similar, is a major pain in the ass. It would be the same for a Canadian having to type 'ß' on an irregular basis.
  - vbezhenar 8y ago
    How do I type it with US layout on Windows?

Someone 8y ago

I don’t think the fact that Unicode identifiers can be confusing is a big problem. If you trust some third party enough to download code they wrote and run it, the added risk that you confuse a few identifiers and run code you didn’t intend to run isn’t that big. If this happens in code you wrote, you fix it when you discover it or learn to live with it, just as when you find the same problems with ASCII identifiers. Those also can be confusing (Il0 vs I1O, color vs colour, misspellings, etc) Also, inconsistent transliteration to ASCII can introduce problems. Using native strings can prevent that.

KKKKkkkk1 8y ago

The importance of Unicode is in letting people write programs in foreign languages. I had a Pascal textbook written in the 80s in which all the variables had German names. That's no longer possible today. Shitty variable naming using ASCII, on the other hand, is still possible and is actually quite prevalent.

tworingsoft 8y ago

> How would syntax highlighting catch that? Those are valid identifiers! Author here, you're absolutely right. When I wrote this, I was thinking of some of the more obvious tricks like the alternate semicolon, parens or curlies. I'm gonna edit the article to qualify that, and add a note about maybe using a SwiftLint exclusion rule containing all these unicode characters. Thanks for the discussion! :)