19 ms·

The Invisible JavaScript Backdoor

573 points by davidbarker 5y ago

dlsa 5y ago
Compilers and interpreters need a new pass to detect these characters in code and treat them as hard errors. This doesn't stop their use in comments where presumably they are still ok. Alternatively, there needs to be an uptake of the use of code linters and pretty printers. A bit of both perhaps.
Suvitruf 5y ago
This issue and example used are more about data validation and escaping characters.
cphoover 5y ago
I imagine this can be defended again pretty easily with a lint rule that prevents these unicode characters in variables. pretty ingenious little hack though. The eslint rule id-match, which require identifiers to match a specified regular expression, would be useful here. For example: "id-match": ["error", "^[a-z]+([A-Z][a-z]+)*$"]
- bluepnume 5y ago
  A malicious PR could also add the character to your eslintrc too though. You'd be forgiven for seeing the line change in the diff and thinking it was just some reformatting.
  - jraph 5y ago
    It seems like things displaying diffs could use a specific color for lines only changed by formatting or indentation (indentation can have significant meaning like in Python but this would probably be good enough)
    - dotancohen 5y ago
      I believe that Git diff - which has features not supported by regular diff such as --word-diff - can differentiate between whitespace-only line changes. The Jetbrains IDE, which I believe uses Git diff behind the scenes, will show who originally wrote a line even if it has been whitespace-reformatted later.
  - cphoover 5y ago
    that would show up on a diff and would elicit a question in code review hopefully of why the .eslintrc file was being changed in this way. This another good argument for a comprehensive code review process. also you could lock this file down with a CODEOWNERS file so only certain trusted contributors could modify the lint configuration. You could also do exclusionary pattern matching to make sure none of the bad characters do exist in identifier names... Or you could write your eslint configuration as a separate module to be npm installed... or you could write a eslint rule plugin that disallows non-ascii identifiers and then npm install that... lots of different ways to skin this cat to add security.
Klaster_1 5y ago
A similar thing to the Reddit post mentioned in the article happened to me too: I used a not-a-space character that looks like a space once, the text editor autocompletion remembered it and would occasionally substitute it for space. The code looked OK, but compilation failed or threw syntax errors in run time. This continued for several years until I completely reinstalled the editor, with full cleanup.
- Enginerrrd 5y ago
  Oh my God. That's nightmare level error-inducing.
- yepthatsreality 5y ago
  Glances at Microsoft GitHub Copilot.
- Arrath 5y ago
  Auto-complete gremlins can be the absolute worst, I would have been tearing my hair out.
- bavell 5y ago
  Dev nightmare fuel
More-nitors 5y ago
maybe someone should make a linter for this...
- pabs3 5y ago
  There is one for Go called glyphcheck: https://github.com/NebulousLabs/glyphcheck https://github.com/NebulousLabs/glyphcheck
- trevinhofmann 5y ago
  Added to my own ESLint config: https://github.com/trevinhofmann/eslint-config-principled/pull/5/files https://github.com/trevinhofmann/eslint-config-principled/pu...
jabbany 5y ago
IIRC Rust has some compiler-level defenses against these glyph based attacks (ref: https://twitter.com/skyslasher11/status/1152824207555698688 https://twitter.com/skyslasher11/status/1152824207555698688) Perhaps one could do something similar in JS as well. Like have a config that will make an interpreter fail if it encounters unescaped unicode in variable names. It does not prevent any unicode variable names, but you just have to escape them if the are from some list of "abusable characters". (At least Chrome seems to be happy with `var \u6D4B\u8BD5 = 1;`)
- goldsteinq 5y ago
  You can just do `#![forbid(non_ascii_idents)]` in Rust. It'll prevent this kind of attacks completely and you shouldn't need non-ASCII idents anyway.
  - Tepix 5y ago
    That seems like throwing out the baby with the bathwater. We don't all want to go back to the IT stone age of 1963.
    - lifthrasiir 5y ago
      `#![forbid(...)]` is a crate-wide attribute, so it is more like a policy (that is good to have if your code would be entirely ASCII).
    - Ygg2 5y ago
      Agreed. I use Unicode identifiers to spot shitcode. This would really hamper my detection abilities. Jokes aside, if you're writing Unicode identifiers it means you're not writing your code to be read by a broad audience.
    - goldsteinq 5y ago
      You still can use Unicode in comments and string literals. You just can't use non-ASCII characters in identifiers. Unicode in identifiers is just a bad idea. 1. It creates a security consideration with confusable identifiers (and lints don't always catch these) 2. It breaks tooling with RTL identifiers 3. It may not render correctly depending on fonts 4. It may be hard to type depending on keyboard layout 5. There really isn't a good reason to use non-ASCII idents anyway
nosianu 5y ago
First thing I did when I first read the story was check my editor. I already had the "Zero Width Characters locator" plugin installed, but that covered less than a handful of specific space character type codes. Still, the result was good: Looks like IDEA editors like Webstorm show invisible characters with colored background and a warning. My test was from that first article and also now from this one copy the example code they contained or linked to from the browser into an open file. Screenshot: https://i.imgur.com/ColuRNB.png https://i.imgur.com/ColuRNB.png
- kingcharles 5y ago
  Is there a plugin for detecting homoglyphs like these genderless vs. male zombies I made: https://kingcharles.one/unistrange.html https://kingcharles.one/unistrange.html
- dotancohen 5y ago
  Interesting. PhpStorm highlights the variable after `timeout` but does not highlight the variable after `http://example.com/ http://example.com/`. Even pressing F2 to go to the next error goes to the first variable (the highlighted one) but not the second. However, placing the cursor on either does highlight the second. I'm using the Darcula scheme. Your screenshot obscures the second occurrence, so we cannot see if your light theme has the same issue with the second occurrence not being highlighted as Darcula has. Screenshot: https://i.imgur.com/FxwUkVz.png https://i.imgur.com/FxwUkVz.png
  - nosianu 5y ago
    You are right, I missed the other one, it is not reported. You can see there is something because it takes space, but you have to deliberately go there to see it. There also is no warning from having the "No trailing spaces" setting active, so it is not seen as a space character even if it shows as such. I'll write an Issue on youtrack, I'm sure they'll fix it. From the well over hundred issues I ever reported about 2/3rd were fixed (rest is obsolete, only a few that are really still open). EDIT: Bug report submitted.
vjeux 5y ago
Another reason to use prettier, this will be formated in a confusing way and has a higher chance of being spotted by a human!
gostsamo 5y ago
The benefit of being blind: the screen reader announces invisible characters and I could detect the invisible variable.
- IceWreck 5y ago
  How hard is it to program while being blind ? What sort of development do you do ? i understand that frontend is impossible but what other difficulties do you face ? Are indent based langauges like python harder than bracket based languages ?
  - gostsamo 5y ago
    Hi, Front end is not entirely impossible, but impossible on doing pixel perfect designs. Otherwise, I know blind people who do FE, not sure if most of it is professional though. Indent based languages are actually easier. Every screen reader has a way to announce indentation in code, while brackets could be confusing if not formatted or verbose if properly announced. My main issues are dev tools with bad accessibility. Also, it takes me more time to get acquainted with new code and sometimes omophones in the source code which require extra attention. Filtering through logs is also a bitch in most cases. Besides the dev tools, you can summarize the rest as bad IO speed.
    - IceWreck 5y ago
      Hey, thats cool. Thank you.
    - akavel 5y ago
      Do you have some tricks for how you handle filtering through logs? Or some ideas if there could be a tool that could help you or mitigate your most critical issue[s]? I found filtering through longs a major pain even for a fully sighted person like me, so I wrote a tool to help me with that, but it's fully in a "TUI" paradigm (i.e. curses-like), so I presume it wouldn't help you much (https://github.com/akavel/up https://github.com/akavel/up). No promises, given that the tool as is scratched my itch, but I am honestly curious if something similar could reduce your PITA, including whether this specific tool could be made useful for you through some minimal effort on my side.
josteink 5y ago
Listen guys, don't get me wrong. As someone with Ø in my name, and both Å and Ø in my address, don't get me started on poorly written systems which cannot handle unicode properly. I've seen my name and address mangled in shipping forms, in airline tickets (every time) and even in my marriage-papers since I married abroad. I literally have personal reasons for getting everyone, and I mean everyone, on the unicode bandwagon. That said... Maybe it's because I'm a child of the late 70s and early 80s and learned to program on computers which simply didn't have non-ASCII characters at all... But can't we all just sit down and admit that allowing non-ASCII characters in programming-language identifiers was a bad idea? Can't we in the next revision of EcmaScript (or Rust, or whatever) mandate ASCII-only identifiers when in strict mode or using modules or whatever? Having invisible characters represent executable code is not just a dumb a idea, it's so hazardous that you might call it borderline malicious. There has to be some way to undo this damage, without breaking compatibility with the code which is already out there, right?
- Dagonfly 5y ago
  Adding a variable decorator/annotation like @Unicode(german,french) would be a good stop-gap. You could only use ASCII characters unless you specified the script that you want to use. One could even set a max limit on how many scripts per variable. Because while I have used German characters in variables before (only if I'm referring to some law or spec), I never had a use case for more than 2 scripts within one variable.
  - lifthrasiir 5y ago
    For your information the relevant Unicode specification is the Script_Extensions property [1]. (You can't easily filter by languages, so you should filter by scripts.) [1] https://www.unicode.org/reports/tr24/tr24-32.html#Script_Extensions https://www.unicode.org/reports/tr24/tr24-32.html#Script_Ext...
  - est31 5y ago
    The multiple scripts per variable thing is implemented in Rust via a lint. For the explicit enabling of single scripts, I have suggested that for Rust, but sadly people preferred allowing all identifiers (while giving an option to only have ascii but I'd argue this is unfair for anyone who only wants to use a specific non-ascii language, why do they have to suddenly allow all languages in their code base?). There are also practical concerns, like who says what a language is, which characters it contains, how that language is called, etc? Someone has to maintain all these lists.
fergie 5y ago
Running prettier on the code makes the "hidden" variables fairly obvious -> https://imgur.com/a/MhhRpwq https://imgur.com/a/MhhRpwq That said, nothing on my buildchain actually throws an error or warning.
- brabel 5y ago
  Obvious as in an empty line? Not very obvious to me.
- DyslexicAtheist 5y ago
  >> That said, nothing on my buildchain actually throws an error or warning. use hooks for CI on pre-commit / merge and pull requests e.g. like this pre-commit which would catch bi-directional trojan sources: #!/usr/bin/env python3 import sys import subprocess bidi_chars = '\u202A\u202B\u202D\u202E\u2066\u2067\u2068\u202C\u2069' for line in sys.stdin: old, new, ref = line.split() diff = subprocess.run(['git', 'diff', old, new], stdout=subprocess.PIPE, stderr=subprocess.STDOUT, text=True) if diff.returncode != 0: print(diff.stdout) sys.exit(f'git diff ended with rc={diff.returncode}, receive TERMINATED') if any(c in diff.stdout for c in bidi_chars): print(diff.stdout) sys.exit('Possible Trojan Source Attack, receive REFUSED') I wish github/gitlab would provide such features available out of the box which also follow best practice, so people can stop pasting them from the web or reinvent our own version in every team ...
onion2k 5y ago
If you combined this attack with Whitespace (https://en.wikipedia.org/wiki/Whitespace_(programming_language) https://en.wikipedia.org/wiki/Whitespace_(programming_langua...) you could embed entire programs in your JS code.
- tyingq 5y ago
  Perl has a module called Acme::Bleach that does that for you.
est 5y ago
IO operations, especially involving a subprocess is prone to have backdoors. They are practically unverifiable. I've yet to see a unicode backdoor in pure algorithmic flows.
- willvarfar 5y ago
  Some examples of historic attacks you could embed in algorithms: “Salami slicing” is a kind of embezzlement where eg an insider programs the computer to credit small amounts to the last account (and then opens an account with a name beginning with Z). In the 90s there was a massive hushed up scandal where the programmers developing the early Barclaycard made the pseudo random number generator for pin codes just issue three distinct pins. This meant that a stolen card could be easily used because they could guess any pin in three goes before the ATM swallowed the card. This is hardly an exhaustive list. It’s just to get peoples cogs turning… :)
  - null_object 5y ago
    > In the 90s there was a massive hushed up scandal where the programmers developing the early Barclaycard made the pseudo random number generator for pin codes just issue three distinct pins. This meant that a stolen card could be easily used because they could guess any pin in three goes before the ATM swallowed the card. Citation for this?
    - willvarfar 5y ago
      Took some digging to find any working links these days. The three pin thing is on page two but it doesn’t name which bank; I may have misremembered and it might not have been Barclays. The whole article is a good starting point for digging into other vulnerabilities and exploits too https://www.theregister.com/2005/10/21/phantoms_and_rogues/ https://www.theregister.com/2005/10/21/phantoms_and_rogues/
sihox 5y ago
Just being curious I've pasted the example to Geany and VSCode and in both this invisible character was visible :) I can't remember setting some special character / whitespace visibility options but I think it is good to have this kind of options always on.
laktak 5y ago
This shows up in standard Vim as a [HF] symbol.
- kreetx 5y ago
  Same in vanilla emacs.
mirekrusin 5y ago
I don’t think this is JavaScript specific. It’s like saying “BMW cars in Paris stop working if you pour sugar in tank”.
- bestham 5y ago
  The point is not that the vulnerability is a trait of javascript but to make a demonstration on how different unicode characters can be used to create a vulnerability, exemplified by a piece of javascript.
  - mirekrusin 5y ago
    Yes, I understand it, but title and content doesn't explicitly mention this "detail" that same attact vector exists for other languages and data formats.
- dspillett 5y ago
  > I don’t think this is JavaScript specific. It isn't, any relatively dynamic language is going to have these or similar issues. Many moons ago I saw similar examples in bash, I'm sure they are possible in PHP, ..., ..., ... In fact, even the more strict languages probably do to: the “accidentally run something malicious via care-free use of exec” is an issue in just every language that has “exec”/similar - it is a data trusting error in the programmer's logic not an issue with the language itself. The dynamic nature of some of JS's syntax is just one way to pollute the data being fed to exec amongst the other sources (user input, being too trusting of config in the DB or filesystem, and so forth). Javascript is a very good option to use for examples though: most devs know it well enough and it is everywhere so the potential scale of the danger is obvious, even more so in light of people being far too trusting of dependencies pulled via NPM and the recent examples of malicious updates getting into common packages. Maybe the title could be a bit less click-baity, though I'm not sure what would be used instead that wouldn't be overly wordy for a punchy article title.
  - fergie 5y ago
    Strong typing doesn't fix the issue that invisible characters can be used as variable names.
Eriks 5y ago
good reason to not use comma when destructuring an object
- smhg 5y ago
  You mean the last trailing comma? Or not destructions into multiple variables?
aww_dang 5y ago
Yes, the Unicode characters are a problem. But do the norms and tooling play a role here as well? Explicitly casting types, like String parameters to integers would make this much more explicit. The convenience of accessing parameters via destructuring, vs explicitly request.getParameter("\u3164"). Having a static array of permissible commands declared elsewhere. There's something to be said for verbosity and explicitness. Where the tooling and norms shun it, these 'invisible' backdoors can gain advantage.
lifthrasiir 5y ago
Hey, you have missed U+FFA0 HALFWIDTH HANGUL FILLER which has about the same property as U+3164 HANGUL FILLER! Surely I expected this coming ever since I've seen the purported Trojan "attack", as the Hangul fillers are pretty much the only characters that are (X)ID_Start and have no visible glyphs [1]. If (X)ID_Continue is also considered ZWJ and ZWNJ would be another contenders. Attacks using those characters have much better chance than the Trojan "attack", but you need a very specific code to execute the attack. It should be obvious that a typical coding convention easily prevents them. As much like the purported Trojan "attack", this kind of attacks need a better code review and tooling. You don't need to remove non-ASCII identifiers from existing languages: they have their uses when an entirety of your team speak languages not using Latin script. But you should be able to catch a new use of non-ASCII characters throughout your code base and compare that with your expectation. [1] The Hangul filler comes from a legacy mechanism of KS X 1001 for unencoded Hangul syllables (it had only 2,350 out of 11,172 modern syllables). The half-width Hangul filler probably comes from a duplicate encoding of the filler in the IBM code page 933 to ensure round-trip conversion. Both are never used in practice, except for probably the Hangul filler that was briefly implemented by Mozilla and removed due to the compatibility issue.
rafaelturk 5y ago
Nice article, it was a fund reading it. Albeit I don't think this kind of attack is restricted to just JavaScript.
nathell 5y ago
What was wrong with only allowing ASCII in identifiers?
- jgalt212 5y ago
  seriously, I'd fire anyone who put any emoji in an identifier.
  - chrismorgan 5y ago
    Sensible languages follow UAX #31 <https://www.unicode.org/reports/tr31/ https://www.unicode.org/reports/tr31/> for Unicode identifiers, which doesn’t allow emoji.
    - skrebbel 5y ago
      But it does allow hangul fillers, apparently.
speleding 5y ago
I think the recommendation to disallow any non-ASCII character is throwing out the baby with the bathwater. How about code that wants to display some emojis? It would be cumbersome to use hex unicode everywhere. And while localisations should typically happen in a separate language file, it's very common to want some text in code intended for a single audience. Blocking all the confusables might be tricky, and an allow list would be endless. Perhaps some magic pre-processor comment that says "allow unicode in this file".
- YetAnotherNick 5y ago
  Do you really have to write emoji in the code string? Similarly with international language characters. The sane thing is to use either json config files or i18n libraries.
  - speleding 5y ago
    If you are writing something intended for a single audience using i18n libraries can be unnecessary overhead. And emoji can also be icons like ⌘ that can be useful to display in the UI.
- josteink 5y ago
  > I think the recommendation to disallow any non-ASCII character is throwing out the baby with the bathwater. Not throwing out all non-ASCII characters from code-files. Just throwing them out as being invalid identifiers in your code (think variables, function-names, etc). > How about code that wants to display some emojis? Fine. You quote that emoji in a string, and it's golden. You try to make a variable with the name of an emoji however, you code crashes. That sounds fine to me.
  - speleding 5y ago
    That would close this particular attack (but not the BIDI one the article mentions). But there is probably already too much code out there with π=3.14 in it to be feasible to do this.
robertrbairdii 5y ago
There's definitely a benefit to using a linter and a tool such as prettier. Using prettier pushes the hidden character onto an additional line in the checkCommands array which makes it much easier to spot that something is wrong even if you're not using the trailingCommas setting. https://imgur.com/a/gYKylyH https://imgur.com/a/gYKylyH I think this eslint rule would also be able to defend against the initial destructuring of the query object by defining a regex that identifiers have to match which would exclude those invisible characters https://eslint.org/docs/rules/id-match https://eslint.org/docs/rules/id-match
- Andys 5y ago
  Indeed, this wouldn't even compile in Go.
reilly3000 5y ago
While this is an interesting hack, the larger issue in the example is allowing any query parameter to write into a subprocess. exec() immediately throws flags for me, especially when it isn’t necessary like in the case of making an http call. Even when it isn’t passing arbitrary inputs from the web to the command line, it’s susceptible to DoS that could crash the whole kernel instead of just the web server. I get that this is just a contrived example to show the risk of hidden characters, but please don’t use process.exec() unless you have no other options.
SenpaiHurricane 5y ago
TS + Intellij https://ibb.co/MfLLNQL https://ibb.co/MfLLNQL
kuon 5y ago
My editor (vim) will warn me with a loud visual red block for any non ascii char outside a string literal. But I do not think that is enough. Compiler and interpreter must be more strict.
- jrochkind1 5y ago
  I'm surprised VS Code doesn't at least have that option. (Or does it?)
  - myfonj 5y ago
    It has `editor.renderControlCharacters` but only recently started displaying few dangerous previously invisible ones (directional overrides) natively [1], but besides that you had to use extension that adds highlights for non-ascii non-whitelisted [2] or predefined [3] characters. [1] https://github.com/microsoft/vscode/issues/116939 https://github.com/microsoft/vscode/issues/116939 [2] https://marketplace.visualstudio.com/items?itemName=nachocab.highlight-dodgy-characters https://marketplace.visualstudio.com/items?itemName=nachocab... [3] https://marketplace.visualstudio.com/items?itemName=nhoizey.gremlins https://marketplace.visualstudio.com/items?itemName=nhoizey....
- tomxor 5y ago
  Mind sharing the relevant config line? I thought this was default but just realised it only does the <FFFF> thing when there is no printable glyph available. Allowing printable unicode in strings seems like a nice balance if it can be done reliably.
  - fatheart 5y ago
    After seeing this thread I added the following to my vimrc: highlight link NonASCII Error autocmd Syntax * :syntax match NonASCII "[^\d0-\d127]" Obviously haven't been using it long, and I'm not confident enough in my vim knowledge to vouch for its correctness, but it works in the limited amount of scenarios I tested so far.
  - kuon 5y ago
cyberpsybin 5y ago
This needs to be patched. Although at this point there might be code depending on this.
ludovicianul 5y ago
On a similar note, if you want to test your REST APIs for weird characters, I built a tool for this: (https://github.com/Endava/cats#leadingcontrolcharsinfieldstrimvalidatefuzzer-trailingcontrolcharsinfieldstrimvalidatefuzzer-and-onlycontrolcharsinfieldstrimvalidatefuzzer https://github.com/Endava/cats#leadingcontrolcharsinfieldstr...)
geoduck14 5y ago
This is MOST interesting. I wonder if Git or Stack Overflow should highlight non Ascii characters to reduce malicious actors using this in code.
FrankyHollywood 5y ago
> In our experience non-ASCII characters are pretty rare in code. Many development teams chose to use English as the primary development language Is this true for the whole world, or just Europe/US?
- utrack 5y ago
  Yep, pretty much (Russian here)
- afavour 5y ago
  It is. The way it was explained to me, all the APIs you use are in English so naming variables in your local language is futile at best and would just require constant context switching. I remember many moons ago MooTools announced international API translations as an April Fools joke. It did make me wonder if there’s an interesting programming experiment to be done there… but I’m a native English speaker so I’m not best positioned to know!
  - mijamo 5y ago
    Not my experience. Plenty of codebases have variables in local language in France Germany and Sweden (where I have experience). I actually have encountered a lot of problem with English codebases in those countries as they often try to translate regional concepts that are not directly translatable. This is particularly annoying when it comes to administrative stuff where one English word can refer to different local concepts (ex: geographical divisions of the territory) and translations always clumsy. I have even seen nasty bugs come from there, where a "county" had a different meaning in different places of the code as different teams had different idea of what a county was but didn't discuss it.
lovasoa 5y ago
The `cmd &&` looks fishy in their example, and would probably have been removed in a review. Instead, one could write : const { ping, curl, ㅤ } = req.query; const checkCommands = [ ping && 'ping -c 1 google.com', curl && 'curl -s http://example.com/', ㅤ ]; await Promise.all(checkCommands.map(cmd => cmd && exec(cmd, { timeout: 5_000 }))); This way the `cmd &&` is justified
- testASW2 5y ago
  d
Mockapapella 5y ago
If anyone is interested, I wrote an article a while back exploring which unicode characters Python allows you to set variables equal to: https://www.thelisowe.com/why-can-be-a-variable-in-python-but-not/ https://www.thelisowe.com/why-can-be-a-variable-in-python-bu... This was originally done with the goal of trying to hide/encode one program within another using non-displayable characters (such as zero width spaces), I just never got around to it. But reading this article has kind of reignited that interest for me and I think I might take another crack at that soon.
wnevets 5y ago
If you use Sublime Text the Gremlins[0] package will detect and light up these kind of characters [0] https://packagecontrol.io/packages/Gremlins https://packagecontrol.io/packages/Gremlins
Cameri 5y ago
Is there an eslint plugin to prevent invisible characters on .js/.ts files?
greyhair 5y ago
From what I have been reading lately, web3 will fix all of this. \s
egberts1 5y ago
Take two of those Hangu (“blank” and “half-blank”) characters you make two variable names… Couple it with JS obfuscator… (something, something)… Profit!
ewagsjr 5y ago
They put the answer in quotes right after they ask if you can find it. Highlight it and hit control + f.
Lorin 5y ago
I use the "nhoizey.gremlins" VSC extension which would cover these.
rurban 5y ago
A, the famous hangul filler. That's actually a Unicode bug they refuse to fix for some years now. It's still listed as identifier. I fixed that in my interpreter cperl. The next bugs are actually all JavaScript bugs, as they accept Unicode identifiers but don't check against the Unicode security guidelines, ignoring any profile. Accepting bidi, mixed scripts, unnormalized identifiers. This is very common, 99% of all interpreters and compilers don't care about Unicode security at all. They are rather proud to accept everyone, and point fingers at colleagues who only accept ASCII english. Identifiers need to be identifiable by a human. That's the whole point. And the system needs to block illegal identifiers. Similar to filesystem drivers, which consists of path names as identifiers, but the driver writers think they are beyond such human issues. For them there is only garbage in, garbage out. Their pathnames are certainly not identiable. A directory can consist of bidi names, or Russian and Greek mixed scripts who all look the same. Or just not normalized. There can be a multiple of visually duplicate names, and you never know which is which. At least with domain names they came up with a punycode solution, but this was only the tip on the iceberg. And it was a rather awkward workaround.

573 points by davidbarker 5y ago

dlsa 5y ago

Compilers and interpreters need a new pass to detect these characters in code and treat them as hard errors. This doesn't stop their use in comments where presumably they are still ok. Alternatively, there needs to be an uptake of the use of code linters and pretty printers. A bit of both perhaps.

Suvitruf 5y ago

This issue and example used are more about data validation and escaping characters.

cphoover 5y ago

I imagine this can be defended again pretty easily with a lint rule that prevents these unicode characters in variables. pretty ingenious little hack though. The eslint rule id-match, which require identifiers to match a specified regular expression, would be useful here. For example: "id-match": ["error", "^[a-z]+([A-Z][a-z]+)*$"]

bluepnume 5y ago
A malicious PR could also add the character to your eslintrc too though. You'd be forgiven for seeing the line change in the diff and thinking it was just some reformatting.
- jraph 5y ago
  It seems like things displaying diffs could use a specific color for lines only changed by formatting or indentation (indentation can have significant meaning like in Python but this would probably be good enough)
  - dotancohen 5y ago
    I believe that Git diff - which has features not supported by regular diff such as --word-diff - can differentiate between whitespace-only line changes. The Jetbrains IDE, which I believe uses Git diff behind the scenes, will show who originally wrote a line even if it has been whitespace-reformatted later.
- cphoover 5y ago
  that would show up on a diff and would elicit a question in code review hopefully of why the .eslintrc file was being changed in this way. This another good argument for a comprehensive code review process. also you could lock this file down with a CODEOWNERS file so only certain trusted contributors could modify the lint configuration. You could also do exclusionary pattern matching to make sure none of the bad characters do exist in identifier names... Or you could write your eslint configuration as a separate module to be npm installed... or you could write a eslint rule plugin that disallows non-ascii identifiers and then npm install that... lots of different ways to skin this cat to add security.

Klaster_1 5y ago

A similar thing to the Reddit post mentioned in the article happened to me too: I used a not-a-space character that looks like a space once, the text editor autocompletion remembered it and would occasionally substitute it for space. The code looked OK, but compilation failed or threw syntax errors in run time. This continued for several years until I completely reinstalled the editor, with full cleanup.

Enginerrrd 5y ago
Oh my God. That's nightmare level error-inducing.
yepthatsreality 5y ago
Glances at Microsoft GitHub Copilot.
Arrath 5y ago
Auto-complete gremlins can be the absolute worst, I would have been tearing my hair out.
bavell 5y ago
Dev nightmare fuel

More-nitors 5y ago

maybe someone should make a linter for this...

pabs3 5y ago
There is one for Go called glyphcheck: https://github.com/NebulousLabs/glyphcheck https://github.com/NebulousLabs/glyphcheck
trevinhofmann 5y ago
Added to my own ESLint config: https://github.com/trevinhofmann/eslint-config-principled/pull/5/files https://github.com/trevinhofmann/eslint-config-principled/pu...

jabbany 5y ago

IIRC Rust has some compiler-level defenses against these glyph based attacks (ref: https://twitter.com/skyslasher11/status/1152824207555698688 https://twitter.com/skyslasher11/status/1152824207555698688) Perhaps one could do something similar in JS as well. Like have a config that will make an interpreter fail if it encounters unescaped unicode in variable names. It does not prevent any unicode variable names, but you just have to escape them if the are from some list of "abusable characters". (At least Chrome seems to be happy with `var \u6D4B\u8BD5 = 1;`)

goldsteinq 5y ago
You can just do `#![forbid(non_ascii_idents)]` in Rust. It'll prevent this kind of attacks completely and you shouldn't need non-ASCII idents anyway.
- Tepix 5y ago
  That seems like throwing out the baby with the bathwater. We don't all want to go back to the IT stone age of 1963.
  - lifthrasiir 5y ago
    `#![forbid(...)]` is a crate-wide attribute, so it is more like a policy (that is good to have if your code would be entirely ASCII).
  - Ygg2 5y ago
    Agreed. I use Unicode identifiers to spot shitcode. This would really hamper my detection abilities. Jokes aside, if you're writing Unicode identifiers it means you're not writing your code to be read by a broad audience.
  - goldsteinq 5y ago
    You still can use Unicode in comments and string literals. You just can't use non-ASCII characters in identifiers. Unicode in identifiers is just a bad idea. 1. It creates a security consideration with confusable identifiers (and lints don't always catch these) 2. It breaks tooling with RTL identifiers 3. It may not render correctly depending on fonts 4. It may be hard to type depending on keyboard layout 5. There really isn't a good reason to use non-ASCII idents anyway

nosianu 5y ago

First thing I did when I first read the story was check my editor. I already had the "Zero Width Characters locator" plugin installed, but that covered less than a handful of specific space character type codes. Still, the result was good: Looks like IDEA editors like Webstorm show invisible characters with colored background and a warning. My test was from that first article and also now from this one copy the example code they contained or linked to from the browser into an open file. Screenshot: https://i.imgur.com/ColuRNB.png https://i.imgur.com/ColuRNB.png

kingcharles 5y ago
Is there a plugin for detecting homoglyphs like these genderless vs. male zombies I made: https://kingcharles.one/unistrange.html https://kingcharles.one/unistrange.html
dotancohen 5y ago
Interesting. PhpStorm highlights the variable after `timeout` but does not highlight the variable after `http://example.com/ http://example.com/`. Even pressing F2 to go to the next error goes to the first variable (the highlighted one) but not the second. However, placing the cursor on either does highlight the second. I'm using the Darcula scheme. Your screenshot obscures the second occurrence, so we cannot see if your light theme has the same issue with the second occurrence not being highlighted as Darcula has. Screenshot: https://i.imgur.com/FxwUkVz.png https://i.imgur.com/FxwUkVz.png
- nosianu 5y ago
  You are right, I missed the other one, it is not reported. You can see there is something because it takes space, but you have to deliberately go there to see it. There also is no warning from having the "No trailing spaces" setting active, so it is not seen as a space character even if it shows as such. I'll write an Issue on youtrack, I'm sure they'll fix it. From the well over hundred issues I ever reported about 2/3rd were fixed (rest is obsolete, only a few that are really still open). EDIT: Bug report submitted.

vjeux 5y ago

Another reason to use prettier, this will be formated in a confusing way and has a higher chance of being spotted by a human!

gostsamo 5y ago

The benefit of being blind: the screen reader announces invisible characters and I could detect the invisible variable.

IceWreck 5y ago
How hard is it to program while being blind ? What sort of development do you do ? i understand that frontend is impossible but what other difficulties do you face ? Are indent based langauges like python harder than bracket based languages ?
- gostsamo 5y ago
  Hi, Front end is not entirely impossible, but impossible on doing pixel perfect designs. Otherwise, I know blind people who do FE, not sure if most of it is professional though. Indent based languages are actually easier. Every screen reader has a way to announce indentation in code, while brackets could be confusing if not formatted or verbose if properly announced. My main issues are dev tools with bad accessibility. Also, it takes me more time to get acquainted with new code and sometimes omophones in the source code which require extra attention. Filtering through logs is also a bitch in most cases. Besides the dev tools, you can summarize the rest as bad IO speed.
  - IceWreck 5y ago
    Hey, thats cool. Thank you.
  - akavel 5y ago
    Do you have some tricks for how you handle filtering through logs? Or some ideas if there could be a tool that could help you or mitigate your most critical issue[s]? I found filtering through longs a major pain even for a fully sighted person like me, so I wrote a tool to help me with that, but it's fully in a "TUI" paradigm (i.e. curses-like), so I presume it wouldn't help you much (https://github.com/akavel/up https://github.com/akavel/up). No promises, given that the tool as is scratched my itch, but I am honestly curious if something similar could reduce your PITA, including whether this specific tool could be made useful for you through some minimal effort on my side.

josteink 5y ago

Listen guys, don't get me wrong. As someone with Ø in my name, and both Å and Ø in my address, don't get me started on poorly written systems which cannot handle unicode properly. I've seen my name and address mangled in shipping forms, in airline tickets (every time) and even in my marriage-papers since I married abroad. I literally have personal reasons for getting everyone, and I mean everyone, on the unicode bandwagon. That said... Maybe it's because I'm a child of the late 70s and early 80s and learned to program on computers which simply didn't have non-ASCII characters at all... But can't we all just sit down and admit that allowing non-ASCII characters in programming-language identifiers was a bad idea? Can't we in the next revision of EcmaScript (or Rust, or whatever) mandate ASCII-only identifiers when in strict mode or using modules or whatever? Having invisible characters represent executable code is not just a dumb a idea, it's so hazardous that you might call it borderline malicious. There has to be some way to undo this damage, without breaking compatibility with the code which is already out there, right?

Dagonfly 5y ago
Adding a variable decorator/annotation like @Unicode(german,french) would be a good stop-gap. You could only use ASCII characters unless you specified the script that you want to use. One could even set a max limit on how many scripts per variable. Because while I have used German characters in variables before (only if I'm referring to some law or spec), I never had a use case for more than 2 scripts within one variable.
- lifthrasiir 5y ago
  For your information the relevant Unicode specification is the Script_Extensions property [1]. (You can't easily filter by languages, so you should filter by scripts.) [1] https://www.unicode.org/reports/tr24/tr24-32.html#Script_Extensions https://www.unicode.org/reports/tr24/tr24-32.html#Script_Ext...
- est31 5y ago
  The multiple scripts per variable thing is implemented in Rust via a lint. For the explicit enabling of single scripts, I have suggested that for Rust, but sadly people preferred allowing all identifiers (while giving an option to only have ascii but I'd argue this is unfair for anyone who only wants to use a specific non-ascii language, why do they have to suddenly allow all languages in their code base?). There are also practical concerns, like who says what a language is, which characters it contains, how that language is called, etc? Someone has to maintain all these lists.

fergie 5y ago

Running prettier on the code makes the "hidden" variables fairly obvious -> https://imgur.com/a/MhhRpwq https://imgur.com/a/MhhRpwq That said, nothing on my buildchain actually throws an error or warning.

brabel 5y ago
Obvious as in an empty line? Not very obvious to me.
DyslexicAtheist 5y ago
>> That said, nothing on my buildchain actually throws an error or warning. use hooks for CI on pre-commit / merge and pull requests e.g. like this pre-commit which would catch bi-directional trojan sources: #!/usr/bin/env python3 import sys import subprocess bidi_chars = '\u202A\u202B\u202D\u202E\u2066\u2067\u2068\u202C\u2069' for line in sys.stdin: old, new, ref = line.split() diff = subprocess.run(['git', 'diff', old, new], stdout=subprocess.PIPE, stderr=subprocess.STDOUT, text=True) if diff.returncode != 0: print(diff.stdout) sys.exit(f'git diff ended with rc={diff.returncode}, receive TERMINATED') if any(c in diff.stdout for c in bidi_chars): print(diff.stdout) sys.exit('Possible Trojan Source Attack, receive REFUSED') I wish github/gitlab would provide such features available out of the box which also follow best practice, so people can stop pasting them from the web or reinvent our own version in every team ...

onion2k 5y ago

If you combined this attack with Whitespace (https://en.wikipedia.org/wiki/Whitespace_(programming_language) https://en.wikipedia.org/wiki/Whitespace_(programming_langua...) you could embed entire programs in your JS code.

tyingq 5y ago
Perl has a module called Acme::Bleach that does that for you.

est 5y ago

IO operations, especially involving a subprocess is prone to have backdoors. They are practically unverifiable. I've yet to see a unicode backdoor in pure algorithmic flows.

willvarfar 5y ago
Some examples of historic attacks you could embed in algorithms: “Salami slicing” is a kind of embezzlement where eg an insider programs the computer to credit small amounts to the last account (and then opens an account with a name beginning with Z). In the 90s there was a massive hushed up scandal where the programmers developing the early Barclaycard made the pseudo random number generator for pin codes just issue three distinct pins. This meant that a stolen card could be easily used because they could guess any pin in three goes before the ATM swallowed the card. This is hardly an exhaustive list. It’s just to get peoples cogs turning… :)
- null_object 5y ago
  > In the 90s there was a massive hushed up scandal where the programmers developing the early Barclaycard made the pseudo random number generator for pin codes just issue three distinct pins. This meant that a stolen card could be easily used because they could guess any pin in three goes before the ATM swallowed the card. Citation for this?
  - willvarfar 5y ago
    Took some digging to find any working links these days. The three pin thing is on page two but it doesn’t name which bank; I may have misremembered and it might not have been Barclays. The whole article is a good starting point for digging into other vulnerabilities and exploits too https://www.theregister.com/2005/10/21/phantoms_and_rogues/ https://www.theregister.com/2005/10/21/phantoms_and_rogues/

sihox 5y ago

Just being curious I've pasted the example to Geany and VSCode and in both this invisible character was visible :) I can't remember setting some special character / whitespace visibility options but I think it is good to have this kind of options always on.

laktak 5y ago

This shows up in standard Vim as a [HF] symbol.

kreetx 5y ago
Same in vanilla emacs.

mirekrusin 5y ago

I don’t think this is JavaScript specific. It’s like saying “BMW cars in Paris stop working if you pour sugar in tank”.

bestham 5y ago
The point is not that the vulnerability is a trait of javascript but to make a demonstration on how different unicode characters can be used to create a vulnerability, exemplified by a piece of javascript.
- mirekrusin 5y ago
  Yes, I understand it, but title and content doesn't explicitly mention this "detail" that same attact vector exists for other languages and data formats.
dspillett 5y ago
> I don’t think this is JavaScript specific. It isn't, any relatively dynamic language is going to have these or similar issues. Many moons ago I saw similar examples in bash, I'm sure they are possible in PHP, ..., ..., ... In fact, even the more strict languages probably do to: the “accidentally run something malicious via care-free use of exec” is an issue in just every language that has “exec”/similar - it is a data trusting error in the programmer's logic not an issue with the language itself. The dynamic nature of some of JS's syntax is just one way to pollute the data being fed to exec amongst the other sources (user input, being too trusting of config in the DB or filesystem, and so forth). Javascript is a very good option to use for examples though: most devs know it well enough and it is everywhere so the potential scale of the danger is obvious, even more so in light of people being far too trusting of dependencies pulled via NPM and the recent examples of malicious updates getting into common packages. Maybe the title could be a bit less click-baity, though I'm not sure what would be used instead that wouldn't be overly wordy for a punchy article title.
- fergie 5y ago
  Strong typing doesn't fix the issue that invisible characters can be used as variable names.

Eriks 5y ago

good reason to not use comma when destructuring an object

smhg 5y ago
You mean the last trailing comma? Or not destructions into multiple variables?

aww_dang 5y ago

Yes, the Unicode characters are a problem. But do the norms and tooling play a role here as well? Explicitly casting types, like String parameters to integers would make this much more explicit. The convenience of accessing parameters via destructuring, vs explicitly request.getParameter("\u3164"). Having a static array of permissible commands declared elsewhere. There's something to be said for verbosity and explicitness. Where the tooling and norms shun it, these 'invisible' backdoors can gain advantage.

lifthrasiir 5y ago

Hey, you have missed U+FFA0 HALFWIDTH HANGUL FILLER which has about the same property as U+3164 HANGUL FILLER! Surely I expected this coming ever since I've seen the purported Trojan "attack", as the Hangul fillers are pretty much the only characters that are (X)ID_Start and have no visible glyphs [1]. If (X)ID_Continue is also considered ZWJ and ZWNJ would be another contenders. Attacks using those characters have much better chance than the Trojan "attack", but you need a very specific code to execute the attack. It should be obvious that a typical coding convention easily prevents them. As much like the purported Trojan "attack", this kind of attacks need a better code review and tooling. You don't need to remove non-ASCII identifiers from existing languages: they have their uses when an entirety of your team speak languages not using Latin script. But you should be able to catch a new use of non-ASCII characters throughout your code base and compare that with your expectation. [1] The Hangul filler comes from a legacy mechanism of KS X 1001 for unencoded Hangul syllables (it had only 2,350 out of 11,172 modern syllables). The half-width Hangul filler probably comes from a duplicate encoding of the filler in the IBM code page 933 to ensure round-trip conversion. Both are never used in practice, except for probably the Hangul filler that was briefly implemented by Mozilla and removed due to the compatibility issue.

rafaelturk 5y ago

Nice article, it was a fund reading it. Albeit I don't think this kind of attack is restricted to just JavaScript.

nathell 5y ago

What was wrong with only allowing ASCII in identifiers?

jgalt212 5y ago
seriously, I'd fire anyone who put any emoji in an identifier.
- chrismorgan 5y ago
  Sensible languages follow UAX #31 <https://www.unicode.org/reports/tr31/ https://www.unicode.org/reports/tr31/> for Unicode identifiers, which doesn’t allow emoji.
  - skrebbel 5y ago
    But it does allow hangul fillers, apparently.

speleding 5y ago

I think the recommendation to disallow any non-ASCII character is throwing out the baby with the bathwater. How about code that wants to display some emojis? It would be cumbersome to use hex unicode everywhere. And while localisations should typically happen in a separate language file, it's very common to want some text in code intended for a single audience. Blocking all the confusables might be tricky, and an allow list would be endless. Perhaps some magic pre-processor comment that says "allow unicode in this file".

YetAnotherNick 5y ago
Do you really have to write emoji in the code string? Similarly with international language characters. The sane thing is to use either json config files or i18n libraries.
- speleding 5y ago
  If you are writing something intended for a single audience using i18n libraries can be unnecessary overhead. And emoji can also be icons like ⌘ that can be useful to display in the UI.
josteink 5y ago
> I think the recommendation to disallow any non-ASCII character is throwing out the baby with the bathwater. Not throwing out all non-ASCII characters from code-files. Just throwing them out as being invalid identifiers in your code (think variables, function-names, etc). > How about code that wants to display some emojis? Fine. You quote that emoji in a string, and it's golden. You try to make a variable with the name of an emoji however, you code crashes. That sounds fine to me.
- speleding 5y ago
  That would close this particular attack (but not the BIDI one the article mentions). But there is probably already too much code out there with π=3.14 in it to be feasible to do this.

robertrbairdii 5y ago

There's definitely a benefit to using a linter and a tool such as prettier. Using prettier pushes the hidden character onto an additional line in the checkCommands array which makes it much easier to spot that something is wrong even if you're not using the trailingCommas setting. https://imgur.com/a/gYKylyH https://imgur.com/a/gYKylyH I think this eslint rule would also be able to defend against the initial destructuring of the query object by defining a regex that identifiers have to match which would exclude those invisible characters https://eslint.org/docs/rules/id-match https://eslint.org/docs/rules/id-match

Andys 5y ago
Indeed, this wouldn't even compile in Go.

reilly3000 5y ago

While this is an interesting hack, the larger issue in the example is allowing any query parameter to write into a subprocess. exec() immediately throws flags for me, especially when it isn’t necessary like in the case of making an http call. Even when it isn’t passing arbitrary inputs from the web to the command line, it’s susceptible to DoS that could crash the whole kernel instead of just the web server. I get that this is just a contrived example to show the risk of hidden characters, but please don’t use process.exec() unless you have no other options.

SenpaiHurricane 5y ago

TS + Intellij https://ibb.co/MfLLNQL https://ibb.co/MfLLNQL

kuon 5y ago

My editor (vim) will warn me with a loud visual red block for any non ascii char outside a string literal. But I do not think that is enough. Compiler and interpreter must be more strict.

jrochkind1 5y ago
I'm surprised VS Code doesn't at least have that option. (Or does it?)
- myfonj 5y ago
  It has `editor.renderControlCharacters` but only recently started displaying few dangerous previously invisible ones (directional overrides) natively [1], but besides that you had to use extension that adds highlights for non-ascii non-whitelisted [2] or predefined [3] characters. [1] https://github.com/microsoft/vscode/issues/116939 https://github.com/microsoft/vscode/issues/116939 [2] https://marketplace.visualstudio.com/items?itemName=nachocab.highlight-dodgy-characters https://marketplace.visualstudio.com/items?itemName=nachocab... [3] https://marketplace.visualstudio.com/items?itemName=nhoizey.gremlins https://marketplace.visualstudio.com/items?itemName=nhoizey....
tomxor 5y ago
Mind sharing the relevant config line? I thought this was default but just realised it only does the <FFFF> thing when there is no printable glyph available. Allowing printable unicode in strings seems like a nice balance if it can be done reliably.
- fatheart 5y ago
  After seeing this thread I added the following to my vimrc: highlight link NonASCII Error autocmd Syntax * :syntax match NonASCII "[^\d0-\d127]" Obviously haven't been using it long, and I'm not confident enough in my vim knowledge to vouch for its correctness, but it works in the limited amount of scenarios I tested so far.
- kuon 5y ago

cyberpsybin 5y ago

This needs to be patched. Although at this point there might be code depending on this.

ludovicianul 5y ago

On a similar note, if you want to test your REST APIs for weird characters, I built a tool for this: (https://github.com/Endava/cats#leadingcontrolcharsinfieldstrimvalidatefuzzer-trailingcontrolcharsinfieldstrimvalidatefuzzer-and-onlycontrolcharsinfieldstrimvalidatefuzzer https://github.com/Endava/cats#leadingcontrolcharsinfieldstr...)

geoduck14 5y ago

This is MOST interesting. I wonder if Git or Stack Overflow should highlight non Ascii characters to reduce malicious actors using this in code.

FrankyHollywood 5y ago

> In our experience non-ASCII characters are pretty rare in code. Many development teams chose to use English as the primary development language Is this true for the whole world, or just Europe/US?

utrack 5y ago
Yep, pretty much (Russian here)
afavour 5y ago
It is. The way it was explained to me, all the APIs you use are in English so naming variables in your local language is futile at best and would just require constant context switching. I remember many moons ago MooTools announced international API translations as an April Fools joke. It did make me wonder if there’s an interesting programming experiment to be done there… but I’m a native English speaker so I’m not best positioned to know!
- mijamo 5y ago
  Not my experience. Plenty of codebases have variables in local language in France Germany and Sweden (where I have experience). I actually have encountered a lot of problem with English codebases in those countries as they often try to translate regional concepts that are not directly translatable. This is particularly annoying when it comes to administrative stuff where one English word can refer to different local concepts (ex: geographical divisions of the territory) and translations always clumsy. I have even seen nasty bugs come from there, where a "county" had a different meaning in different places of the code as different teams had different idea of what a county was but didn't discuss it.

lovasoa 5y ago

The `cmd &&` looks fishy in their example, and would probably have been removed in a review. Instead, one could write : const { ping, curl, ㅤ } = req.query; const checkCommands = [ ping && 'ping -c 1 google.com', curl && 'curl -s http://example.com/', ㅤ ]; await Promise.all(checkCommands.map(cmd => cmd && exec(cmd, { timeout: 5_000 }))); This way the `cmd &&` is justified

testASW2 5y ago
d

Mockapapella 5y ago

If anyone is interested, I wrote an article a while back exploring which unicode characters Python allows you to set variables equal to: https://www.thelisowe.com/why-can-be-a-variable-in-python-but-not/ https://www.thelisowe.com/why-can-be-a-variable-in-python-bu... This was originally done with the goal of trying to hide/encode one program within another using non-displayable characters (such as zero width spaces), I just never got around to it. But reading this article has kind of reignited that interest for me and I think I might take another crack at that soon.

wnevets 5y ago

If you use Sublime Text the Gremlins[0] package will detect and light up these kind of characters [0] https://packagecontrol.io/packages/Gremlins https://packagecontrol.io/packages/Gremlins

Cameri 5y ago

Is there an eslint plugin to prevent invisible characters on .js/.ts files?

greyhair 5y ago

From what I have been reading lately, web3 will fix all of this. \s

egberts1 5y ago

Take two of those Hangu (“blank” and “half-blank”) characters you make two variable names… Couple it with JS obfuscator… (something, something)… Profit!

ewagsjr 5y ago

They put the answer in quotes right after they ask if you can find it. Highlight it and hit control + f.

Lorin 5y ago

I use the "nhoizey.gremlins" VSC extension which would cover these.

rurban 5y ago

A, the famous hangul filler. That's actually a Unicode bug they refuse to fix for some years now. It's still listed as identifier. I fixed that in my interpreter cperl. The next bugs are actually all JavaScript bugs, as they accept Unicode identifiers but don't check against the Unicode security guidelines, ignoring any profile. Accepting bidi, mixed scripts, unnormalized identifiers. This is very common, 99% of all interpreters and compilers don't care about Unicode security at all. They are rather proud to accept everyone, and point fingers at colleagues who only accept ASCII english. Identifiers need to be identifiable by a human. That's the whole point. And the system needs to block illegal identifiers. Similar to filesystem drivers, which consists of path names as identifiers, but the driver writers think they are beyond such human issues. For them there is only garbage in, garbage out. Their pathnames are certainly not identiable. A directory can consist of bidi names, or Russian and Greek mixed scripts who all look the same. Or just not normalized. There can be a multiple of visually duplicate names, and you never know which is which. At least with domain names they came up with a punycode solution, but this was only the tip on the iceberg. And it was a rather awkward workaround.