11 ms·

With most attack vectors, like buffer overflows, csrf, SQL injection, I (and many developers that aren't security domain experts) am acutely aware of best pract

by klabb3 5y ago

With most attack vectors, like buffer overflows, csrf, SQL injection, I (and many developers that aren't security domain experts) am acutely aware of best practices and mitigations. I know exactly where they can appear and how to prevent them. (Well not literally but it's a simple web search away). With unicode, I really don't know what to do. What strings need to be sanitized (or validated)? File names? Urls? How do I sanitize them without causing agony for most of the world? Are there other unicode attack vectors?

staticassertion 5y ago
I don't think you do anything tbh. It's not really like those other situations where there's a straightforward, static answer. The problem is honestly just that unicode is used to convey things like "this is a PDF" or "this is going to execute code" and it's attacker controlled. It's a terrible UX that's been abused for as long as paths have existed. So I guess hope that operating systems will find a better UX for "this executes" and that they'll stop "a process executed" from being a game over situation for users.
- klabb3 5y ago
  I think you're right about paths but aside from that we have website urls, usernames or similar identifiers.. People look at those and assume that they can compare them. If I see `google.com` I assume it consists of certain characters that represent a specific institution. Are we doomed to live with ascii for the forseeable future or is there some form of reduced set of unicode that we can use without these types of impersonation attacks?
  - BoorishBears 5y ago
    you shouldn't get away with an IDN Homograph attack in any modern browser They just turn your url into punycode, and more recently display that instead of the raw url: https://en.m.wikipedia.org/wiki/Punycode https://en.m.wikipedia.org/wiki/Punycode
    - account42 5y ago
      > and more recently display that instead of the raw url Technically, punicode is the raw URL and old enough browsers will display all non-ASCII domains as punycode. It was only for a short while that browsers naively decoded punicode without restrictions.
  - staticassertion 5y ago
    That's basically why Google wants to get rid of urls. They're terrible at conveying trust.
    - toomanydoubts 5y ago
      >That's basically why Google wants to get rid of urls. Wink-wink ;)
    - rightbyte 5y ago
      Google want you to search on Google to see ads and be spied upon, not use a direct url.
- cryptonector 5y ago
  There are certainly things that can be done. UTR #36 covers some of the things that can be done. Besides refusing to render U+202E when it is between characters of scripts that aren't bi-directional and flow in the same direction, the UI could display the string in ways that make it clear (e.g., different color, maybe add a warning tooltip, maybe add a dialog around an operation that could be dangerous, or maybe refuse to perform dangerous operations).
  - account42 5y ago
    Another option for filenames would be to recognize that the basename and extensions are distinct parts and the extension should always be displayed after the basename no matter what is in the basename. Of course, this adds lots of complexity so ¯\_(ツ)_/¯
    - cryptonector 5y ago
      Problem is that there's cases like foo.tar.gz. What's the extension there?
      - deleted 5y ago
        [deleted]
cryptonector 5y ago
For starters, there's the Unicode Technical Report (UTR) #36 [0], "Unicode Security Considerations". It covers this particular issue in section 2.5 [1]. The general answer is to not allow (e.g., ignore it, remove it, but do not render it) in any identifiers, filenames, domainname labels, usernames, and other security-sensitive strings, U+202E between characters of scripts that are not bi-directional and which flow in the same direction. More generally, do not allow mixing of scripts in certain contexts -- for example, do not allow mixed scripts in a single domainname label. Some of these rules need to be implemented by, e.g., DNS registries, and may need to be tailored to their specific needs (e.g., you might find that .kr has to allow mixing of ASCII letters and Hangul characters because it's common in South Korea to add "ing" to names to make brands out of them). UTR #36 doesn't go far enough, in my opinion. [0] https://unicode.org/reports/tr36/ [1] https://www.unicode.org/reports/tr36/#Bidirectional_Text_Spoofing
- chrismorgan 5y ago
  It should be noted that the problems of U+202E and such are not inherent to all Bidi_Control characters: Right-to-Left Marker, Left-to-Right Marker and Arabic Language Marker are all safe (and useful)—though UTR #36 suggests going far enough (e.g. “use only LTR or RTL, and if RTL, first and last characters must be strong”) that they wouldn’t be useful. It’s specific to the characters that override the default stuff, which means Embedding, Isolate and Override characters. The Trojan Source paper has a decent treatment of the area too, seen in a different application: https://trojansource.codes/trojan-source.pdf https://trojansource.codes/trojan-source.pdf.
  - phkahler 5y ago
    The problem seems to be that Unicode has gone a bit too far. No need to have the ability to do this.
    - dtech 5y ago
      The need seems quite obvious when mixing LTR and RTL languages, e.g. embed an American company or English phrase in a RTL language text.
rich_sasha 5y ago
It feels like the whole “everything is Unicode now” movement was misguided to me. Some things are meant to be simple, and Unicode is massively complex. True, these are rare-ish corner cases, but every programmer knows it’s only ever a matter of time until you hit any possible corner case. I’m not saying, screw people whose languages don’t fit into ASCII (I’m one of them btw), but I really don’t want to know the vagaries of Unicode representation to inspect a file name. Maybe Unicode needs a “safe mode” where only outright characters are allowed? Or maybe the unavoidable complexity is a design flaw, for some uses anyway. I feel like we’re close to someone discovering Unicode is Turing-complete...
- deleted 5y ago
  [deleted]
- dotancohen 5y ago
  > Maybe Unicode needs a “safe mode” where only outright characters are allowed? What is safe for me may not be safe for you. I need RLM characters in my terminal emulator [1], you probably don't. And who knows, maybe RLM characters could prove to be an attack vector, by e.g. displaying filenames or other text backwards. Every Unicode character and code point exists because somebody, somewhere, needs it. [1] https://bugs.kde.org/show_bug.cgi?id=450219 https://bugs.kde.org/show_bug.cgi?id=450219
  - mtizim 5y ago
    I highly doubt that last statement is true: 𓂺, ꙮ, 𒀱 .
    - dotancohen 5y ago
      Point taken! I rephrase: Every Unicode character and code point exists because somebody, somewhere, thought that somebody would find it useful.
      - thaumasiotes 5y ago
        That isn't true either; many Unicode characters exist solely because they were part of some other encoding standard that Unicode preserved in its entirety. For example, all of ASCII is preserved in that way. That's also why 囍, a character which does not have any linguistic use, is in there - it was included in an earlier Chinese encoding standard. Nobody thinks it was a good idea to separate U+2126 OHM SIGN from U+03A9 GREEK CAPITAL LETTER OMEGA either.
samwillis 5y ago
There are two broad types of attack vectors, those that target a program, and those that target a person. Most Unicode attack vectors seem to target people, they are ways to obscure from a person the true intention of what something is. As a developer it’s our job to try and mitigate this through how we display the data, warn about risks and block particular attacks. So it’s about working out how to prevent a user from making a mistake. It seems to me in this case obviously Unicode became “a thing” after the development of a file extension. Clearly if it had been the other way around the file extension would have no meaning, maybe that’s what we need to move towards, store that metadata elsewhere on the file system. Fundamentally Unicode from and “untrusted” origin shouldn’t be trusted in an executable context.
- deleted 5y ago
  [deleted]
- dgrunwald 5y ago
  We don't need to store the metadata differently. The on-disk format is still unambiguous: the part after the last dot is the extension. It's only the graphical display that is ambiguous, because the last dot is not necessarily the rightmost dot. This is easily solved with a UI change, e.g. move the file extension from the "file name" column to a separately rendered "file extension" column. One could even translate the file extension into something more user-friendly (e.g. .exe -> Application), making it a "file type" column. Which is of course what Windows Explorer has been doing for decades already. Of course, that results in everyone turning the classic file extension back on...
  - samwillis 5y ago
    I think one of the troubles here is the difference between a GUI and a command line type app. Very true on a GUI you can hide the extension and display file type in a column, which is what Windows and MacOS both increasingly do. However on the command line its still fundamentally part of the filename, to fix that is a larger job and my suggestion is that it should not be part of he filename, but metadata on the file/filesystem. The other problem with hiding the extension is that you then have "file name" collisions, two files will appear to have the same name but only differentiate by the "type". I think that's wrong.
dotancohen 5y ago
> With unicode, I really don't know what to do. What strings need to be sanitized (or validated)? Just yesterday I filed a bug on the KDE terminal emulator [1] for stripping away too many characters. I'm sure that the fine devs had no idea which characters are safe to leave in and which are not. I certainly have no idea myself. [1] https://bugs.kde.org/show_bug.cgi?id=450219 https://bugs.kde.org/show_bug.cgi?id=450219
rurban 5y ago
Apply the Unicode security guidelines for identifiers (TR39) to names, simple as that. Identifiers such as variable names, filenames, Login names, domain names, emails should be identifiable. Eg. use my libu8ident. For technical details see the C++ proposal to use secure identifiers: https://rurban.github.io/libu8ident/doc/P2528R1.html https://rurban.github.io/libu8ident/doc/P2528R1.html

by klabb3 5y ago

staticassertion 5y ago

I don't think you do anything tbh. It's not really like those other situations where there's a straightforward, static answer. The problem is honestly just that unicode is used to convey things like "this is a PDF" or "this is going to execute code" and it's attacker controlled. It's a terrible UX that's been abused for as long as paths have existed. So I guess hope that operating systems will find a better UX for "this executes" and that they'll stop "a process executed" from being a game over situation for users.

klabb3 5y ago
I think you're right about paths but aside from that we have website urls, usernames or similar identifiers.. People look at those and assume that they can compare them. If I see `google.com` I assume it consists of certain characters that represent a specific institution. Are we doomed to live with ascii for the forseeable future or is there some form of reduced set of unicode that we can use without these types of impersonation attacks?
- BoorishBears 5y ago
  you shouldn't get away with an IDN Homograph attack in any modern browser They just turn your url into punycode, and more recently display that instead of the raw url: https://en.m.wikipedia.org/wiki/Punycode https://en.m.wikipedia.org/wiki/Punycode
  - account42 5y ago
    > and more recently display that instead of the raw url Technically, punicode is the raw URL and old enough browsers will display all non-ASCII domains as punycode. It was only for a short while that browsers naively decoded punicode without restrictions.
- staticassertion 5y ago
  That's basically why Google wants to get rid of urls. They're terrible at conveying trust.
  - toomanydoubts 5y ago
    >That's basically why Google wants to get rid of urls. Wink-wink ;)
  - rightbyte 5y ago
    Google want you to search on Google to see ads and be spied upon, not use a direct url.
cryptonector 5y ago
There are certainly things that can be done. UTR #36 covers some of the things that can be done. Besides refusing to render U+202E when it is between characters of scripts that aren't bi-directional and flow in the same direction, the UI could display the string in ways that make it clear (e.g., different color, maybe add a warning tooltip, maybe add a dialog around an operation that could be dangerous, or maybe refuse to perform dangerous operations).
- account42 5y ago
  Another option for filenames would be to recognize that the basename and extensions are distinct parts and the extension should always be displayed after the basename no matter what is in the basename. Of course, this adds lots of complexity so ¯\_(ツ)_/¯
  - cryptonector 5y ago
    Problem is that there's cases like foo.tar.gz. What's the extension there?
    - deleted 5y ago
      [deleted]

cryptonector 5y ago

For starters, there's the Unicode Technical Report (UTR) #36 [0], "Unicode Security Considerations". It covers this particular issue in section 2.5 [1]. The general answer is to not allow (e.g., ignore it, remove it, but do not render it) in any identifiers, filenames, domainname labels, usernames, and other security-sensitive strings, U+202E between characters of scripts that are not bi-directional and which flow in the same direction. More generally, do not allow mixing of scripts in certain contexts -- for example, do not allow mixed scripts in a single domainname label. Some of these rules need to be implemented by, e.g., DNS registries, and may need to be tailored to their specific needs (e.g., you might find that .kr has to allow mixing of ASCII letters and Hangul characters because it's common in South Korea to add "ing" to names to make brands out of them). UTR #36 doesn't go far enough, in my opinion. [0] https://unicode.org/reports/tr36/ [1] https://www.unicode.org/reports/tr36/#Bidirectional_Text_Spoofing

chrismorgan 5y ago
It should be noted that the problems of U+202E and such are not inherent to all Bidi_Control characters: Right-to-Left Marker, Left-to-Right Marker and Arabic Language Marker are all safe (and useful)—though UTR #36 suggests going far enough (e.g. “use only LTR or RTL, and if RTL, first and last characters must be strong”) that they wouldn’t be useful. It’s specific to the characters that override the default stuff, which means Embedding, Isolate and Override characters. The Trojan Source paper has a decent treatment of the area too, seen in a different application: https://trojansource.codes/trojan-source.pdf https://trojansource.codes/trojan-source.pdf.
- phkahler 5y ago
  The problem seems to be that Unicode has gone a bit too far. No need to have the ability to do this.
  - dtech 5y ago
    The need seems quite obvious when mixing LTR and RTL languages, e.g. embed an American company or English phrase in a RTL language text.

rich_sasha 5y ago

It feels like the whole “everything is Unicode now” movement was misguided to me. Some things are meant to be simple, and Unicode is massively complex. True, these are rare-ish corner cases, but every programmer knows it’s only ever a matter of time until you hit any possible corner case. I’m not saying, screw people whose languages don’t fit into ASCII (I’m one of them btw), but I really don’t want to know the vagaries of Unicode representation to inspect a file name. Maybe Unicode needs a “safe mode” where only outright characters are allowed? Or maybe the unavoidable complexity is a design flaw, for some uses anyway. I feel like we’re close to someone discovering Unicode is Turing-complete...

deleted 5y ago
[deleted]
dotancohen 5y ago
> Maybe Unicode needs a “safe mode” where only outright characters are allowed? What is safe for me may not be safe for you. I need RLM characters in my terminal emulator [1], you probably don't. And who knows, maybe RLM characters could prove to be an attack vector, by e.g. displaying filenames or other text backwards. Every Unicode character and code point exists because somebody, somewhere, needs it. [1] https://bugs.kde.org/show_bug.cgi?id=450219 https://bugs.kde.org/show_bug.cgi?id=450219
- mtizim 5y ago
  I highly doubt that last statement is true: 𓂺, ꙮ, 𒀱 .
  - dotancohen 5y ago
    Point taken! I rephrase: Every Unicode character and code point exists because somebody, somewhere, thought that somebody would find it useful.
    - thaumasiotes 5y ago
      That isn't true either; many Unicode characters exist solely because they were part of some other encoding standard that Unicode preserved in its entirety. For example, all of ASCII is preserved in that way. That's also why 囍, a character which does not have any linguistic use, is in there - it was included in an earlier Chinese encoding standard. Nobody thinks it was a good idea to separate U+2126 OHM SIGN from U+03A9 GREEK CAPITAL LETTER OMEGA either.

samwillis 5y ago

There are two broad types of attack vectors, those that target a program, and those that target a person. Most Unicode attack vectors seem to target people, they are ways to obscure from a person the true intention of what something is. As a developer it’s our job to try and mitigate this through how we display the data, warn about risks and block particular attacks. So it’s about working out how to prevent a user from making a mistake. It seems to me in this case obviously Unicode became “a thing” after the development of a file extension. Clearly if it had been the other way around the file extension would have no meaning, maybe that’s what we need to move towards, store that metadata elsewhere on the file system. Fundamentally Unicode from and “untrusted” origin shouldn’t be trusted in an executable context.

deleted 5y ago
[deleted]
dgrunwald 5y ago
We don't need to store the metadata differently. The on-disk format is still unambiguous: the part after the last dot is the extension. It's only the graphical display that is ambiguous, because the last dot is not necessarily the rightmost dot. This is easily solved with a UI change, e.g. move the file extension from the "file name" column to a separately rendered "file extension" column. One could even translate the file extension into something more user-friendly (e.g. .exe -> Application), making it a "file type" column. Which is of course what Windows Explorer has been doing for decades already. Of course, that results in everyone turning the classic file extension back on...
- samwillis 5y ago
  I think one of the troubles here is the difference between a GUI and a command line type app. Very true on a GUI you can hide the extension and display file type in a column, which is what Windows and MacOS both increasingly do. However on the command line its still fundamentally part of the filename, to fix that is a larger job and my suggestion is that it should not be part of he filename, but metadata on the file/filesystem. The other problem with hiding the extension is that you then have "file name" collisions, two files will appear to have the same name but only differentiate by the "type". I think that's wrong.

dotancohen 5y ago

> With unicode, I really don't know what to do. What strings need to be sanitized (or validated)? Just yesterday I filed a bug on the KDE terminal emulator [1] for stripping away too many characters. I'm sure that the fine devs had no idea which characters are safe to leave in and which are not. I certainly have no idea myself. [1] https://bugs.kde.org/show_bug.cgi?id=450219 https://bugs.kde.org/show_bug.cgi?id=450219

rurban 5y ago

Apply the Unicode security guidelines for identifiers (TR39) to names, simple as that. Identifiers such as variable names, filenames, Login names, domain names, emails should be identifiable. Eg. use my libu8ident. For technical details see the C++ proposal to use secure identifiers: https://rurban.github.io/libu8ident/doc/P2528R1.html https://rurban.github.io/libu8ident/doc/P2528R1.html