4 ms·

For starters, there's the Unicode Technical Report (UTR) #36 [0], "Unicode Security Considerations". It covers this particular issue in section 2.5 [1]. The g

For starters, there's the Unicode Technical Report (UTR) #36 [0], "Unicode Security Considerations". It covers this particular issue in section 2.5 [1]. The general answer is to not allow (e.g., ignore it, remove it, but do not render it) in any identifiers, filenames, domainname labels, usernames, and other security-sensitive strings, U+202E between characters of scripts that are not bi-directional and which flow in the same direction. More generally, do not allow mixing of scripts in certain contexts -- for example, do not allow mixed scripts in a single domainname label. Some of these rules need to be implemented by, e.g., DNS registries, and may need to be tailored to their specific needs (e.g., you might find that .kr has to allow mixing of ASCII letters and Hangul characters because it's common in South Korea to add "ing" to names to make brands out of them). UTR #36 doesn't go far enough, in my opinion. [0] https://unicode.org/reports/tr36/ [1] https://www.unicode.org/reports/tr36/#Bidirectional_Text_Spoofing

chrismorgan 5y ago
It should be noted that the problems of U+202E and such are not inherent to all Bidi_Control characters: Right-to-Left Marker, Left-to-Right Marker and Arabic Language Marker are all safe (and useful)—though UTR #36 suggests going far enough (e.g. “use only LTR or RTL, and if RTL, first and last characters must be strong”) that they wouldn’t be useful. It’s specific to the characters that override the default stuff, which means Embedding, Isolate and Override characters. The Trojan Source paper has a decent treatment of the area too, seen in a different application: https://trojansource.codes/trojan-source.pdf https://trojansource.codes/trojan-source.pdf.
- phkahler 5y ago
  The problem seems to be that Unicode has gone a bit too far. No need to have the ability to do this.
  - dtech 5y ago
    The need seems quite obvious when mixing LTR and RTL languages, e.g. embed an American company or English phrase in a RTL language text.
    - phkahler 5y ago
      Why does it need to overwrite or obscure existing characters?
      - _nalply 5y ago
        The Right-To-Left Override (U+202E) does not obscure nor overwrite existing characters. It is an instruction to switch the text direction to right-to-left like for Arabic and Hebrew. In the article it is shown how this can be misused to show executable files as a different file type: file[U+202e]fdp.exe shown as fileexe.pdf ~~~~~~~~^^^^^^^ | |--- read this right-to-left, i.e. exe.pdf |----------- this is the Unicode character Right-To-Left Override
        kazinator 5y ago
        It certainly obscures and overwrites. When you replace "fdp.exe" by "exe.pdf", you're obscuring and overwriting the leading "f" by "e", then the "d" by "x" and so on.
      - dtech 5y ago
        It doesn't overwrite or obscure, it switches rendering direction. E.g. \u202E\u41\u42\u43 starts outputting from the right: \u41 from the right, then 42, then 43. That's CBA, altough in normal English you'd render that as ABC.
jrochkind1 5y ago
More programming languages/platforms need to include built-in routines for various kinds of unicode sanitization.
- cryptonector 5y ago
  Yes.
  - deleted 5y ago
    [deleted]
- kazinator 5y ago
  Or maybe Unicode shouldn't include this crap in the first place? If you "sanitize" Unicode you're just removing that functionality. Remove that character, and the right to left reversal wont' be there, right? If you have to break it in the name of security and sanity, it shouldn't exist in the first place.
  - jrochkind1 5y ago
    The functions/codepoints are there for reasons. If you want to research the reasons and decide that text processing doens't need them, that's one thing. But to simply dismiss it off-hand with the argument "if you have to remove it for security it shouldn't exist in the first place" is just ridiculous. It's like saying "if you have to properly escape parameters in SQL to avoid injection attacks, SQL shouldn't exit in the first place!" I am continually baffled about why the HN comment sections are such a hotbed of anti-unicode sentiment. Perhaps most of y'all don't have to write software that deals with non-English text in any significant way. If you just didn't have buffers you wouldn't have to avoid buffer overflow attacks! Come on.
    - kazinator 5y ago
      > if you have to properly escape parameters in SQL to avoid injection attacks, SQL shouldn't exit in the first place!" Yes, this aspect of SQL is poorly conceived because the only way is to manipulate it with text processing. Parameters should be inserted into a query language using AST manipulation, like a Lisp backquote. Now within a particular program, we can wrap something like that around SQL generation, and solve the escaping problems in one place. That approach doesn't apply to character data, which has to be literal and primitive in many situations.

chrismorgan 5y ago

It should be noted that the problems of U+202E and such are not inherent to all Bidi_Control characters: Right-to-Left Marker, Left-to-Right Marker and Arabic Language Marker are all safe (and useful)—though UTR #36 suggests going far enough (e.g. “use only LTR or RTL, and if RTL, first and last characters must be strong”) that they wouldn’t be useful. It’s specific to the characters that override the default stuff, which means Embedding, Isolate and Override characters. The Trojan Source paper has a decent treatment of the area too, seen in a different application: https://trojansource.codes/trojan-source.pdf https://trojansource.codes/trojan-source.pdf.

phkahler 5y ago
The problem seems to be that Unicode has gone a bit too far. No need to have the ability to do this.
- dtech 5y ago
  The need seems quite obvious when mixing LTR and RTL languages, e.g. embed an American company or English phrase in a RTL language text.
  - phkahler 5y ago
    Why does it need to overwrite or obscure existing characters?
    - _nalply 5y ago
      The Right-To-Left Override (U+202E) does not obscure nor overwrite existing characters. It is an instruction to switch the text direction to right-to-left like for Arabic and Hebrew. In the article it is shown how this can be misused to show executable files as a different file type: file[U+202e]fdp.exe shown as fileexe.pdf ~~~~~~~~^^^^^^^ | |--- read this right-to-left, i.e. exe.pdf |----------- this is the Unicode character Right-To-Left Override
      - kazinator 5y ago
        It certainly obscures and overwrites. When you replace "fdp.exe" by "exe.pdf", you're obscuring and overwriting the leading "f" by "e", then the "d" by "x" and so on.
    - dtech 5y ago
      It doesn't overwrite or obscure, it switches rendering direction. E.g. \u202E\u41\u42\u43 starts outputting from the right: \u41 from the right, then 42, then 43. That's CBA, altough in normal English you'd render that as ABC.

jrochkind1 5y ago

More programming languages/platforms need to include built-in routines for various kinds of unicode sanitization.

cryptonector 5y ago
Yes.
- deleted 5y ago
  [deleted]
kazinator 5y ago
Or maybe Unicode shouldn't include this crap in the first place? If you "sanitize" Unicode you're just removing that functionality. Remove that character, and the right to left reversal wont' be there, right? If you have to break it in the name of security and sanity, it shouldn't exist in the first place.
- jrochkind1 5y ago
  The functions/codepoints are there for reasons. If you want to research the reasons and decide that text processing doens't need them, that's one thing. But to simply dismiss it off-hand with the argument "if you have to remove it for security it shouldn't exist in the first place" is just ridiculous. It's like saying "if you have to properly escape parameters in SQL to avoid injection attacks, SQL shouldn't exit in the first place!" I am continually baffled about why the HN comment sections are such a hotbed of anti-unicode sentiment. Perhaps most of y'all don't have to write software that deals with non-English text in any significant way. If you just didn't have buffers you wouldn't have to avoid buffer overflow attacks! Come on.
  - kazinator 5y ago
    > if you have to properly escape parameters in SQL to avoid injection attacks, SQL shouldn't exit in the first place!" Yes, this aspect of SQL is poorly conceived because the only way is to manipulate it with text processing. Parameters should be inserted into a query language using AST manipulation, like a Lisp backquote. Now within a particular program, we can wrap something like that around SQL generation, and solve the escaping problems in one place. That approach doesn't apply to character data, which has to be literal and primitive in many situations.