6 ms·

Honest question: would it be that bad to mandate and enforce 100% ASCII source files? Arguably every and any Unicode character and, well, arguably even any str

Honest question: would it be that bad to mandate and enforce 100% ASCII source files? Arguably every and any Unicode character and, well, arguably even any string of characters can (should?) go to a properties/resources file (properties/resources files which, btw, also greatly simplifies i18n/l10n). Then build/commit/test hooks could be used to enforce that source code files are indeed 100% ASCII. I know, I know... Some are going to lament they don't have their shiny Unicode symbols right in their source file. But... It looks like you get what you pay for. Bruce Schneier wrote it when Unicode came out btw: "Unicode is too complex to ever be secure".

dotancohen 5y ago
Though I comment source in English, lots of people that I work with comment in other languages. כולל עברית, מימין לשמאל.‏
EamonnMR 5y ago
Having readable unicode in string literals is nice.
Jeff_Brown 5y ago
> Bruce Schneier wrote it when Unicode came out btw: "Unicode is too complex to ever be secure". It's astounding to me that there's room for such complexity in it. I thought it was just a lot of symbols. What other rules does Unicode have besides changing the order sometimes?
- ncc-erik 5y ago
  The one a lot of folks know about was the soft hyphen (U+00AD) to bypass swear filters. I was able to use normalization to create XSS attacks.
supperburg 5y ago
How dare you suggest something sensible. The mob will soon be knocking at your door.
sfgweilr4f 5y ago
Give it a few years and unicode will probably be turing-complete. For reasons... likely not good ones though.
- wizzwizz4 5y ago
  Unicode rendering already requires multiple finite state machines.
  - sfgweilr4f 5y ago
    Alternate cause of SkyNet: a distributed horde of unicode renderers become self-aware. Emojis become command and control codes.
btbuildem 5y ago
That was my first thought -- run all your source through an ASCII-only filter, the problem goes away.
- iforgotpassword 5y ago
  For projects like the Linux kernel this should be absolutely feasible. A few names in headers get mangled and lose their accents but that should be acceptable. Other projects... Well there's already a couple examples in this comment section why it won't be that easy.
thereddaikon 5y ago
Seems the bigger complaint isn't lack of fancy unicode in comments its non-english speakers with non latin alphabets wont be able to comment in their native language. I'll leave it up to others to discuss how important this is or isn't.
InfiniteRand 5y ago
Might be nice to have an easy tool to scan files and whitelist characters from specific alphabets, because in most international teams I think you'll have a common language for comments, and so I think it's unlikely that you'll need say European and Indic and Chinese characters in one code base. Except the one pain point I can see - @author annotations in the source code, if you have an international team you might end up with a variety of scripts in that field, in my mind that's something that can be lived without, but I can imagine some people being sensitive about that.
SavantIdiot 5y ago
Wouldn't simply stripping comments before doing any other processing solve the problem? I know there are plenty of programs that sprinkle code into comments, from Emacs to linters. Or is this obviously naive? Seems to me that if you need to put code in the comments, you've got a bigger problem. I know people like tab hints and lint overrides, but maybe it is time to focus on separation of concerns at a higher level?
- josephcsible 5y ago
  > Wouldn't simply stripping comments before doing any other processing solve the problem? No, because the exploit line doesn't contain a comment. It just looks like it does.
aidenn0 5y ago
Even ignoring the fact that non-english speakers will want to (and do; I worked on a compiler with SJIS support 20 years ago) write comments in their native languages, it's unavoidable to have non-ascii characters in your string literals unless you want to regress user-interfaces to 1980.
- account42 5y ago
  > it's unavoidable to have non-ascii characters in your string literal It might not be ideal for you, but you can always use escape codes in your string literals so it is at least possible to avoid all non-ascii characters in your code.

dotancohen 5y ago

Though I comment source in English, lots of people that I work with comment in other languages. כולל עברית, מימין לשמאל.‏

EamonnMR 5y ago

Having readable unicode in string literals is nice.

Jeff_Brown 5y ago

> Bruce Schneier wrote it when Unicode came out btw: "Unicode is too complex to ever be secure". It's astounding to me that there's room for such complexity in it. I thought it was just a lot of symbols. What other rules does Unicode have besides changing the order sometimes?

ncc-erik 5y ago
The one a lot of folks know about was the soft hyphen (U+00AD) to bypass swear filters. I was able to use normalization to create XSS attacks.

supperburg 5y ago

How dare you suggest something sensible. The mob will soon be knocking at your door.

sfgweilr4f 5y ago

Give it a few years and unicode will probably be turing-complete. For reasons... likely not good ones though.

wizzwizz4 5y ago
Unicode rendering already requires multiple finite state machines.
- sfgweilr4f 5y ago
  Alternate cause of SkyNet: a distributed horde of unicode renderers become self-aware. Emojis become command and control codes.

btbuildem 5y ago

That was my first thought -- run all your source through an ASCII-only filter, the problem goes away.

iforgotpassword 5y ago
For projects like the Linux kernel this should be absolutely feasible. A few names in headers get mangled and lose their accents but that should be acceptable. Other projects... Well there's already a couple examples in this comment section why it won't be that easy.

thereddaikon 5y ago

Seems the bigger complaint isn't lack of fancy unicode in comments its non-english speakers with non latin alphabets wont be able to comment in their native language. I'll leave it up to others to discuss how important this is or isn't.

InfiniteRand 5y ago

Might be nice to have an easy tool to scan files and whitelist characters from specific alphabets, because in most international teams I think you'll have a common language for comments, and so I think it's unlikely that you'll need say European and Indic and Chinese characters in one code base. Except the one pain point I can see - @author annotations in the source code, if you have an international team you might end up with a variety of scripts in that field, in my mind that's something that can be lived without, but I can imagine some people being sensitive about that.

SavantIdiot 5y ago

Wouldn't simply stripping comments before doing any other processing solve the problem? I know there are plenty of programs that sprinkle code into comments, from Emacs to linters. Or is this obviously naive? Seems to me that if you need to put code in the comments, you've got a bigger problem. I know people like tab hints and lint overrides, but maybe it is time to focus on separation of concerns at a higher level?

josephcsible 5y ago
> Wouldn't simply stripping comments before doing any other processing solve the problem? No, because the exploit line doesn't contain a comment. It just looks like it does.

aidenn0 5y ago

Even ignoring the fact that non-english speakers will want to (and do; I worked on a compiler with SJIS support 20 years ago) write comments in their native languages, it's unavoidable to have non-ascii characters in your string literals unless you want to regress user-interfaces to 1980.

account42 5y ago
> it's unavoidable to have non-ascii characters in your string literal It might not be ideal for you, but you can always use escape codes in your string literals so it is at least possible to avoid all non-ascii characters in your code.