5 ms·
> Cambridge research clearly shows that most compilers can be tricked with Unicode into processing code in a different way than a reader would expect it to be p
by smsm42 5y ago
> Cambridge research clearly shows that most compilers can be tricked with Unicode into processing code in a different way than a reader would expect it to be processed.
Unless I misunderstand the premise, this in not right. The compiler is not "tricked" into doing anything different - it interprets the code the same way as it always did. It's like saying "rm" command "can be tricked into" deleting important files. The rm tool doesn't know which files are important to you, and the compiler doesn't - and shouldn't - know what you consider to be "correct" code. It would correctly compile any code that is syntactically correct - if there are strings inside that look weird to you, it doesn't matter to the compiler.
The entity that can be "tricked" here is the reviewer of the code - who, indeed, might probably be tricked into accepting code that does something different than they'd think it does (though it'd require a very clever attacker to for the code to both do something nefarious with Unicode and still look innocent and not weird to the reviewer). Fortunately, this is quite easy to fix - just don't accept any patches with source code that have any non-ASCII outside small set of localization resources (proper code would have localizable resources outside the code anyway, tbh) and no Unicode would ever trick you.
- __alexs 5y ago> Fortunately, this is quite easy to fix - just don't accept any patches with source code that have any non-ASCII outside small set of localization resources There are plenty of projects out there written by people who aren't English speakers who depend on the Unicode capabilities of languages to write code that is actually readable to them. Turning that off is far from a solution.
- ivanhoe 5y agoDoes anyone actually do that in a production code? I myself am not native English speaker and use unicode when writing in my mother tongue, but in 20+ years of programming I've never seen anyone using non-ascii chars in their professionally written code? Of course, you use the language in localization files, and perhaps in comments occasionally - especially in TODO stuff that's not meant to be permanent - but not in the actual code, like e.g. for a variable or function names. I'd actually consider it a bad idea, as it limits significantly who can manage that code in the future.
- Cthulhu_ 5y agoIt's a very western / Anglosphere attitude, and I think you underestimate how much code is produced in e.g. China and Japan nowadays, with comments in their native language. How would you name a FooBarWicket if you don't speak a word of English? I mean don't get me wrong, ideally everybody writes code in perfect English and sticks to a set of ~50 ascii characters, but it's not an ideal world and you have to keep other languages and cultures in mind.
- amenod 5y agoI would argue that even if you decide that you are using some other language and not English, there is only a well-defined subset of Unicode characters that should ever be allowed in the codebase. Bidi override control characters are clearly not among them, whichever language you choose.
- rbanffy 5y ago> Bidi override control characters are clearly not among them, whichever language you choose. Not sure how would you write a comment in an RTL human language in the middle of LTR code without it. Lots of people write learn RTL languages well before writing any code. What compilers can do is to process those characters and assign them semantic value that makes the code equivalent to what is expected to be rendered. Now, bidi overrides in identifier names is a nightmare I’d prefer to avoid.
- amenod 5y agoThe same way as you write a comment in a LTR human language in the middle of RTL code - you don't. You stick to either LTR or RTL. This is code, not prose.
- rbanffy 5y agoCode is meant to be read and, occasionally, executed. Comments are usually ignored by compilers and are targeted towards humans.
- smsm42 5y agoCan you give an example? I've never seen a project (outside domains on APL, etc.) that seriously relied on any Unicode capabilities in the code itself (again, I am not talking about localized strings). My native language is not English, I've worked with people all over Europe, China, India, Japan, Israel, etc. - there are a lot of exciting i18n/l10n problems but I have never seen much of what a compiler would need to be concerned with.
- klohto 5y agoYou argument away your own fix. Proposed fix is like if rm was limited to files outside of /sys, plenty of projects depend on the standardized behavior.
- robin_reala 5y agoAPL developers would disagree.
- smsm42 5y agoFor people that spend their days reviewing APL code, the concerns of mere mortals are not important.
- deleted 5y ago[deleted]
- Sebb767 5y ago> The rm tool doesn't know which files are important to you, and the compiler doesn't - and shouldn't - know what you consider to be "correct" code. This is actually no longer true. Many rm implementations today prevent you from deleting a path including the root directory, unless you explicitly specify `--no-preserve-root`. Similarly, a lot of compilers tend to warn you or outright stop if they detect code that is very likely to be buggy - the rust compiler warning about these control characters is just the latest example. Of course, in theory, each tool should do its job and the user should be the boundary to know whats right. In practice, though, these heuristics tend to catch bugs-to-be 95% of the time (at least in my experience) and are easily disabled otherwise, so they are good to have.
- wizzwizz4 5y agoI couldn't care less about my root directory. The only things I care about are the motherboard firmware and the /home directory, and nothing prevents `rm` from deleting those. The `--one-file-system` or `--preserve-root=all` flags are more useful than `--preserve-root`, but they're not defaults. (For a good reason: compatibility.)