12 ms·
How can you be fooled by the U+202E trick? (2013)
- legalcorrection 5y agoSeems like something Microsoft should patch.
- Waterluvian 5y agoPlease correct me if I’m overreacting but I feel like Unicode in most cases is just a giant security flaw and the only reason we tolerate it is because it does tremendous good making things more equitable for other languages and cultures. Unicode in URLs and usernames and filenames is just so easy to trick people with. Will there be Unicode email addresses?
- WalterGR 5y agothe only reason we tolerate it is because it does tremendous good making things more equitable for other languages and cultures. Let me rewrite that from the opposite perspective. The reason I use it is so that I can write in my language.
- dredmorbius 5y agoWriting in your language is fine. Writing in a language the reader doesn't understand is ... not so fine. Writing in a way to give the appearance of one message but the machine-recognised existence of another is ... wrong, malicious, and harmful. At root the issue is that encodings and graphical presentations aren't the same thing. 7-bit ASCII is limited and constrained, but as a universally understood encoding those specific characteristics are useful benefits. Yes, it means that representations are limited. But that's the essential trade-off for a lack of ambiguity. And even within ASCII, there are homoglyphs or near-homoglyphs: {0O,1lI, 5S} being the most frequently encountered. Kerning and ligatures may present others, as with {m, rn}. In historical documents, distinguishing {ſ, f}.
- klabb3 5y agoOther languages? Other than what?
- bryanrasmussen 5y agoGiven the ways that English works, one of which is the removal of all sorts of deliberate and tedious indications as to what one is talking about usually found in the Germanic languages, when one writes in English and uses the phrase 'other languages' but there is no specific language under discussion it should be understood that other languages refers to languages other than English.
- happytoexplain 5y agoThe parent is making the point that the language the grandparent is referring to is irrelevant to the language of an arbitrary individual included in the word "we".
- klabb3 5y agoIt was just rhetorical. As the sibling comment alluded to, it can come off as quite anglocentric. There is some historical relevance to this point - much of the technology we use today was created with such biases baked in. I apologize if this came off as snarky.
- hashimotonomora 5y agoIf I’m writing industrial automation, server-side applications, video rendering codecs, why do I need to make it much more complex with Unicode? I just need and want pure 1 byte ASCII.
- happytoexplain 5y agoThe grandparent's assertion was in the general case, not specific cases. However, even for your specific cases, it is possible - even likely - to at some point handle user-facing text (i.e. text that is reasonable to localize), at the very least in the form of error messages. Also, the complexity of Unicode is typically already well implemented by default in most environments, and the real issue only comes at the intersection of user-facing text and functional strings (e.g. URLs and filenames), which is a problem that Unicode is another tool to exploit, but not a problem that Unicode invented. We have been trying to detangle the dual nature of URLs/filenames for a long time. However, tangential to the grandparent's topic, I do imagine there are very specialized cases where Unicode is pretty objectively unnecessary.
- bryanrasmussen 5y ago>Please correct me if I’m overreacting but I feel like Unicode in most cases is just a giant security flaw well ok, it seems to me that it is more likely that in most cases Unicode probably has no effect on security one way or another, but there are a few edge cases where it does. Here is one case where using a particular character in a filename led to problems 8 years ago in an operating system historically known for not having the greatest security. We can see here also a clear example of the edge case theory, almost every character in every language supported by Windows would not have caused any sort of problem when used in a filename. But there are a few characters where they would, this is probably an example of things programmers think they know about unicode or language or whatever, because we go around thinking that each unicode code point just represents a character in a language but some encodings have ways to represent little weird behaviors of particular languages, for example interlinear annotation characters https://www.unicode.org/charts/nameslist/n_FFF0.html https://www.unicode.org/charts/nameslist/n_FFF0.html and it is generally weird behaviors that end up being security hazards (not sure if there is any security hazard in interlinear annotation characters but wouldn't be surprised)
- mkotowski 5y ago> Will there be Unicode email addresses? Apparently, yes. [0] > making things more equitable for other languages and cultures Well, yes. People often hate how compilcated Unicode is, but they also tend to forget that even ASCII was not sufficient for writing English in a proper way. I am not by any means old, but even I still remember the period on the web when non-Unicode encodings were relatively common and how problematic it was. And, with assumption that by “other cultures and languages” you mean non-English speaking regions: Tolerate is a wrong word. Most people are not English native speakers. If we would go the route of deciding who is tolerating who, nations using Chinese Hanzi and its derivatives would be probably the first to claim they tolerate all the others by the speakers’ headcount alone. Moreover, deciding what is/can be executed is not so obvious in my opinion. PDFs can have JavaScript embedded, SVG too. Flash games? Python scripts on computers with python installed? It is not just the names, but the very knowledge of what can be considered “executable.” [0]: https://en.wikipedia.org/wiki/Unicode_and_email https://en.wikipedia.org/wiki/Unicode_and_email
- cryptonector 5y agoASCII is enough to write English, or, at least, it used to be back when ASCII was a multi-byte encoding for Latin! I know, it sounds crazy, but it used to be a thing to use over-strike using back-space. E.g., ë was e BS ", ø was o BS /, etc. It was a thing back in the days of typewriters. I believe this is the reason that upper-case letters in Spanish are allowed to be written without accents even when they are needed otherwise, that accented upper-case letters were hard to produce with typewriters and ASCII fonts.
- quietbritishjim 5y ago> ASCII was a multi-byte encoding for Latin The comment you're replying to just said ASCII without qualification. By definition this is not multibyte, or even whole byte (assuming by byte you mean octet). ASCII is a 7 bit standard, anything else is some other encoding.
- cryptonector 5y ago
- cryptonector 5y agoUnicode certainly has been a source of security vulnerabilities. Are you overreacting? Unless you were proposing we rip it out, no, you're not. Unicode isn't going anywhere. That's because the scripts that human languages use aren't going anywhere. And people do mix scripts, too. There are answers though. First, Unicode has been a learning experience, and we're still all learning. Second, one of the outputs of all that learning that has happened so far is UTR #36 (https://www.unicode.org/reports/tr36/ https://www.unicode.org/reports/tr36/), which does cover a lot of these things.
- drran 5y agoASCII certainly has been a source of security vulnerabilities.
- dredmorbius 5y agoExamples? I mean, obXkcd: https://xkcd.com/327/ https://xkcd.com/327/ But spelling out what you had in mind would be helpful here.
- drran 5y ago\n, \r, \v \0, ", ', </script>, &, %, 1 vs I vs l, O vs 0, etc.
- dredmorbius 5y agoThanks. Still, a limited set. Yes, errors are made and occur. They're reasonably easy to code defensively against. Unicode ... vastly expands the attack interface.
- drran 5y agoASCII is part of Unicode, so for Unicode it's also «a limited set. Yes, errors are made and occur. They're reasonably easy to code defensively against». > Unicode ... vastly expands the attack interface. Limit yourself to 0..9 to be safe.
- rhacker 5y agos/Unicode/Computers/ Same thing really.
- simion314 5y agoYou can have security issues with non escaped ASCII too, so you probably want to fix the tools and educate people about other languages and cultures (or even if you just educate them to be capitalists they would understand that selling their iDevice or CoolApp to make cash would require it to work with non english , so unicode means more money not we tolerate your language because we are too nice)
- zeepzeep 5y ago> Will there be Unicode email addresses? So you don't know about mailoji.com? It was on HN a couple months ago I believe.
- izacus 5y ago> Please correct me if I’m overreacting but I feel like Unicode in most cases is just a giant security flaw and the only reason we tolerate it is because it does tremendous good making things more equitable for other languages and cultures. Well, that's a lot like saying "software, in general, is just a giant security flaw we tolerate because it does useful things". It's kinda the point aint it? In 21st century, even Americans aren't able to write their own names and place names with pure ASCII anymore. So Unicode is there from necessity, not as an addon.
- dredmorbius 5y agoAll technology is a means to an end or goal. All technologies have intended / desireable, and unintended / undesireable effects. The more powerful and flexible a technology, the more likely there are unintended / undesireable effects. Many of those unintended / undesireable effects are themselves not obviously apparent, not immediately manifest, or both. All of which makes risk assessment all the more difficult. Software, in that sense, is inherently a security flaw, as it virtually always manifests unintended and/or undesireable effects. This isn't necessarily an argument to ban all software, though the Butlerian Jihad are taking notes. It is an argument, however, for acknowledging risks, raising awareness of them, and taking reasonable steps to guard against them.
- jrochkind1 5y agoThe only reason we "tolerate" unicode in, say, usernames, is because it does things like let people with non-ascii names have usernames with their names, right... but this is kind of a big reason!
- beeforpork 5y agoOK, I correct you: you overeact. And, yes, of course, email addresses in local languages, why the heck not? People in the world want to use their language! And yeah, even Hebrew, Tibetan, or Mongolian (in vertical script). That comment of yours reads to me like an lazy post of a unilingual or uniscriptal cultural imperialist who really does not care about other languages. Your attitude makes me rather angry, because Unicode is such an amazing achievement for the world, and your comment just comes off totally ignorant of that. It should be clear that combining the world's languages into a common standard is really, really difficult and will inevitably create something that is much more complex than your beloved ASCII. And it should also be clear that in such a complex standard, you cannot (a) solve all problems the first time you try, (b) you cannot try a second time, because no-one will adopt yet another such standard. So just read those Unicode documents in order to understand. Those people really try and there are security consideration documents, and they are extended all the time. And take those new security warnings for what they are: they are problems with a complex system. It's expected. So when they get known, react calmly and figure out whether you need to fix anything.
- MichaelZuo 5y agoThere doesn’t seem to be any way to fix this issue though? Unless you mean breaking the standard such as forcing special behaviour for this particular character such that it simply cannot be used in a filename. But that would defeat the purpose of Unicode, it would turn, at the very least, into Duocode.
- marcosdumay 5y agoHum... Most standards allow that kind of special cases. For filenames, POSIX doesn't, but your viewer can render it on any way (and it already renders many dangerous characters - ASCII had them too - in non-obvious ways).
- MichaelZuo 5y agoThis wouldn’t be a standard allowing a special case, it would be forcing all current and future machines to implement a special case. What if one vendor decides not to care? Then there would be a de facto split, it’s likely why the unicode committee doesn’t directly fix the issue.
- brandmeyer 5y agoI agree with you. TR36 is outright terrifying. I might feel more comfortable if the recommendations were already baked into foundational libraries like libicu, but they aren't. Its as if I'm wading into a world with as many footguns and silent killers as C's undefined behavior rules. In C we've spent decades developing tools to help detect and mitigate the problems and they are still regularly exploited. Unicode only has widely-available mitigation tools for a handful of the easiest problems to solve. C's flaws lead to the development of several different competing programming languages. Various flaws in cryptography lead to the development of algorithms that achieve the same result, but have far fewer implementation gotchas. IMO, the time has come for the development of a truly strict variant of Unicode that still supports the primary objective, but learns from Unicode's mistakes.
- dredmorbius 5y agoI think there's a fair case for that argument. There are certainly contexts in which Unicode unambiguously and demonstrably leads to security weaknesses and issues. See generally homoglyph attacks. At the heart of the lie and damage is the existence of a message which appears to say one thing but in fact says something different. It's the very limited nature of 7-bit ASCII, 128 characters in total, which provide its utility here. Yes, this means that texts in other languages must be represented by transliterations and approximations. That's ... simply a necessary trade-off. We see this in other domains, in which for the purposes of reducing ambiguity and emphasizing clarity standardisation is adopted. Internationally, air traffic control communications occur in English, and aircraft navigation uses feet (altitude) and nautical miles (dstance) units. Through the early 20th century, the language of diplomacy was French. The language of much scientific discourse, particularly in physics, was German. And for the Catholic Church, Latin was abandoned for mass only in the 1960s. Trading and maritime cultures tend to creat pidgin languages --- common amongst participants, but foreign to all, as distinguished from a creole, an amalgam language with native speakers. A key problem with computers is that the encodings used to create visual glyphs and the glyphs themselves are two distinct entities, and there can be a tremendous amount of ambiguity and confusion over similarly-appearing characters. Or, in many cases, glyphs cannot be represented at all. Where the full expressive value of language is required --- within texts, in descriptive fields, and in local or native contexts, I'm ... mostly ... open to Unicode (though it can still present problems). Where what is foremost in functionality is broad and universal understanding, selectinga small standardised and widely-recognised characterset has tremendous value, and no amount of emotive shaming changes that fact. As an example, OpenStreetMap generally represents local place names in local language and charactersets. This may preserve respect or integrity to the local culture. As a user of the map, however, not knowing that language or charcterset, it is utterly useless to me. Or, quite frankly, anyone not specifically literate in that language and writing system. It's worth considering that the characterset and language in question are themselves, adoptions and impositions: English was brought into Britain by invaders, the alphabet used itself is Roman, based on Greek and originally Phoenecian glyphs. English has adopted or incorporated terms from a huge set of other languages (rendering its own internal consistency ... low ... and making it confusing to learn). International communications and signage, at airports, on roadways, in public buildings, on electronic devices, aims at small message sets and consistent, widely-recognised symbols, shapes, fonts, and colours. That is a context in which the freedoms of unfettered Unicode adoption are in fact hazardous. (Yes, many of those symbols now have Unicode code points. It is the symbol set and glyph set which is constrained in public usage.) And the simple fact is that a widely recognised encoding system will most often reflect on some power structure or hierarchy, as that's how these encodings become known --- English, Roman Alphabet, French, German, Latin, etc. Small minor powers tend not to find their writing systems widely adopted (yes, there are exceptions: use of Greek within the Roman empire, Hindu numbering systems). Again, exceptions.
- rurban 5y agoThat's why unicode published the security guidelines and mechanisms to avoid such attacks. In 2004 already. The problem is that nobody cared. Browsers invented punycode instead of following tr39, email ditto. But ok, at least something. Java did it, cperl did, rust did it. Everybody else is vulnerable. Esp. most other programming languages, filesystems and login systems. https://github.com/rurban/libu8ident/blob/master/doc/c11.md https://github.com/rurban/libu8ident/blob/master/doc/c11.md
- klabb3 5y agoWith most attack vectors, like buffer overflows, csrf, SQL injection, I (and many developers that aren't security domain experts) am acutely aware of best practices and mitigations. I know exactly where they can appear and how to prevent them. (Well not literally but it's a simple web search away). With unicode, I really don't know what to do. What strings need to be sanitized (or validated)? File names? Urls? How do I sanitize them without causing agony for most of the world? Are there other unicode attack vectors?
- staticassertion 5y agoI don't think you do anything tbh. It's not really like those other situations where there's a straightforward, static answer. The problem is honestly just that unicode is used to convey things like "this is a PDF" or "this is going to execute code" and it's attacker controlled. It's a terrible UX that's been abused for as long as paths have existed. So I guess hope that operating systems will find a better UX for "this executes" and that they'll stop "a process executed" from being a game over situation for users.
- klabb3 5y agoI think you're right about paths but aside from that we have website urls, usernames or similar identifiers.. People look at those and assume that they can compare them. If I see `google.com` I assume it consists of certain characters that represent a specific institution. Are we doomed to live with ascii for the forseeable future or is there some form of reduced set of unicode that we can use without these types of impersonation attacks?
- BoorishBears 5y agoyou shouldn't get away with an IDN Homograph attack in any modern browser They just turn your url into punycode, and more recently display that instead of the raw url: https://en.m.wikipedia.org/wiki/Punycode https://en.m.wikipedia.org/wiki/Punycode
- 5y ago
- svnpenn 5y agoWarning, page vseems to hijack the back button
- ludovicianul 5y agoI've written an API Fuzzer that (among other things) tests specifically for things like this: https://github.com/Endava/cats https://github.com/Endava/cats (with a lot more other unicode special chars). People many times forget how big the Unicode standard is and that a lot of Control Characters are used by different processors for specific logic not-very-obvious logic.
- jeisc 5y agoWho was assigned to think about security when creating Unicode?
- teddyh 5y agoMark Davis <markdavis@google.com> and Michel Suignard <michel@suignard.com>, according to <https://www.unicode.org/reports/tr36/ https://www.unicode.org/reports/tr36/>.
- alkonaut 5y agoThis is why file names in applications shouldn't be considered "just a sequence of bytes forming any name". It should be a tiny subset of valid strings for some definition of "valid". Even if file systems or shells allow anything in a file name, you don't need applications to allow everything. A system SaveAs dialog, or a graphical file explorer should just refuse to open or rename a file to contain U+202E or even something remotely nonobvious. Power users always have a backdoor (open a shell and rename the file) while "most users" should be presented with simple/sane/safe behavior.
- Ekaros 5y agoRealistically you don't even want to allow good chunk of ASCII in filenames. Actually the subset that you would like to allow is small, namely lower and uppercase, numbers, ., - and _. That is what 65 out of 127... Maybe a few more at stretch, but even some of those are questionable with some history like ~.
- blendergeek 5y agoAs far as uppercase goes, do we really need a difference between the 'Documents' folder and the 'documents' folder? What is the use case for this?
- Ekaros 5y agoThat is also an open question. The whole case-sensitivity for filenames. I don't really think it is needed.
- Diti 5y agoHow would you implement caching, or looking up for the decryption key for some directory in a lookup table, if `Documents` and `documents` must both resolve to the same entry? Some kind of normalization would be needed, right? Then you need to introduce encoding, and [Unicode?] normalization. Shivers
- blendergeek 5y ago
- farzher 5y agoif you do this to an exe on windows it's automatically detected as a virus and deleted. i just got tricked into deleting my exe file :(
- skim_milk 5y agoI remember back in the day before websites started focusing on input sanitization you could get away with really messing up website designs with that Unicode character. Long time ago I made a group on Steam with just the Unicode RTL character, years before they started banning the character. Then a decade later, some Russians found it and started using the group as their clan tag to make their character info appear backwards in Counter Strike GO - https://www.youtube.com/watch?v=TZCw_cYIY_g https://www.youtube.com/watch?v=TZCw_cYIY_g
- deleted 5y ago[deleted]
- atum47 5y agoBack in the web chat days, this is how we used to clone other people's nickname, haha. If I'm not mistaken you'd press alt + 0160 on the numpad to insert a invisible char.
- sysadmin21 5y agoI'm curious the best way to protect end users or yourself from this, if it still exists. I wonder how trivial it would be for either antivirus software or a user written program to check the file for this?
- deleted 5y ago[deleted]
- murkle 5y agodocument\u202Efdp.exe -> (looks like) documentexe.pdf With the \u202e character in, what does HN do with it? documentfdp.exe
- Mr_Modulo 5y agoThis is almost moot because Windows hides file extensions by default and has done so since like Windows 95. So the user has to enable showing file extensions for known file types. You might as well name the file document.doc.exe
- spixy 5y agoIf you are pro user, you will notice the extra extension that shouldnt be there, if you are basic user, you dont care about extensions and try to run everything anyway.