5 ms·
Please correct me if I’m overreacting but I feel like Unicode in most cases is just a giant security flaw and the only reason we tolerate it is because it does
by Waterluvian 5y ago
Please correct me if I’m overreacting but I feel like Unicode in most cases is just a giant security flaw and the only reason we tolerate it is because it does tremendous good making things more equitable for other languages and cultures.
Unicode in URLs and usernames and filenames is just so easy to trick people with.
Will there be Unicode email addresses?
- WalterGR 5y agothe only reason we tolerate it is because it does tremendous good making things more equitable for other languages and cultures. Let me rewrite that from the opposite perspective. The reason I use it is so that I can write in my language.
- dredmorbius 5y agoWriting in your language is fine. Writing in a language the reader doesn't understand is ... not so fine. Writing in a way to give the appearance of one message but the machine-recognised existence of another is ... wrong, malicious, and harmful. At root the issue is that encodings and graphical presentations aren't the same thing. 7-bit ASCII is limited and constrained, but as a universally understood encoding those specific characteristics are useful benefits. Yes, it means that representations are limited. But that's the essential trade-off for a lack of ambiguity. And even within ASCII, there are homoglyphs or near-homoglyphs: {0O,1lI, 5S} being the most frequently encountered. Kerning and ligatures may present others, as with {m, rn}. In historical documents, distinguishing {ſ, f}.
- klabb3 5y agoOther languages? Other than what?
- bryanrasmussen 5y agoGiven the ways that English works, one of which is the removal of all sorts of deliberate and tedious indications as to what one is talking about usually found in the Germanic languages, when one writes in English and uses the phrase 'other languages' but there is no specific language under discussion it should be understood that other languages refers to languages other than English.
- happytoexplain 5y agoThe parent is making the point that the language the grandparent is referring to is irrelevant to the language of an arbitrary individual included in the word "we".
- klabb3 5y agoIt was just rhetorical. As the sibling comment alluded to, it can come off as quite anglocentric. There is some historical relevance to this point - much of the technology we use today was created with such biases baked in. I apologize if this came off as snarky.
- hashimotonomora 5y agoIf I’m writing industrial automation, server-side applications, video rendering codecs, why do I need to make it much more complex with Unicode? I just need and want pure 1 byte ASCII.
- happytoexplain 5y agoThe grandparent's assertion was in the general case, not specific cases. However, even for your specific cases, it is possible - even likely - to at some point handle user-facing text (i.e. text that is reasonable to localize), at the very least in the form of error messages. Also, the complexity of Unicode is typically already well implemented by default in most environments, and the real issue only comes at the intersection of user-facing text and functional strings (e.g. URLs and filenames), which is a problem that Unicode is another tool to exploit, but not a problem that Unicode invented. We have been trying to detangle the dual nature of URLs/filenames for a long time. However, tangential to the grandparent's topic, I do imagine there are very specialized cases where Unicode is pretty objectively unnecessary.
- tom_ 5y agoBeing charitable: other than the one(s) in the current code page?
- bryanrasmussen 5y ago>Please correct me if I’m overreacting but I feel like Unicode in most cases is just a giant security flaw well ok, it seems to me that it is more likely that in most cases Unicode probably has no effect on security one way or another, but there are a few edge cases where it does. Here is one case where using a particular character in a filename led to problems 8 years ago in an operating system historically known for not having the greatest security. We can see here also a clear example of the edge case theory, almost every character in every language supported by Windows would not have caused any sort of problem when used in a filename. But there are a few characters where they would, this is probably an example of things programmers think they know about unicode or language or whatever, because we go around thinking that each unicode code point just represents a character in a language but some encodings have ways to represent little weird behaviors of particular languages, for example interlinear annotation characters https://www.unicode.org/charts/nameslist/n_FFF0.html https://www.unicode.org/charts/nameslist/n_FFF0.html and it is generally weird behaviors that end up being security hazards (not sure if there is any security hazard in interlinear annotation characters but wouldn't be surprised)
- mkotowski 5y ago> Will there be Unicode email addresses? Apparently, yes. [0] > making things more equitable for other languages and cultures Well, yes. People often hate how compilcated Unicode is, but they also tend to forget that even ASCII was not sufficient for writing English in a proper way. I am not by any means old, but even I still remember the period on the web when non-Unicode encodings were relatively common and how problematic it was. And, with assumption that by “other cultures and languages” you mean non-English speaking regions: Tolerate is a wrong word. Most people are not English native speakers. If we would go the route of deciding who is tolerating who, nations using Chinese Hanzi and its derivatives would be probably the first to claim they tolerate all the others by the speakers’ headcount alone. Moreover, deciding what is/can be executed is not so obvious in my opinion. PDFs can have JavaScript embedded, SVG too. Flash games? Python scripts on computers with python installed? It is not just the names, but the very knowledge of what can be considered “executable.” [0]: https://en.wikipedia.org/wiki/Unicode_and_email https://en.wikipedia.org/wiki/Unicode_and_email
- cryptonector 5y agoASCII is enough to write English, or, at least, it used to be back when ASCII was a multi-byte encoding for Latin! I know, it sounds crazy, but it used to be a thing to use over-strike using back-space. E.g., ë was e BS ", ø was o BS /, etc. It was a thing back in the days of typewriters. I believe this is the reason that upper-case letters in Spanish are allowed to be written without accents even when they are needed otherwise, that accented upper-case letters were hard to produce with typewriters and ASCII fonts.
- quietbritishjim 5y ago> ASCII was a multi-byte encoding for Latin The comment you're replying to just said ASCII without qualification. By definition this is not multibyte, or even whole byte (assuming by byte you mean octet). ASCII is a 7 bit standard, anything else is some other encoding.
- cryptonector 5y ago
- cryptonector 5y agoUnicode certainly has been a source of security vulnerabilities. Are you overreacting? Unless you were proposing we rip it out, no, you're not. Unicode isn't going anywhere. That's because the scripts that human languages use aren't going anywhere. And people do mix scripts, too. There are answers though. First, Unicode has been a learning experience, and we're still all learning. Second, one of the outputs of all that learning that has happened so far is UTR #36 (https://www.unicode.org/reports/tr36/ https://www.unicode.org/reports/tr36/), which does cover a lot of these things.
- drran 5y agoASCII certainly has been a source of security vulnerabilities.
- dredmorbius 5y agoExamples? I mean, obXkcd: https://xkcd.com/327/ https://xkcd.com/327/ But spelling out what you had in mind would be helpful here.
- drran 5y ago\n, \r, \v \0, ", ', </script>, &, %, 1 vs I vs l, O vs 0, etc.
- dredmorbius 5y agoThanks. Still, a limited set. Yes, errors are made and occur. They're reasonably easy to code defensively against. Unicode ... vastly expands the attack interface.
- drran 5y agoASCII is part of Unicode, so for Unicode it's also «a limited set. Yes, errors are made and occur. They're reasonably easy to code defensively against». > Unicode ... vastly expands the attack interface. Limit yourself to 0..9 to be safe.
- rhacker 5y agos/Unicode/Computers/ Same thing really.
- simion314 5y agoYou can have security issues with non escaped ASCII too, so you probably want to fix the tools and educate people about other languages and cultures (or even if you just educate them to be capitalists they would understand that selling their iDevice or CoolApp to make cash would require it to work with non english , so unicode means more money not we tolerate your language because we are too nice)
- zeepzeep 5y ago> Will there be Unicode email addresses? So you don't know about mailoji.com? It was on HN a couple months ago I believe.
- izacus 5y ago> Please correct me if I’m overreacting but I feel like Unicode in most cases is just a giant security flaw and the only reason we tolerate it is because it does tremendous good making things more equitable for other languages and cultures. Well, that's a lot like saying "software, in general, is just a giant security flaw we tolerate because it does useful things". It's kinda the point aint it? In 21st century, even Americans aren't able to write their own names and place names with pure ASCII anymore. So Unicode is there from necessity, not as an addon.
- dredmorbius 5y agoAll technology is a means to an end or goal. All technologies have intended / desireable, and unintended / undesireable effects. The more powerful and flexible a technology, the more likely there are unintended / undesireable effects. Many of those unintended / undesireable effects are themselves not obviously apparent, not immediately manifest, or both. All of which makes risk assessment all the more difficult. Software, in that sense, is inherently a security flaw, as it virtually always manifests unintended and/or undesireable effects. This isn't necessarily an argument to ban all software, though the Butlerian Jihad are taking notes. It is an argument, however, for acknowledging risks, raising awareness of them, and taking reasonable steps to guard against them.
- jrochkind1 5y agoThe only reason we "tolerate" unicode in, say, usernames, is because it does things like let people with non-ascii names have usernames with their names, right... but this is kind of a big reason!
- beeforpork 5y agoOK, I correct you: you overeact. And, yes, of course, email addresses in local languages, why the heck not? People in the world want to use their language! And yeah, even Hebrew, Tibetan, or Mongolian (in vertical script). That comment of yours reads to me like an lazy post of a unilingual or uniscriptal cultural imperialist who really does not care about other languages. Your attitude makes me rather angry, because Unicode is such an amazing achievement for the world, and your comment just comes off totally ignorant of that. It should be clear that combining the world's languages into a common standard is really, really difficult and will inevitably create something that is much more complex than your beloved ASCII. And it should also be clear that in such a complex standard, you cannot (a) solve all problems the first time you try, (b) you cannot try a second time, because no-one will adopt yet another such standard. So just read those Unicode documents in order to understand. Those people really try and there are security consideration documents, and they are extended all the time. And take those new security warnings for what they are: they are problems with a complex system. It's expected. So when they get known, react calmly and figure out whether you need to fix anything.
- MichaelZuo 5y agoThere doesn’t seem to be any way to fix this issue though? Unless you mean breaking the standard such as forcing special behaviour for this particular character such that it simply cannot be used in a filename. But that would defeat the purpose of Unicode, it would turn, at the very least, into Duocode.
- marcosdumay 5y agoHum... Most standards allow that kind of special cases. For filenames, POSIX doesn't, but your viewer can render it on any way (and it already renders many dangerous characters - ASCII had them too - in non-obvious ways).
- MichaelZuo 5y agoThis wouldn’t be a standard allowing a special case, it would be forcing all current and future machines to implement a special case. What if one vendor decides not to care? Then there would be a de facto split, it’s likely why the unicode committee doesn’t directly fix the issue.
- brandmeyer 5y agoI agree with you. TR36 is outright terrifying. I might feel more comfortable if the recommendations were already baked into foundational libraries like libicu, but they aren't. Its as if I'm wading into a world with as many footguns and silent killers as C's undefined behavior rules. In C we've spent decades developing tools to help detect and mitigate the problems and they are still regularly exploited. Unicode only has widely-available mitigation tools for a handful of the easiest problems to solve. C's flaws lead to the development of several different competing programming languages. Various flaws in cryptography lead to the development of algorithms that achieve the same result, but have far fewer implementation gotchas. IMO, the time has come for the development of a truly strict variant of Unicode that still supports the primary objective, but learns from Unicode's mistakes.
- dredmorbius 5y agoI think there's a fair case for that argument. There are certainly contexts in which Unicode unambiguously and demonstrably leads to security weaknesses and issues. See generally homoglyph attacks. At the heart of the lie and damage is the existence of a message which appears to say one thing but in fact says something different. It's the very limited nature of 7-bit ASCII, 128 characters in total, which provide its utility here. Yes, this means that texts in other languages must be represented by transliterations and approximations. That's ... simply a necessary trade-off. We see this in other domains, in which for the purposes of reducing ambiguity and emphasizing clarity standardisation is adopted. Internationally, air traffic control communications occur in English, and aircraft navigation uses feet (altitude) and nautical miles (dstance) units. Through the early 20th century, the language of diplomacy was French. The language of much scientific discourse, particularly in physics, was German. And for the Catholic Church, Latin was abandoned for mass only in the 1960s. Trading and maritime cultures tend to creat pidgin languages --- common amongst participants, but foreign to all, as distinguished from a creole, an amalgam language with native speakers. A key problem with computers is that the encodings used to create visual glyphs and the glyphs themselves are two distinct entities, and there can be a tremendous amount of ambiguity and confusion over similarly-appearing characters. Or, in many cases, glyphs cannot be represented at all. Where the full expressive value of language is required --- within texts, in descriptive fields, and in local or native contexts, I'm ... mostly ... open to Unicode (though it can still present problems). Where what is foremost in functionality is broad and universal understanding, selectinga small standardised and widely-recognised characterset has tremendous value, and no amount of emotive shaming changes that fact. As an example, OpenStreetMap generally represents local place names in local language and charactersets. This may preserve respect or integrity to the local culture. As a user of the map, however, not knowing that language or charcterset, it is utterly useless to me. Or, quite frankly, anyone not specifically literate in that language and writing system. It's worth considering that the characterset and language in question are themselves, adoptions and impositions: English was brought into Britain by invaders, the alphabet used itself is Roman, based on Greek and originally Phoenecian glyphs. English has adopted or incorporated terms from a huge set of other languages (rendering its own internal consistency ... low ... and making it confusing to learn). International communications and signage, at airports, on roadways, in public buildings, on electronic devices, aims at small message sets and consistent, widely-recognised symbols, shapes, fonts, and colours. That is a context in which the freedoms of unfettered Unicode adoption are in fact hazardous. (Yes, many of those symbols now have Unicode code points. It is the symbol set and glyph set which is constrained in public usage.) And the simple fact is that a widely recognised encoding system will most often reflect on some power structure or hierarchy, as that's how these encodings become known --- English, Roman Alphabet, French, German, Latin, etc. Small minor powers tend not to find their writing systems widely adopted (yes, there are exceptions: use of Greek within the Roman empire, Hindu numbering systems). Again, exceptions.
- rurban 5y agoThat's why unicode published the security guidelines and mechanisms to avoid such attacks. In 2004 already. The problem is that nobody cared. Browsers invented punycode instead of following tr39, email ditto. But ok, at least something. Java did it, cperl did, rust did it. Everybody else is vulnerable. Esp. most other programming languages, filesystems and login systems. https://github.com/rurban/libu8ident/blob/master/doc/c11.md https://github.com/rurban/libu8ident/blob/master/doc/c11.md