4 ms·

> I love Unicode quirks that OSes and big software developers miss It's more the development and specs of Unicode that did miss a lot of things. When Unicode

> I love Unicode quirks that OSes and big software developers miss It's more the development and specs of Unicode that did miss a lot of things. When Unicode came out Bruce Schneier envisioned all the attacks that'd inevitably come and he was, of course, totally right. As he put it a quarter of century ago (2000): "Unicode is just too complex to ever be secure". It's Unicode that did fuck up, not developers who are forced to deal with non-sensical things because somewhere a committee decided that it'd make sense to use a broken system for domain names (for example).

derefr 3y ago
Isn't almost all the complexity of Unicode inherent in the problem of trying to encode documents written in every written language, though? (Esp. when including documents written in multiple languages at once — and often switching languages within a single sentence, such as when a statement in one language quotes text in another language.) Or do you have thoughts on what a "simplified Unicode" — that accomplishes all the same goals — would look like?
- Nextgrid 3y ago
  Problem is that the executability of files is (at least on Windows) communicated in-band in the filename. In-band signalling was proven to be a bad idea ever since the early days of phone phreaking. Unicode is fine for what it is - conveying text across languages and scripts. The problem is that some environments use the filename as more than what it should be (a memo field for the user and not an instruction for the machine whether to execute the file as code).
- thaumasiotes 3y ago
  > Isn't almost all the complexity of Unicode inherent in the problem of trying to encode documents written in every written language, though? (Esp. when including documents written in multiple languages at once — and often switching languages within a single sentence, such as when a statement in one language quotes text in another language.) What's complex about that? That's just an ordinary sequence of code points, no different than if there was only one language involved. Unicode has problems where the same thing is represented several different ways. It's not a problem for different things to be represented in different ways.
  - avgcorrection 3y ago
    > Unicode has problems where the same thing is represented several different ways. That's needed for compatibility with legacy encodings.
    - derefr 3y ago
      Canonically-equivalent codepoints, yes, exist for embeddings of legacy encodings into Unicode. But when most people — including the GGP poster — complain about this, they aren't usually thinking of canonically-equivalent codepoints, but rather of homoglyphs: codepoints that just happen to have the same conventional pictorial representation. But which have different (collation/splitting/etc) properties; or, more interestingly, different (machine-readable!) semantics. Most often, phishing "text confusion" is done using homoglyphs. (And, in fact, in some text-processing libraries — e.g. Punycode — "text confusion" can only happen using homoglyphs, as canonically-equivalent Unicode codepoints get normalized to just one option.) The reason that the homoglyphic Latin "M" and Cyrillic "М" codepoints exist in Unicode isn't for legacy reasons. It's rather because Cyrillic "M" sorts after Cyrillic "Р" (and both of them come before Cyrillic "С"!) You can't have alternate collation orderings if you're using the same codepoints for both languages. The reason that both Greek "π" and mathematical "𝜋" exist in Unicode, is because one codepoint "is" a letter, and the other codepoint "is" a symbol. This "is"-ness has nothing to do with the glyphs, but rather is basically a type system inside Unicode, that user-agents besides those that render text often rely upon. "𝜋" (the symbol) can carry a particular semantic meaning in text to machine user-agents, that "π" (the letter) does not. Translation systems should treat them differently. Dictionaries should treat them differently. Filter rules for usernames should treat them differently. Etc. (And that's beside the fact that they also have different Unicode properties. "π", as a letter, doesn't create implicit split-boundaries on either side of it; while "𝜋", as a symbol, does.)
      - avgcorrection 3y ago
        Good points.
      - farnsworth 3y ago
avgcorrection 3y ago
Why do OSs like Linux distributions accept all characters in filenames except null and `/`? This causes problems even with ASCII (space, linefeed), and it's not like you need control characters like ACK in your filename.
- tbrownaw 3y ago
  Applying policies to restrict which `byte[]` values are valid index keys is not the "one thing" that a filesystem is supposed to do well.
  - carlmr 3y ago
    You're saying this as if it's self-evident. Why not?
    - yakubin 3y ago
      Because your filesystem is a storage layer. Its concern is storage, not display. It shouldn’t concern itself with how bytes are interpreted by your terminal, GUI, whatever. Those will also differ based on the particular program doing the interpreting.
remus 3y ago
> As he put it a quarter of century ago (2000): "Unicode is just too complex to ever be secure". > It's Unicode that did fuck up, not developers who are forced to deal with non-sensical things because somewhere a committee decided that it'd make sense to use a broken system for domain names (for example). I'm not sure that's unicode's fault. Their goal is to produce a standard for encoding ~all written text. It may seem like this would be something cool to support in your program of choice, but in lots of applications that flexibility leads to vulnerabilities. I think developers need to be aware of these risks and design around them (for example by limiting the allowed characters in certain contexts).
- avgcorrection 3y ago
  And even if you restrict yourself to the venerable US-ASCII you might have to check for metacharacters to avoid XSS.
BeFlatXIII 3y ago
What should Unicode have done instead?

derefr 3y ago

Isn't almost all the complexity of Unicode inherent in the problem of trying to encode documents written in every written language, though? (Esp. when including documents written in multiple languages at once — and often switching languages within a single sentence, such as when a statement in one language quotes text in another language.) Or do you have thoughts on what a "simplified Unicode" — that accomplishes all the same goals — would look like?

Nextgrid 3y ago
Problem is that the executability of files is (at least on Windows) communicated in-band in the filename. In-band signalling was proven to be a bad idea ever since the early days of phone phreaking. Unicode is fine for what it is - conveying text across languages and scripts. The problem is that some environments use the filename as more than what it should be (a memo field for the user and not an instruction for the machine whether to execute the file as code).
thaumasiotes 3y ago
> Isn't almost all the complexity of Unicode inherent in the problem of trying to encode documents written in every written language, though? (Esp. when including documents written in multiple languages at once — and often switching languages within a single sentence, such as when a statement in one language quotes text in another language.) What's complex about that? That's just an ordinary sequence of code points, no different than if there was only one language involved. Unicode has problems where the same thing is represented several different ways. It's not a problem for different things to be represented in different ways.
- avgcorrection 3y ago
  > Unicode has problems where the same thing is represented several different ways. That's needed for compatibility with legacy encodings.
  - derefr 3y ago
    Canonically-equivalent codepoints, yes, exist for embeddings of legacy encodings into Unicode. But when most people — including the GGP poster — complain about this, they aren't usually thinking of canonically-equivalent codepoints, but rather of homoglyphs: codepoints that just happen to have the same conventional pictorial representation. But which have different (collation/splitting/etc) properties; or, more interestingly, different (machine-readable!) semantics. Most often, phishing "text confusion" is done using homoglyphs. (And, in fact, in some text-processing libraries — e.g. Punycode — "text confusion" can only happen using homoglyphs, as canonically-equivalent Unicode codepoints get normalized to just one option.) The reason that the homoglyphic Latin "M" and Cyrillic "М" codepoints exist in Unicode isn't for legacy reasons. It's rather because Cyrillic "M" sorts after Cyrillic "Р" (and both of them come before Cyrillic "С"!) You can't have alternate collation orderings if you're using the same codepoints for both languages. The reason that both Greek "π" and mathematical "𝜋" exist in Unicode, is because one codepoint "is" a letter, and the other codepoint "is" a symbol. This "is"-ness has nothing to do with the glyphs, but rather is basically a type system inside Unicode, that user-agents besides those that render text often rely upon. "𝜋" (the symbol) can carry a particular semantic meaning in text to machine user-agents, that "π" (the letter) does not. Translation systems should treat them differently. Dictionaries should treat them differently. Filter rules for usernames should treat them differently. Etc. (And that's beside the fact that they also have different Unicode properties. "π", as a letter, doesn't create implicit split-boundaries on either side of it; while "𝜋", as a symbol, does.)
    - avgcorrection 3y ago
      Good points.
    - farnsworth 3y ago

avgcorrection 3y ago

Why do OSs like Linux distributions accept all characters in filenames except null and `/`? This causes problems even with ASCII (space, linefeed), and it's not like you need control characters like ACK in your filename.

tbrownaw 3y ago
Applying policies to restrict which `byte[]` values are valid index keys is not the "one thing" that a filesystem is supposed to do well.
- carlmr 3y ago
  You're saying this as if it's self-evident. Why not?
  - yakubin 3y ago
    Because your filesystem is a storage layer. Its concern is storage, not display. It shouldn’t concern itself with how bytes are interpreted by your terminal, GUI, whatever. Those will also differ based on the particular program doing the interpreting.

remus 3y ago

> As he put it a quarter of century ago (2000): "Unicode is just too complex to ever be secure". > It's Unicode that did fuck up, not developers who are forced to deal with non-sensical things because somewhere a committee decided that it'd make sense to use a broken system for domain names (for example). I'm not sure that's unicode's fault. Their goal is to produce a standard for encoding ~all written text. It may seem like this would be something cool to support in your program of choice, but in lots of applications that flexibility leads to vulnerabilities. I think developers need to be aware of these risks and design around them (for example by limiting the allowed characters in certain contexts).

avgcorrection 3y ago
And even if you restrict yourself to the venerable US-ASCII you might have to check for metacharacters to avoid XSS.

BeFlatXIII 3y ago

What should Unicode have done instead?