4 ms·

Isn't almost all the complexity of Unicode inherent in the problem of trying to encode documents written in every written language, though? (Esp. when including

by derefr 3y ago

Isn't almost all the complexity of Unicode inherent in the problem of trying to encode documents written in every written language, though? (Esp. when including documents written in multiple languages at once — and often switching languages within a single sentence, such as when a statement in one language quotes text in another language.) Or do you have thoughts on what a "simplified Unicode" — that accomplishes all the same goals — would look like?

Nextgrid 3y ago
Problem is that the executability of files is (at least on Windows) communicated in-band in the filename. In-band signalling was proven to be a bad idea ever since the early days of phone phreaking. Unicode is fine for what it is - conveying text across languages and scripts. The problem is that some environments use the filename as more than what it should be (a memo field for the user and not an instruction for the machine whether to execute the file as code).
thaumasiotes 3y ago
> Isn't almost all the complexity of Unicode inherent in the problem of trying to encode documents written in every written language, though? (Esp. when including documents written in multiple languages at once — and often switching languages within a single sentence, such as when a statement in one language quotes text in another language.) What's complex about that? That's just an ordinary sequence of code points, no different than if there was only one language involved. Unicode has problems where the same thing is represented several different ways. It's not a problem for different things to be represented in different ways.
- avgcorrection 3y ago
  > Unicode has problems where the same thing is represented several different ways. That's needed for compatibility with legacy encodings.
  - derefr 3y ago
    Canonically-equivalent codepoints, yes, exist for embeddings of legacy encodings into Unicode. But when most people — including the GGP poster — complain about this, they aren't usually thinking of canonically-equivalent codepoints, but rather of homoglyphs: codepoints that just happen to have the same conventional pictorial representation. But which have different (collation/splitting/etc) properties; or, more interestingly, different (machine-readable!) semantics. Most often, phishing "text confusion" is done using homoglyphs. (And, in fact, in some text-processing libraries — e.g. Punycode — "text confusion" can only happen using homoglyphs, as canonically-equivalent Unicode codepoints get normalized to just one option.) The reason that the homoglyphic Latin "M" and Cyrillic "М" codepoints exist in Unicode isn't for legacy reasons. It's rather because Cyrillic "M" sorts after Cyrillic "Р" (and both of them come before Cyrillic "С"!) You can't have alternate collation orderings if you're using the same codepoints for both languages. The reason that both Greek "π" and mathematical "𝜋" exist in Unicode, is because one codepoint "is" a letter, and the other codepoint "is" a symbol. This "is"-ness has nothing to do with the glyphs, but rather is basically a type system inside Unicode, that user-agents besides those that render text often rely upon. "𝜋" (the symbol) can carry a particular semantic meaning in text to machine user-agents, that "π" (the letter) does not. Translation systems should treat them differently. Dictionaries should treat them differently. Filter rules for usernames should treat them differently. Etc. (And that's beside the fact that they also have different Unicode properties. "π", as a letter, doesn't create implicit split-boundaries on either side of it; while "𝜋", as a symbol, does.)
    - avgcorrection 3y ago
      Good points.
    - farnsworth 3y ago
      I hadn't known that point about sort order. Wait, so suppose a language that uses the same alphabet as English used a reversed alphabet ordering. Would unicode include z-a somewhere else to support that language?
      - 3y ago

by derefr 3y ago

Nextgrid 3y ago

Problem is that the executability of files is (at least on Windows) communicated in-band in the filename. In-band signalling was proven to be a bad idea ever since the early days of phone phreaking. Unicode is fine for what it is - conveying text across languages and scripts. The problem is that some environments use the filename as more than what it should be (a memo field for the user and not an instruction for the machine whether to execute the file as code).

thaumasiotes 3y ago

> Isn't almost all the complexity of Unicode inherent in the problem of trying to encode documents written in every written language, though? (Esp. when including documents written in multiple languages at once — and often switching languages within a single sentence, such as when a statement in one language quotes text in another language.) What's complex about that? That's just an ordinary sequence of code points, no different than if there was only one language involved. Unicode has problems where the same thing is represented several different ways. It's not a problem for different things to be represented in different ways.

avgcorrection 3y ago
> Unicode has problems where the same thing is represented several different ways. That's needed for compatibility with legacy encodings.
- derefr 3y ago
  Canonically-equivalent codepoints, yes, exist for embeddings of legacy encodings into Unicode. But when most people — including the GGP poster — complain about this, they aren't usually thinking of canonically-equivalent codepoints, but rather of homoglyphs: codepoints that just happen to have the same conventional pictorial representation. But which have different (collation/splitting/etc) properties; or, more interestingly, different (machine-readable!) semantics. Most often, phishing "text confusion" is done using homoglyphs. (And, in fact, in some text-processing libraries — e.g. Punycode — "text confusion" can only happen using homoglyphs, as canonically-equivalent Unicode codepoints get normalized to just one option.) The reason that the homoglyphic Latin "M" and Cyrillic "М" codepoints exist in Unicode isn't for legacy reasons. It's rather because Cyrillic "M" sorts after Cyrillic "Р" (and both of them come before Cyrillic "С"!) You can't have alternate collation orderings if you're using the same codepoints for both languages. The reason that both Greek "π" and mathematical "𝜋" exist in Unicode, is because one codepoint "is" a letter, and the other codepoint "is" a symbol. This "is"-ness has nothing to do with the glyphs, but rather is basically a type system inside Unicode, that user-agents besides those that render text often rely upon. "𝜋" (the symbol) can carry a particular semantic meaning in text to machine user-agents, that "π" (the letter) does not. Translation systems should treat them differently. Dictionaries should treat them differently. Filter rules for usernames should treat them differently. Etc. (And that's beside the fact that they also have different Unicode properties. "π", as a letter, doesn't create implicit split-boundaries on either side of it; while "𝜋", as a symbol, does.)
  - avgcorrection 3y ago
    Good points.
  - farnsworth 3y ago
    I hadn't known that point about sort order. Wait, so suppose a language that uses the same alphabet as English used a reversed alphabet ordering. Would unicode include z-a somewhere else to support that language?
    - 3y ago