6 ms·

What this gets right down to is that Unicode is a flawed idea: the meaning/behavior/whatever of characters is insanely dependent on their context. The problem

What this gets right down to is that Unicode is a flawed idea: the meaning/behavior/whatever of characters is insanely dependent on their context. The problem was never gazillions of code pages, but our inability to write C to deal with that amount of complexity circa 1990. With modern machines, and good programming languages with good type systems, I absolutely think we could store a language per string, and concatenate into a polylinguistic rope if needed. This would hopefully push us away from stringly-typed crap in general.

throwaway_pdp09 6y ago
> the meaning/behavior/whatever of characters is insanely dependent on their context I wish you would give an example instead of just proclaiming crapness. You know, so we n00bs can learn something.
- lazulicurio 6y ago
  How about: case folding for the letter 'I' is dependent on whether the locale is Turkish or not. ;)
- toast0 6y ago
  Different languages have different rules for change case (as seen here) or what to do when translitterating to 7-bit ascii, in French, you can mostly drop accents if you need to, in German, you need to transform an umlaut to an e following the vowel. Of course, many languages don't have a way to translitterate to 7-bit ascii. Sorting of strings is language dependent, but I don't know that there's a defined order for mixed language lists, so I guess user's context works if you're sorting for user purposes, but if you're sorting for machine purposes, you better not use the locale aware sort without telling it a hardcoded locale that doesn't change between localization library versions.
- throwaway_pdp09 6y ago
  @toast0, @lazulicurio, both of your points seem to illustrate the complexities of the languages, not "...that Unicode is a flawed idea" as the original poster said. AFAIKS this is intrinsic complexity showing itself and does not make any indication of how it should be done correctly, or better.
  - lazulicurio 6y ago
    > both of your points seem to illustrate the complexities of the languages, not "...that Unicode is a flawed idea" The flaw in Unicode is that it punts on the intrinsic complexity---pretending that codepoints have language-independent, plain-text, semantic meaning. A couple of threads that have molded my views over time: I can't write my name in Unicode https://news.ycombinator.com/item?id=9219162 https://news.ycombinator.com/item?id=9219162 (Specifically these two comments https://news.ycombinator.com/item?id=9220530 https://news.ycombinator.com/item?id=9220530 and https://news.ycombinator.com/item?id=9220970 https://news.ycombinator.com/item?id=9220970) Why isn't the external link symbol in Unicode? https://news.ycombinator.com/item?id=23016832 https://news.ycombinator.com/item?id=23016832
    - Ericson2314 6y ago
      > The flaw in Unicode is that it punts on the intrinsic complexity---pretending that codepoints have language-independent, plain-text, semantic meaning. > Pretending "plain text" isn't an oxymoron FTFY :)
  - Ericson2314 6y ago
    The benefit of looking at languages/scripts in isolation is that the combinatorial explosion of all languages/scripts at once is dodged. E.g. lookalike charaters, and social engineering by using a vs а. (One is Cyrillic). I don't want to even define "a == а". I want Latin and Cyrillic to be different types of characters, and that expression to be ill-typed. This solves the Turkish problem, where the upper case I is two different charters in two different types (Turkish Roman script?), and the case folding functions likewise have disjoint types.
    - a1369209993 6y ago
      > I want Latin and Cyrillic to be different types of characters How do you concatenate English and Ру́сская text, and what is the type of this sentence?
      - 6y ago
arcticbull 6y ago
Unicode goes to great pains to avoid ascribing any meaning/behavior/whatever to character sets. Because to your point you can’t. Unicode is actually incredibly well thought out. That’s why we have values, code points and grapheme clusters. I don’t think the Unicode standard even defines casing except in the human-readable names ascribed to code points. If you want to build a polylinguistic rope you can certainly do that with Unicode, but you won’t have solved anything because language alone without context doesn’t really define many of the operations you’re describing. The answer is usually the same as “doctor it hurts when I...” — stop doing it. Stop manipulating user input without context. Stop trying to limit user visible strings by character count, use pixel width in the rendered font. And so on.
- jfk13 6y ago
  > I don’t think the Unicode standard even defines casing except in the human-readable names ascribed to code points Sure it does; the Unicode Character Database includes fields for the lowercase, uppercase and titlecase mappings. But it also acknowledges that these are just default mappings, and may need to be tailored for specific languages/locales.
- Ericson2314 6y ago
  Unicode is well thought out! And that's what makes it hard to critique :). I think it's one of the best-maintained, well-thought out standards there is, but I still think the premise is wrong. If all that good effort went into something along the lines I am describing, where languages, or at least scripts, cannot be arbitrary mixed at the character level, I think we would have an even better result with the same level of effort.
  - arcticbull 6y ago
    If you treat Unicode as your backing representation -- a pile of glyphs -- you can build what you're asking on top right?
    - Ericson2314 6y ago

throwaway_pdp09 6y ago

> the meaning/behavior/whatever of characters is insanely dependent on their context I wish you would give an example instead of just proclaiming crapness. You know, so we n00bs can learn something.

lazulicurio 6y ago
How about: case folding for the letter 'I' is dependent on whether the locale is Turkish or not. ;)
toast0 6y ago
Different languages have different rules for change case (as seen here) or what to do when translitterating to 7-bit ascii, in French, you can mostly drop accents if you need to, in German, you need to transform an umlaut to an e following the vowel. Of course, many languages don't have a way to translitterate to 7-bit ascii. Sorting of strings is language dependent, but I don't know that there's a defined order for mixed language lists, so I guess user's context works if you're sorting for user purposes, but if you're sorting for machine purposes, you better not use the locale aware sort without telling it a hardcoded locale that doesn't change between localization library versions.
throwaway_pdp09 6y ago
@toast0, @lazulicurio, both of your points seem to illustrate the complexities of the languages, not "...that Unicode is a flawed idea" as the original poster said. AFAIKS this is intrinsic complexity showing itself and does not make any indication of how it should be done correctly, or better.
- lazulicurio 6y ago
  > both of your points seem to illustrate the complexities of the languages, not "...that Unicode is a flawed idea" The flaw in Unicode is that it punts on the intrinsic complexity---pretending that codepoints have language-independent, plain-text, semantic meaning. A couple of threads that have molded my views over time: I can't write my name in Unicode https://news.ycombinator.com/item?id=9219162 https://news.ycombinator.com/item?id=9219162 (Specifically these two comments https://news.ycombinator.com/item?id=9220530 https://news.ycombinator.com/item?id=9220530 and https://news.ycombinator.com/item?id=9220970 https://news.ycombinator.com/item?id=9220970) Why isn't the external link symbol in Unicode? https://news.ycombinator.com/item?id=23016832 https://news.ycombinator.com/item?id=23016832
  - Ericson2314 6y ago
    > The flaw in Unicode is that it punts on the intrinsic complexity---pretending that codepoints have language-independent, plain-text, semantic meaning. > Pretending "plain text" isn't an oxymoron FTFY :)
- Ericson2314 6y ago
  The benefit of looking at languages/scripts in isolation is that the combinatorial explosion of all languages/scripts at once is dodged. E.g. lookalike charaters, and social engineering by using a vs а. (One is Cyrillic). I don't want to even define "a == а". I want Latin and Cyrillic to be different types of characters, and that expression to be ill-typed. This solves the Turkish problem, where the upper case I is two different charters in two different types (Turkish Roman script?), and the case folding functions likewise have disjoint types.
  - a1369209993 6y ago
    > I want Latin and Cyrillic to be different types of characters How do you concatenate English and Ру́сская text, and what is the type of this sentence?
    - 6y ago

arcticbull 6y ago

Unicode goes to great pains to avoid ascribing any meaning/behavior/whatever to character sets. Because to your point you can’t. Unicode is actually incredibly well thought out. That’s why we have values, code points and grapheme clusters. I don’t think the Unicode standard even defines casing except in the human-readable names ascribed to code points. If you want to build a polylinguistic rope you can certainly do that with Unicode, but you won’t have solved anything because language alone without context doesn’t really define many of the operations you’re describing. The answer is usually the same as “doctor it hurts when I...” — stop doing it. Stop manipulating user input without context. Stop trying to limit user visible strings by character count, use pixel width in the rendered font. And so on.

jfk13 6y ago
> I don’t think the Unicode standard even defines casing except in the human-readable names ascribed to code points Sure it does; the Unicode Character Database includes fields for the lowercase, uppercase and titlecase mappings. But it also acknowledges that these are just default mappings, and may need to be tailored for specific languages/locales.
Ericson2314 6y ago
Unicode is well thought out! And that's what makes it hard to critique :). I think it's one of the best-maintained, well-thought out standards there is, but I still think the premise is wrong. If all that good effort went into something along the lines I am describing, where languages, or at least scripts, cannot be arbitrary mixed at the character level, I think we would have an even better result with the same level of effort.
- arcticbull 6y ago
  If you treat Unicode as your backing representation -- a pile of glyphs -- you can build what you're asking on top right?
  - Ericson2314 6y ago