4 ms·

Along similar lines, I've thought that it would be useful if Unicode included language marks (i.e. codepoints to identify blocks of text as being written in a s

Along similar lines, I've thought that it would be useful if Unicode included language marks (i.e. codepoints to identify blocks of text as being written in a specific language). It would be strictly more useful than the barebones left-to-right/right-to-left marks (U+200E/U+200F) when deciding how to process and display text. And it would be a step towards correcting the mess that was Han unification.

Ericson2314 6y ago
What this gets right down to is that Unicode is a flawed idea: the meaning/behavior/whatever of characters is insanely dependent on their context. The problem was never gazillions of code pages, but our inability to write C to deal with that amount of complexity circa 1990. With modern machines, and good programming languages with good type systems, I absolutely think we could store a language per string, and concatenate into a polylinguistic rope if needed. This would hopefully push us away from stringly-typed crap in general.
- throwaway_pdp09 6y ago
  > the meaning/behavior/whatever of characters is insanely dependent on their context I wish you would give an example instead of just proclaiming crapness. You know, so we n00bs can learn something.
  - lazulicurio 6y ago
    How about: case folding for the letter 'I' is dependent on whether the locale is Turkish or not. ;)
  - toast0 6y ago
    Different languages have different rules for change case (as seen here) or what to do when translitterating to 7-bit ascii, in French, you can mostly drop accents if you need to, in German, you need to transform an umlaut to an e following the vowel. Of course, many languages don't have a way to translitterate to 7-bit ascii. Sorting of strings is language dependent, but I don't know that there's a defined order for mixed language lists, so I guess user's context works if you're sorting for user purposes, but if you're sorting for machine purposes, you better not use the locale aware sort without telling it a hardcoded locale that doesn't change between localization library versions.
  - throwaway_pdp09 6y ago
    @toast0, @lazulicurio, both of your points seem to illustrate the complexities of the languages, not "...that Unicode is a flawed idea" as the original poster said. AFAIKS this is intrinsic complexity showing itself and does not make any indication of how it should be done correctly, or better.
    - lazulicurio 6y ago
      > both of your points seem to illustrate the complexities of the languages, not "...that Unicode is a flawed idea" The flaw in Unicode is that it punts on the intrinsic complexity---pretending that codepoints have language-independent, plain-text, semantic meaning. A couple of threads that have molded my views over time: I can't write my name in Unicode https://news.ycombinator.com/item?id=9219162 https://news.ycombinator.com/item?id=9219162 (Specifically these two comments https://news.ycombinator.com/item?id=9220530 https://news.ycombinator.com/item?id=9220530 and https://news.ycombinator.com/item?id=9220970 https://news.ycombinator.com/item?id=9220970) Why isn't the external link symbol in Unicode? https://news.ycombinator.com/item?id=23016832 https://news.ycombinator.com/item?id=23016832
      - Ericson2314 6y ago
        > The flaw in Unicode is that it punts on the intrinsic complexity---pretending that codepoints have language-independent, plain-text, semantic meaning. > Pretending "plain text" isn't an oxymoron FTFY :)
    - Ericson2314 6y ago
      The benefit of looking at languages/scripts in isolation is that the combinatorial explosion of all languages/scripts at once is dodged. E.g. lookalike charaters, and social engineering by using a vs а. (One is Cyrillic). I don't want to even define "a == а". I want Latin and Cyrillic to be different types of characters, and that expression to be ill-typed. This solves the Turkish problem, where the upper case I is two different charters in two different types (Turkish Roman script?), and the case folding functions likewise have disjoint types.
- arcticbull 6y ago
  Unicode goes to great pains to avoid ascribing any meaning/behavior/whatever to character sets. Because to your point you can’t. Unicode is actually incredibly well thought out. That’s why we have values, code points and grapheme clusters. I don’t think the Unicode standard even defines casing except in the human-readable names ascribed to code points. If you want to build a polylinguistic rope you can certainly do that with Unicode, but you won’t have solved anything because language alone without context doesn’t really define many of the operations you’re describing. The answer is usually the same as “doctor it hurts when I...” — stop doing it. Stop manipulating user input without context. Stop trying to limit user visible strings by character count, use pixel width in the rendered font. And so on.
  - jfk13 6y ago
    > I don’t think the Unicode standard even defines casing except in the human-readable names ascribed to code points Sure it does; the Unicode Character Database includes fields for the lowercase, uppercase and titlecase mappings. But it also acknowledges that these are just default mappings, and may need to be tailored for specific languages/locales.
  - Ericson2314 6y ago
    Unicode is well thought out! And that's what makes it hard to critique :). I think it's one of the best-maintained, well-thought out standards there is, but I still think the premise is wrong. If all that good effort went into something along the lines I am describing, where languages, or at least scripts, cannot be arbitrary mixed at the character level, I think we would have an even better result with the same level of effort.
    - arcticbull 6y ago
      If you treat Unicode as your backing representation -- a pile of glyphs -- you can build what you're asking on top right?
      - Ericson2314 6y ago
jwilk 6y ago
See RFC 2482 — Language Tagging in Unicode Plain Text: https://tools.ietf.org/html/rfc2482 https://tools.ietf.org/html/rfc2482 But it was deprecated later on: https://tools.ietf.org/html/rfc6082 https://tools.ietf.org/html/rfc6082
- lazulicurio 6y ago
  Interesting. Unfortunate that the deprecation notice doesn't include much rationale. I found at least one mail thread about it[1], which seems to confirm that the main thought was that semantic information about text should be handled at a higher layer (e.g. XML). I can understand that argument for a general purpose tagging mechanism, but language and glyphs are strongly semantically linked. (Somewhat ironically, the previous thread on that mailing list is about the struggles of case folding in a general fashion across multiple language scripts[2]) Edit: I also found [3], which offers the following: ---- - Most of the data sources used to assemble the documents on the Web will not contain these characters; producers, in the process of assembling or serializing the data, will need to introspect and insert the characters as needed—changing the data from the original source. Consumers must then deserialize and introspect the information using an identical agreement. The consumer has no way of knowing if the characters found in the data were inserted by the producer (and should be removed) or if the characters were part of the source data. Overzealous producers might introduce additional and unnecessary characters, for example adding an additional layer of bidi control codes to a string that would not otherwise require it. Equally, an overzealous consumer might remove characters that are needed by or intended for downstream processes. - Another challenge is that many applications that use these data formats have limitations on content, such as length limits or character set restrictions. Inserting additional characters into the data may violate these externally applied requirements, and interfere with processing. In the worst case, portions (or all of) the data value itself might be rejected, corrupted, or lost as a result. - Inserting additional characters changes the identity of the string. This may have important consequences in certain contexts. - Inserting and removing characters from the string is not a common operation for most data serialization libraries. Any processing that adds language or direction controls would need to introspect the string to see if these are already present or might need to do other processing to insert or modify the contents of the string as part of serializing the data. ---- Other than #3 (the one about string identity), I find these wholly unpersuasive. And even #3 isn't even that great a reason considering that programmatic processors have to deal with that issue anyway due to case folding. [1] https://www.unicode.org/mail-arch/unicode-ml/y2010-m11/0039.html https://www.unicode.org/mail-arch/unicode-ml/y2010-m11/0039.... [2] https://www.unicode.org/mail-arch/unicode-ml/y2010-m11/0038.html https://www.unicode.org/mail-arch/unicode-ml/y2010-m11/0038.... [3] https://www.w3.org/TR/string-meta/ https://www.w3.org/TR/string-meta/
kevin_thibedeau 6y ago
Unicode supported this with tag sequences but that is deprecated and unlikely to work with modern libs.

Ericson2314 6y ago

What this gets right down to is that Unicode is a flawed idea: the meaning/behavior/whatever of characters is insanely dependent on their context. The problem was never gazillions of code pages, but our inability to write C to deal with that amount of complexity circa 1990. With modern machines, and good programming languages with good type systems, I absolutely think we could store a language per string, and concatenate into a polylinguistic rope if needed. This would hopefully push us away from stringly-typed crap in general.

throwaway_pdp09 6y ago
> the meaning/behavior/whatever of characters is insanely dependent on their context I wish you would give an example instead of just proclaiming crapness. You know, so we n00bs can learn something.
- lazulicurio 6y ago
  How about: case folding for the letter 'I' is dependent on whether the locale is Turkish or not. ;)
- toast0 6y ago
  Different languages have different rules for change case (as seen here) or what to do when translitterating to 7-bit ascii, in French, you can mostly drop accents if you need to, in German, you need to transform an umlaut to an e following the vowel. Of course, many languages don't have a way to translitterate to 7-bit ascii. Sorting of strings is language dependent, but I don't know that there's a defined order for mixed language lists, so I guess user's context works if you're sorting for user purposes, but if you're sorting for machine purposes, you better not use the locale aware sort without telling it a hardcoded locale that doesn't change between localization library versions.
- throwaway_pdp09 6y ago
  @toast0, @lazulicurio, both of your points seem to illustrate the complexities of the languages, not "...that Unicode is a flawed idea" as the original poster said. AFAIKS this is intrinsic complexity showing itself and does not make any indication of how it should be done correctly, or better.
  - lazulicurio 6y ago
    > both of your points seem to illustrate the complexities of the languages, not "...that Unicode is a flawed idea" The flaw in Unicode is that it punts on the intrinsic complexity---pretending that codepoints have language-independent, plain-text, semantic meaning. A couple of threads that have molded my views over time: I can't write my name in Unicode https://news.ycombinator.com/item?id=9219162 https://news.ycombinator.com/item?id=9219162 (Specifically these two comments https://news.ycombinator.com/item?id=9220530 https://news.ycombinator.com/item?id=9220530 and https://news.ycombinator.com/item?id=9220970 https://news.ycombinator.com/item?id=9220970) Why isn't the external link symbol in Unicode? https://news.ycombinator.com/item?id=23016832 https://news.ycombinator.com/item?id=23016832
    - Ericson2314 6y ago
      > The flaw in Unicode is that it punts on the intrinsic complexity---pretending that codepoints have language-independent, plain-text, semantic meaning. > Pretending "plain text" isn't an oxymoron FTFY :)
  - Ericson2314 6y ago
    The benefit of looking at languages/scripts in isolation is that the combinatorial explosion of all languages/scripts at once is dodged. E.g. lookalike charaters, and social engineering by using a vs а. (One is Cyrillic). I don't want to even define "a == а". I want Latin and Cyrillic to be different types of characters, and that expression to be ill-typed. This solves the Turkish problem, where the upper case I is two different charters in two different types (Turkish Roman script?), and the case folding functions likewise have disjoint types.
arcticbull 6y ago
Unicode goes to great pains to avoid ascribing any meaning/behavior/whatever to character sets. Because to your point you can’t. Unicode is actually incredibly well thought out. That’s why we have values, code points and grapheme clusters. I don’t think the Unicode standard even defines casing except in the human-readable names ascribed to code points. If you want to build a polylinguistic rope you can certainly do that with Unicode, but you won’t have solved anything because language alone without context doesn’t really define many of the operations you’re describing. The answer is usually the same as “doctor it hurts when I...” — stop doing it. Stop manipulating user input without context. Stop trying to limit user visible strings by character count, use pixel width in the rendered font. And so on.
- jfk13 6y ago
  > I don’t think the Unicode standard even defines casing except in the human-readable names ascribed to code points Sure it does; the Unicode Character Database includes fields for the lowercase, uppercase and titlecase mappings. But it also acknowledges that these are just default mappings, and may need to be tailored for specific languages/locales.
- Ericson2314 6y ago
  Unicode is well thought out! And that's what makes it hard to critique :). I think it's one of the best-maintained, well-thought out standards there is, but I still think the premise is wrong. If all that good effort went into something along the lines I am describing, where languages, or at least scripts, cannot be arbitrary mixed at the character level, I think we would have an even better result with the same level of effort.
  - arcticbull 6y ago
    If you treat Unicode as your backing representation -- a pile of glyphs -- you can build what you're asking on top right?
    - Ericson2314 6y ago

jwilk 6y ago

See RFC 2482 — Language Tagging in Unicode Plain Text: https://tools.ietf.org/html/rfc2482 https://tools.ietf.org/html/rfc2482 But it was deprecated later on: https://tools.ietf.org/html/rfc6082 https://tools.ietf.org/html/rfc6082

lazulicurio 6y ago
Interesting. Unfortunate that the deprecation notice doesn't include much rationale. I found at least one mail thread about it[1], which seems to confirm that the main thought was that semantic information about text should be handled at a higher layer (e.g. XML). I can understand that argument for a general purpose tagging mechanism, but language and glyphs are strongly semantically linked. (Somewhat ironically, the previous thread on that mailing list is about the struggles of case folding in a general fashion across multiple language scripts[2]) Edit: I also found [3], which offers the following: ---- - Most of the data sources used to assemble the documents on the Web will not contain these characters; producers, in the process of assembling or serializing the data, will need to introspect and insert the characters as needed—changing the data from the original source. Consumers must then deserialize and introspect the information using an identical agreement. The consumer has no way of knowing if the characters found in the data were inserted by the producer (and should be removed) or if the characters were part of the source data. Overzealous producers might introduce additional and unnecessary characters, for example adding an additional layer of bidi control codes to a string that would not otherwise require it. Equally, an overzealous consumer might remove characters that are needed by or intended for downstream processes. - Another challenge is that many applications that use these data formats have limitations on content, such as length limits or character set restrictions. Inserting additional characters into the data may violate these externally applied requirements, and interfere with processing. In the worst case, portions (or all of) the data value itself might be rejected, corrupted, or lost as a result. - Inserting additional characters changes the identity of the string. This may have important consequences in certain contexts. - Inserting and removing characters from the string is not a common operation for most data serialization libraries. Any processing that adds language or direction controls would need to introspect the string to see if these are already present or might need to do other processing to insert or modify the contents of the string as part of serializing the data. ---- Other than #3 (the one about string identity), I find these wholly unpersuasive. And even #3 isn't even that great a reason considering that programmatic processors have to deal with that issue anyway due to case folding. [1] https://www.unicode.org/mail-arch/unicode-ml/y2010-m11/0039.html https://www.unicode.org/mail-arch/unicode-ml/y2010-m11/0039.... [2] https://www.unicode.org/mail-arch/unicode-ml/y2010-m11/0038.html https://www.unicode.org/mail-arch/unicode-ml/y2010-m11/0038.... [3] https://www.w3.org/TR/string-meta/ https://www.w3.org/TR/string-meta/

kevin_thibedeau 6y ago

Unicode supported this with tag sequences but that is deprecated and unlikely to work with modern libs.