3 ms·

> In practice in the pre-unicode world fonts were encoding-specific, so the encoding did help Nobody wants to go back to a world with hundreds of relevant and

> In practice in the pre-unicode world fonts were encoding-specific, so the encoding did help Nobody wants to go back to a world with hundreds of relevant and thousands of legacy encodings. > when you mixed up your encodings you'd get something that was obviously broken even to someone doesn't speak Chinese or Japanese. Obviously much better when nobody can read anything than forcing some people to read texts where there's the occasional odd ('non-native' if you exaggerate a bit) font style choice. > We don't need codepoints for letters that are half-Japanese half-Korean, or every other conceivable variant Things are more complicated than that, I'm afraid. For one thing there's time, so setting the language in your HTML document or element to Japanese will give what is considered correct in Japanese today, not what was considered correct pre-war or any one time before that. There's no formal way to express that. Second, stylistic choices on one and language + region selector on the other are cross-cutting concerns. I have to say that I find your criticism not very well founded. What you can do today is (taking HTML+CSS as the obvious choice) is proper font choice, `@font-face` declarations, proper downloadable fonts, proper configuration of OpenType font feature flags, if necessary on the basis of Unicode code point ranges. (There's also Unicode variation selectors, bit I've never worked with those so can't say how well those work.) This will get you a long, long way to achieving correct and typographically pleasing output for any text be it Chinese traditional or simplified, Japanese, or Korean. It is some work, but then it is a vexing and somewhat convoluted problem, too. I think one should give font designers more credit here because good CJK fonts typically go to great lengths to satisfy even the finickier type heads among us, enabling variations that I'd frankly just gloss over for heavens sake. It's not like the pre-Unicode chaos of mutually incompatible encodings did anything to help you with any of these points. Rather, it locked you into a rather small (when compared to Unicode) set of codepoints, and you typically only could do one setting per document. So, no writing about Cuneiform or Hieroglyphs in Japanese. Today we can intermingle LTR and RTL scripts, and you can freely mix hundreds of scripts in a single document and a single encoding. None of that was possible in the good old days.

lmm 3y ago
> Nobody wants to go back to a world with hundreds of relevant and thousands of legacy encodings. Japan does (assuming agreeing a unicode-like encoding that actually worked is off the table), for the simple reason that practical support for Japanese in "international" applications has regressed since those days. > Obviously much better when nobody can read anything than forcing some people to read texts where there's the occasional odd ('non-native' if you exaggerate a bit) font style choice. https://wiki.c2.com/?AlmostCorrect https://wiki.c2.com/?AlmostCorrect . Failing obviously and universally is better than failing subtly in edge cases. > What you can do today is (taking HTML+CSS as the obvious choice) is proper font choice, `@font-face` declarations, proper downloadable fonts, proper configuration of OpenType font feature flags, if necessary on the basis of Unicode code point ranges. (There's also Unicode variation selectors, bit I've never worked with those so can't say how well those work.) You can do a lot in HTML because HTML has a lot of specific support - something that would likely also be true in a world without unicode. (In the unicode world we have only one encoding per HTML file, but there's no reason a file format couldn't support multiple encodings in the same way as supporting multiple language spans etc.). > It's not like the pre-Unicode chaos of mutually incompatible encodings did anything to help you with any of these points. Rather, it locked you into a rather small (when compared to Unicode) set of codepoints, and you typically only could do one setting per document. The old system made the easy things easy. In those days effectively any program that could display French or Swedish documents could also display Japanese documents. The unicode world makes some things that were previously hard easier, such as mixing languages (as long as the lanugages you want to mix aren't Japanese), but your file format has to go beyond "plain" text for the most basic functionality of properly displaying monolingual Japanese documents. In the old days if you didn't think about languages at all your program would be broken in every non-English language. Now it's broken only in Japan. That's an improvement for most places, but it's a huge regression in Japan, because previously most programs made some effort to deal with internationalisation issues and now they don't, they just say "it's unicode so we don't have to think about it".
- DemocracyFTW2 3y ago
  I think almost everything you said relies on seeing the old ways through rosy glasses and massively underestimating the daily problems that the multitude (technically, gazillions) of encoding standards and standards on how to encode encoding choices brought to the many people who already worked regularly with computers in the early 90s and before that. I'll just pick a single point I disagree with, and then comment on the one point that to me has true merit. So you write, You can do a lot in HTML because HTML has a lot of specific support - something that would likely also be true in a world without unicode. (In the unicode world we have only one encoding per HTML file, but there's no reason a file format couldn't support multiple encodings in the same way as supporting multiple language spans etc.). Basically you reject a format—HTML+Unicode—that has seen tremendous global acceptance, that has brought us 'rich text' (to quote Microsoft), near-complete universal language support for hundreds of previously entirely unencoded writing systems, ability to display ruby text (i.e. small annotations, parallel version of text; btw. not limited to Japanese at all) and want to replace that with 'a file format [that] support[s] multiple encodings'. I agree that it would sometimes be nice (i.e. nice-to-have) inline encoding switches in HTML. You could totally write a JS method to do exactly that, right now, so go ahead! But to replace HTML+Unicode with 'some unspecified file formats' that 'could have' multiple encodings in a single file? Doesn't sound like I want to go down that particular cul de sac. OK now on to that other point: you write your file format has to go beyond "plain" text for the most basic functionality of properly displaying [some features for some linguistic situations], if I may be so bold and generalize to avoid that fixation on "boo it's Japanese again and only Japanese that has been put at a disadvantage". So where I agree with you is that historically the Unicode consortium was too reluctant to acknowledge responsibility for some features of written text that should be expressible in the very text, not in a 'side channel' (e.g. in HTML tags). This would IMO include markers for start and end of stretches of a language, so you could write "good day, {lang=fr}Messieur{/lang}" where the curlies symbolize where Unicode control character sequences (not unlike those already used for flags) should appear. There's more stuff like that like e.g. indicating position of text relative to other text; we sort of have it in the form of CJK Ideographic Description Characters and Control Character for Ancient Egyptian, but nothing for the general purpose.
  - 3y ago

lmm 3y ago

> Nobody wants to go back to a world with hundreds of relevant and thousands of legacy encodings. Japan does (assuming agreeing a unicode-like encoding that actually worked is off the table), for the simple reason that practical support for Japanese in "international" applications has regressed since those days. > Obviously much better when nobody can read anything than forcing some people to read texts where there's the occasional odd ('non-native' if you exaggerate a bit) font style choice. https://wiki.c2.com/?AlmostCorrect https://wiki.c2.com/?AlmostCorrect . Failing obviously and universally is better than failing subtly in edge cases. > What you can do today is (taking HTML+CSS as the obvious choice) is proper font choice, `@font-face` declarations, proper downloadable fonts, proper configuration of OpenType font feature flags, if necessary on the basis of Unicode code point ranges. (There's also Unicode variation selectors, bit I've never worked with those so can't say how well those work.) You can do a lot in HTML because HTML has a lot of specific support - something that would likely also be true in a world without unicode. (In the unicode world we have only one encoding per HTML file, but there's no reason a file format couldn't support multiple encodings in the same way as supporting multiple language spans etc.). > It's not like the pre-Unicode chaos of mutually incompatible encodings did anything to help you with any of these points. Rather, it locked you into a rather small (when compared to Unicode) set of codepoints, and you typically only could do one setting per document. The old system made the easy things easy. In those days effectively any program that could display French or Swedish documents could also display Japanese documents. The unicode world makes some things that were previously hard easier, such as mixing languages (as long as the lanugages you want to mix aren't Japanese), but your file format has to go beyond "plain" text for the most basic functionality of properly displaying monolingual Japanese documents. In the old days if you didn't think about languages at all your program would be broken in every non-English language. Now it's broken only in Japan. That's an improvement for most places, but it's a huge regression in Japan, because previously most programs made some effort to deal with internationalisation issues and now they don't, they just say "it's unicode so we don't have to think about it".

DemocracyFTW2 3y ago
I think almost everything you said relies on seeing the old ways through rosy glasses and massively underestimating the daily problems that the multitude (technically, gazillions) of encoding standards and standards on how to encode encoding choices brought to the many people who already worked regularly with computers in the early 90s and before that. I'll just pick a single point I disagree with, and then comment on the one point that to me has true merit. So you write, You can do a lot in HTML because HTML has a lot of specific support - something that would likely also be true in a world without unicode. (In the unicode world we have only one encoding per HTML file, but there's no reason a file format couldn't support multiple encodings in the same way as supporting multiple language spans etc.). Basically you reject a format—HTML+Unicode—that has seen tremendous global acceptance, that has brought us 'rich text' (to quote Microsoft), near-complete universal language support for hundreds of previously entirely unencoded writing systems, ability to display ruby text (i.e. small annotations, parallel version of text; btw. not limited to Japanese at all) and want to replace that with 'a file format [that] support[s] multiple encodings'. I agree that it would sometimes be nice (i.e. nice-to-have) inline encoding switches in HTML. You could totally write a JS method to do exactly that, right now, so go ahead! But to replace HTML+Unicode with 'some unspecified file formats' that 'could have' multiple encodings in a single file? Doesn't sound like I want to go down that particular cul de sac. OK now on to that other point: you write your file format has to go beyond "plain" text for the most basic functionality of properly displaying [some features for some linguistic situations], if I may be so bold and generalize to avoid that fixation on "boo it's Japanese again and only Japanese that has been put at a disadvantage". So where I agree with you is that historically the Unicode consortium was too reluctant to acknowledge responsibility for some features of written text that should be expressible in the very text, not in a 'side channel' (e.g. in HTML tags). This would IMO include markers for start and end of stretches of a language, so you could write "good day, {lang=fr}Messieur{/lang}" where the curlies symbolize where Unicode control character sequences (not unlike those already used for flags) should appear. There's more stuff like that like e.g. indicating position of text relative to other text; we sort of have it in the form of CJK Ideographic Description Characters and Control Character for Ancient Egyptian, but nothing for the general purpose.
- 3y ago