6 ms·

Hopefully this will be a big step forward in solving the problem of Han unification: http://en.wikipedia.org/wiki/Han_unification http://en.wikipedia.org/wiki/H

Hopefully this will be a big step forward in solving the problem of Han unification: http://en.wikipedia.org/wiki/Han_unification http://en.wikipedia.org/wiki/Han_unification It's infuriating how many Japanese sites still don't use Unicode, purportedly because of this issue (though I suspect that it's just another example of Japan lagging when it comes to web/computer tech).

est 12y ago
There's some CJK sample on Adobe Typekit blog http://blog.typekit.com/2014/07/15/introducing-source-han-sans/ http://blog.typekit.com/2014/07/15/introducing-source-han-sa...
rurounijones 12y ago
The amount of Japanese systems that are SJIS only is staggering, basically all traditional IT (banks etc) uses it and does not support, nor will ever support, unicode. This, naturally, has an impact beyond those systems' borders.
patio11 12y ago
Every discussion we ever had about going to Unicode: "Is it bidirectional compatible with SJIS?" "No." "OK, so when it breaks, what characters does it break on?" "People's names, mostly." "... And why is this being considered?" "It is very, very convenient for white people. Almost all of their stuff works out of the box."
- yongjik 12y ago
  I wouldn't say it's convenient for white people. Being able to write Japanese on all the non-Japanese people's stuff (basically, most websites and open-source softwares) should be mostly useful to the Japanese.
yuubi 12y ago
Oddly, the following more modest proposal hasn't gotten much traction: characters that share history but have divergent graphical representations in the various dialects of alphabetic script shall share codepoints, and a mechanism beyond the scope of Unicode (like lang attributes or plain guesswork) shall be used to decide whether a given codepoint means L or ᴫ or Λ or whatever.
- fenomas 12y ago
  Actually I think that's roughly how things work today. It's not my area, but here's how this was explained to me recently by a colleague (the lead font guy at Adobe Japan): There are two ways of dealing with glyphs that share code points. The first is TTC (truetype collection) fonts. A TTC is basically one set of glyphs with several sets of mappings (i.e. which code point maps to which glyph). When you install it, assuming your computer groks ttc, your system shows you a separate font for each mapping. Taking for example Source Han Sans, which adobe just released - if you go to the download page[0] and get the complete version (the "OTC" one), you get a bunch of files like "SourceHanSans-Bold.ttc". If you install one of them you'll see four new fonts: "Source Han Sans J", K, SC, and TC. Then when you use the font, depending on which font name you used the system will change which mapping it applies to the combined set of glyphs. (Hence the choice of font name is the selection mechanism you described.) The second way is that TrueType fonts have a way to build locale settings into the font. I'm less clear on the details here but apparently it's similar to TTC behind the scenes, except that the mappings are associated with locales - so in an app that supports TT locales, even if you select "Foo J" as your font, when the locale was simplified Chinese you'd get the SC glyph. Of course now the selection mechanism is whether the application knows what locale the content is. (And also whether it supports the mechanism - I don't know how widespread this is.) Either way though, in principle you get different glyphs for the same code point, depending on context. Or anyway that's the understanding I took away as a font layperson - happy to be corrected. [0] http://sourceforge.net/projects/source-han-sans.adobe/files/ http://sourceforge.net/projects/source-han-sans.adobe/files/
  - yuubi 12y ago
    The modest proposal was to extend this approach with all its complications to western alphabetic scripts where possible. Of course nobody wants to do that for obvious reasons, which also apply to the various languages that use Han-derived characters. The extra mechanisms required to work around unified Han remind me of the pre-unicode days when you needed to know what language a text was in to render it.
- Osmium 12y ago
  Surely there are enough unicode code points for this not be a problem? Can you use the historical character + combining mark (which shows which 'newer' version of the character to use), where the combining mark is ignored if the computer doesn't understand it, and only then it falls back onto guesswork/lang attributes. I don't know if that's a decent solution, but just guesswork doesn't sound like a good idea, because there are bound to be edge cases where it wouldn't work, and then we're back where we started...
unsignedint 12y ago
Part of the problem here is that Japanese Windows using SJIS at its front end, perhaps for "legacy" compatibility issue... (Backend, like filesystem on modern Windows is Unicode; it was apparent when adding files containing Japanese characters into Git; it would work if I'm retrieving the file from another Windows machine, but horribly corrupted when moving to other platforms.) Fortunately, Git guys added fix to convert it into Unicode internally.
ejr 12y ago
It may not be so infuriating when you consider how important names are to Japanese people. Breaking your name is unacceptable and breaking it because of a technical convenience chosen during development would be deeply offensive on top of being unacceptable. Until Unicode stops breaking people's names, it will continue to be the one standard for Japanese systems on and offline. Even when(if?) it stops breaking Japanese names, it will take a very, very long time to roll over existing systems and that's precluding unforeseen problems during the conversion. We should stop before we take the "not following standard" = "broken" ideology. Especially when we consider whom the standard serves best. Edit: By "it will continue to be the one standard for Japanese" in the 2nd paragraph, I meant ShiftJIS not Unicode. That looked a bit unclear.
- deleted 12y ago
  [deleted]
- fenomas 12y ago
  What do you mean by saying that Unicode breaks people's names? The problem isn't that there's anything wrong with Unicode, the problem is that it's not possible to unambiguously convert text from SJIS to unicode and back again, because SJIS has some duplicate mappings for historical reasons (compatibility with different pre-existing encodings as I recall). The same would presumably be true of converting SJIS to any other encoding that didn't have the same duplicated code points.
  - phkahler 12y ago
    >> because SJIS has some duplicate mappings for historical reasons I think that's the problem. Sounds like unicode has some duplicate mapping. Never mind the "historical reasons", it's broken.
  - jessaustin 12y ago
    Why does Unicode get to decide what "duplicate" means in this context? If Mr. Smythe is addressed as "Mr. Smith" in a letter from his bank, is he going to be mollified by the explanation that Unicode considers his preferred representation to be a duplicate?
gioele 12y ago
Please understand that Han unification is _the_ problem. It is clean that Unicode needs to realize that the Han unification is wrong and accepts what the native writers of those languages think about their scripts. To make the problem more understandable to the people that are used to alphabetic scripts, suppose that tomorrow an Asian committee starts creating Uniword, a repertoire that maps complete words to numerical IDs. At a certain point they get to "colour". Uniword committee: Well, that word shares meaning and origin with the other word "color", for which we have already a codepoint, so we will encode them under the same codepoint. GB, Australia and Canada: Ehi! No! To us those are different words; especially, we do not want Mr. Colours to appear as Mr. Color. Uniword commitee: No problem, just add some out-of-band information like "nationality" or "<span lang='en-GB'>" "colour"-people: that will not work, there are so many cases in which this can go wrong. Whenever I copy a field from a DB I also have to extract this extra information? Uniword: yes, that is the problem? C'mon! "colour"-people: but do you need to do that in your applications? Uniword: no, we have one code for every single word in our languages, including codes for very old languages that exist only in two palimpsests. "colour"-people: and why cannot we have the same level of granularity? Uniword: because you have too many words!!! And we started we had only 100k available integers. "colour"-people: and now? Uniword: now we have 2^32. But, yeah, that is not the point; just do how we suggest. This dialog is getting to long. "colour"-people: "dialogue", please.
- ksec 12y ago
  Thanks, I replied a bit to early without reading. I think you capture the problem nicely.
- haberman 12y ago
  This is a great summary, thanks for that. I'd never had this explained in a way I could personally relate to. I remember being concerned about Han unification around the time Ruby 1.9 was released, since this seemed to be one of Ruby's major reasons for being encoding-independent instead of standardizing on Unicode. But I hadn't heard about this issue in a while, except to hear occasionally someone say it's not a problem (maybe it was a Chinese person instead of a Japanese person -- the Wikipedia page says that the Chinese aren't as concerned about Han unification since Traditional Chinese didn't get unified with Simplified Chinese).
glandium 12y ago
Another problem is that while most japanese characters take 2 bytes in SJIS, they take 3 in UTF-8.
ksec 12y ago
I can, partially understand why the Japanese refuse to support Unicode. And while most just adopt unicode simply because of its convenient, it doesn't actually solve the problem behind the Hans Characters in different form and glyph. Over the years, i am starting to think Han Unification is western ways of hacking the CJK Hans problem rather then actually solving it.
- fenomas 12y ago
  I wouldn't say Japanese refuse to support unicode. I'd say that legacy encodings like EUC-JP/JIS/SJIS are still used because it's hard/unfeasible to convert systems and data that were built for earlier encodings to anything newer. But it's not like they offer some particular technical advantage over Unicode - indeed they only reason they don't suffer from CJK issues is that they have no support for C or K. ;) But speaking as a front-end webdev guy, it's been a looong time since I came in contact with any encoding here besides utf-8.
euske 12y ago
Eh, most Japanese people don't know/care about the problem of Han unification. It's mostly because of the legacy data. The thing is that ASCII and UTF-8 is kinda compatible with minor annoyances, while Shift JIS and UTF-8 are entirely different. People don't want to convert a trove of documents to another encoding which might not be supported yet in some apps. Slow software upgrade is another reason. As someone else pointed out, the default encoding of Windows is still Shift JIS, which is totally understandable for compatibility sake. Edit: Besides, a TTF font doesn't have to always use Unicode internally. It supports an arbitrary mapping from bytes (could be in UTF-8 or SJIS) to a glyph number. People who really care about the looks (i.e. printing) have been using a charset for each specific language, such as Adobe-Japan1, which is different from both UTF-8 or Shift JIS.

est 12y ago

There's some CJK sample on Adobe Typekit blog http://blog.typekit.com/2014/07/15/introducing-source-han-sans/ http://blog.typekit.com/2014/07/15/introducing-source-han-sa...

rurounijones 12y ago

The amount of Japanese systems that are SJIS only is staggering, basically all traditional IT (banks etc) uses it and does not support, nor will ever support, unicode. This, naturally, has an impact beyond those systems' borders.

patio11 12y ago

Every discussion we ever had about going to Unicode: "Is it bidirectional compatible with SJIS?" "No." "OK, so when it breaks, what characters does it break on?" "People's names, mostly." "... And why is this being considered?" "It is very, very convenient for white people. Almost all of their stuff works out of the box."

yongjik 12y ago
I wouldn't say it's convenient for white people. Being able to write Japanese on all the non-Japanese people's stuff (basically, most websites and open-source softwares) should be mostly useful to the Japanese.

yuubi 12y ago

Oddly, the following more modest proposal hasn't gotten much traction: characters that share history but have divergent graphical representations in the various dialects of alphabetic script shall share codepoints, and a mechanism beyond the scope of Unicode (like lang attributes or plain guesswork) shall be used to decide whether a given codepoint means L or ᴫ or Λ or whatever.

fenomas 12y ago
Actually I think that's roughly how things work today. It's not my area, but here's how this was explained to me recently by a colleague (the lead font guy at Adobe Japan): There are two ways of dealing with glyphs that share code points. The first is TTC (truetype collection) fonts. A TTC is basically one set of glyphs with several sets of mappings (i.e. which code point maps to which glyph). When you install it, assuming your computer groks ttc, your system shows you a separate font for each mapping. Taking for example Source Han Sans, which adobe just released - if you go to the download page[0] and get the complete version (the "OTC" one), you get a bunch of files like "SourceHanSans-Bold.ttc". If you install one of them you'll see four new fonts: "Source Han Sans J", K, SC, and TC. Then when you use the font, depending on which font name you used the system will change which mapping it applies to the combined set of glyphs. (Hence the choice of font name is the selection mechanism you described.) The second way is that TrueType fonts have a way to build locale settings into the font. I'm less clear on the details here but apparently it's similar to TTC behind the scenes, except that the mappings are associated with locales - so in an app that supports TT locales, even if you select "Foo J" as your font, when the locale was simplified Chinese you'd get the SC glyph. Of course now the selection mechanism is whether the application knows what locale the content is. (And also whether it supports the mechanism - I don't know how widespread this is.) Either way though, in principle you get different glyphs for the same code point, depending on context. Or anyway that's the understanding I took away as a font layperson - happy to be corrected. [0] http://sourceforge.net/projects/source-han-sans.adobe/files/ http://sourceforge.net/projects/source-han-sans.adobe/files/
- yuubi 12y ago
  The modest proposal was to extend this approach with all its complications to western alphabetic scripts where possible. Of course nobody wants to do that for obvious reasons, which also apply to the various languages that use Han-derived characters. The extra mechanisms required to work around unified Han remind me of the pre-unicode days when you needed to know what language a text was in to render it.
Osmium 12y ago
Surely there are enough unicode code points for this not be a problem? Can you use the historical character + combining mark (which shows which 'newer' version of the character to use), where the combining mark is ignored if the computer doesn't understand it, and only then it falls back onto guesswork/lang attributes. I don't know if that's a decent solution, but just guesswork doesn't sound like a good idea, because there are bound to be edge cases where it wouldn't work, and then we're back where we started...

unsignedint 12y ago

Part of the problem here is that Japanese Windows using SJIS at its front end, perhaps for "legacy" compatibility issue... (Backend, like filesystem on modern Windows is Unicode; it was apparent when adding files containing Japanese characters into Git; it would work if I'm retrieving the file from another Windows machine, but horribly corrupted when moving to other platforms.) Fortunately, Git guys added fix to convert it into Unicode internally.

ejr 12y ago

It may not be so infuriating when you consider how important names are to Japanese people. Breaking your name is unacceptable and breaking it because of a technical convenience chosen during development would be deeply offensive on top of being unacceptable. Until Unicode stops breaking people's names, it will continue to be the one standard for Japanese systems on and offline. Even when(if?) it stops breaking Japanese names, it will take a very, very long time to roll over existing systems and that's precluding unforeseen problems during the conversion. We should stop before we take the "not following standard" = "broken" ideology. Especially when we consider whom the standard serves best. Edit: By "it will continue to be the one standard for Japanese" in the 2nd paragraph, I meant ShiftJIS not Unicode. That looked a bit unclear.

deleted 12y ago
[deleted]
fenomas 12y ago
What do you mean by saying that Unicode breaks people's names? The problem isn't that there's anything wrong with Unicode, the problem is that it's not possible to unambiguously convert text from SJIS to unicode and back again, because SJIS has some duplicate mappings for historical reasons (compatibility with different pre-existing encodings as I recall). The same would presumably be true of converting SJIS to any other encoding that didn't have the same duplicated code points.
- phkahler 12y ago
  >> because SJIS has some duplicate mappings for historical reasons I think that's the problem. Sounds like unicode has some duplicate mapping. Never mind the "historical reasons", it's broken.
- jessaustin 12y ago
  Why does Unicode get to decide what "duplicate" means in this context? If Mr. Smythe is addressed as "Mr. Smith" in a letter from his bank, is he going to be mollified by the explanation that Unicode considers his preferred representation to be a duplicate?

gioele 12y ago

Please understand that Han unification is _the_ problem. It is clean that Unicode needs to realize that the Han unification is wrong and accepts what the native writers of those languages think about their scripts. To make the problem more understandable to the people that are used to alphabetic scripts, suppose that tomorrow an Asian committee starts creating Uniword, a repertoire that maps complete words to numerical IDs. At a certain point they get to "colour". Uniword committee: Well, that word shares meaning and origin with the other word "color", for which we have already a codepoint, so we will encode them under the same codepoint. GB, Australia and Canada: Ehi! No! To us those are different words; especially, we do not want Mr. Colours to appear as Mr. Color. Uniword commitee: No problem, just add some out-of-band information like "nationality" or "<span lang='en-GB'>" "colour"-people: that will not work, there are so many cases in which this can go wrong. Whenever I copy a field from a DB I also have to extract this extra information? Uniword: yes, that is the problem? C'mon! "colour"-people: but do you need to do that in your applications? Uniword: no, we have one code for every single word in our languages, including codes for very old languages that exist only in two palimpsests. "colour"-people: and why cannot we have the same level of granularity? Uniword: because you have too many words!!! And we started we had only 100k available integers. "colour"-people: and now? Uniword: now we have 2^32. But, yeah, that is not the point; just do how we suggest. This dialog is getting to long. "colour"-people: "dialogue", please.

ksec 12y ago
Thanks, I replied a bit to early without reading. I think you capture the problem nicely.
haberman 12y ago
This is a great summary, thanks for that. I'd never had this explained in a way I could personally relate to. I remember being concerned about Han unification around the time Ruby 1.9 was released, since this seemed to be one of Ruby's major reasons for being encoding-independent instead of standardizing on Unicode. But I hadn't heard about this issue in a while, except to hear occasionally someone say it's not a problem (maybe it was a Chinese person instead of a Japanese person -- the Wikipedia page says that the Chinese aren't as concerned about Han unification since Traditional Chinese didn't get unified with Simplified Chinese).

glandium 12y ago

Another problem is that while most japanese characters take 2 bytes in SJIS, they take 3 in UTF-8.

ksec 12y ago

I can, partially understand why the Japanese refuse to support Unicode. And while most just adopt unicode simply because of its convenient, it doesn't actually solve the problem behind the Hans Characters in different form and glyph. Over the years, i am starting to think Han Unification is western ways of hacking the CJK Hans problem rather then actually solving it.

fenomas 12y ago
I wouldn't say Japanese refuse to support unicode. I'd say that legacy encodings like EUC-JP/JIS/SJIS are still used because it's hard/unfeasible to convert systems and data that were built for earlier encodings to anything newer. But it's not like they offer some particular technical advantage over Unicode - indeed they only reason they don't suffer from CJK issues is that they have no support for C or K. ;) But speaking as a front-end webdev guy, it's been a looong time since I came in contact with any encoding here besides utf-8.

euske 12y ago

Eh, most Japanese people don't know/care about the problem of Han unification. It's mostly because of the legacy data. The thing is that ASCII and UTF-8 is kinda compatible with minor annoyances, while Shift JIS and UTF-8 are entirely different. People don't want to convert a trove of documents to another encoding which might not be supported yet in some apps. Slow software upgrade is another reason. As someone else pointed out, the default encoding of Windows is still Shift JIS, which is totally understandable for compatibility sake. Edit: Besides, a TTF font doesn't have to always use Unicode internally. It supports an arbitrary mapping from bytes (could be in UTF-8 or SJIS) to a glyph number. People who really care about the looks (i.e. printing) have been using a charset for each specific language, such as Adobe-Japan1, which is different from both UTF-8 or Shift JIS.