4 ms·

> unless and until unicode stops screwing over Japanese Can you say more about this? Are there still reasons to prefer shiftjis over unicode for japanese chara

> unless and until unicode stops screwing over Japanese Can you say more about this? Are there still reasons to prefer shiftjis over unicode for japanese characters?

rtpg 4y ago
They might be alluding to issues from CJK unification. But I don’t believe that many practitioners working with Japanese text consider these concerns more important than the value that Unicode brings to the table. I think that CJK unification leads to awkwardness where some fonts will basically render some kanji the “Chinese way” or the “Japanese way”, so you can’t really have a good font that covers both at once. There’s a good aesthetic argument about not using the same font for both languages, but Unicode outright precludes it I think
- lmm 4y ago
  Virtually every Japanese organisation disagrees, you only have to look at a Japanese website to know as much (or are those somehow not "practitioners working with Japanese text"?). Most will use a non-Unicode encoding and/or resort to rendering important text as images, since it's impossible to rely on Unicode text rendering well.
  - josephg 4y ago
    I wonder if this can ever be fixed. If basically every Japanese organization disagrees with the Unicode consortium to the point that its undermining the use of unicode in entire countries, it might be worth unwinding CJK normalization.
    - Freak_NL 4y ago
      This isn't the case obviously. Japanese organisations are involved in the Unicode process (something evidenced in the input for the several CJK Unified Ideograph Extensions). Thank to Unicode, many historical characters can be represented in digital text without resorting to placeholders or hacks. Anecdotally, I've visited a group enthusiastically exploring the possibilities of the then-new CJK Unified Ideograph Extension B in Tokyo over a decade ago. The CJK issue mentioned here was a bigger problem then than now (most developers know how to handle it properly, e.g. with language declarations in HTML etc.). People will use legacy encodings. That's just a fact of life. Sometimes out of inertia (i.e., it just works, so why bother changing it?), sometimes out of technical limitations of their pipeline, sometimes because of a lack of understanding, rarely out of principle. I'm stuck fixing an issue with some numb skull sending us SAML requests declared as UTF-8 containing some obsolete encoding instead, and this is for Dutch text! This just happens. There is also nothing wrong with text in Shift-JIS, if that encoding encompasses all characters used.
  - squaresmile 4y ago
    From here [0], 94.1% of .jp websites use UTF-8 and 6.5% use Shift-JIS. You can find a list of popular sites still using Shift-JIS here [1]. If you change your PC or your phone locale to Japanese and access mostly Japanese website, the font display on most major OS works well. It's probably fair to assume that's a very common setup for Japanese users, and probably for Chinese and Korean users as well. I think the CJK "problem" is more apparent for international users who don't have the locale setup or dealing with multiple languages plain text at the same time. [0] https://w3techs.com/technologies/segmentation/tld-jp-/character_encoding https://w3techs.com/technologies/segmentation/tld-jp-/charac... [1] https://w3techs.com/technologies/details/en-shiftjis https://w3techs.com/technologies/details/en-shiftjis
    - lmm 4y ago
      Your own link shows that both Shift-JIS and EUC-JP are on an upward trend, which isn't what we'd expect if Unicode was succeeding. I can't see it listing examples of .jp sites using UTF-8; would be interesting to see how many of them are relying on text-as-images.
    - fomine3 4y ago
      Yeah what lmm says is not what happend/thought in Japan. Japanese still uses Shift_JIS because they love (/s) legacy software. Using Shift_JIS isn't a practical solution.
  - rtpg 4y ago
    Random sample of some websites: - Rakuten Bank -> Shift JIS - NHK, the Ministry of Health and Labor Welfare, yahoo japan -> UTF-8 I don't really really know, but this random yahoo answers [0] page seems to agree with me that really you're gonna use UTF-8 for new stuff in general One point that answer brings up that I hadn't thought of: older flipphones would have good shift-JIS support but not necessarily good unicode support My guess is that "text as images" might partly be that, partly what you are saying in a legacy sense, and also partly for the same reason old sites in the US would do that: to make something look the way you want and you already have a photoshop render [0]: https://detail.chiebukuro.yahoo.co.jp/qa/question_detail/q1487980789 https://detail.chiebukuro.yahoo.co.jp/qa/question_detail/q14...
  - GoblinSlayer 4y ago
    Ironically my company sends letters fully rendered as raster images, ugh.
lmm 4y ago
> Can you say more about this? Are there still reasons to prefer shiftjis over unicode for japanese characters? Yes. Unicode uses codepoints that are primarily for Chinese characters to represent Japanese characters that they consider equivalent, even when those Japanese characters have different appearences; as a result, Japanese text in unicode looks bad (readable, but ugly) unless displayed in a specifically Japanese font (in which case you'd have the converse problem of Chinese text looking wrong). The unicode consortium suggests various vaporware approaches to combat this, but the thing that actually works is keeping Japanese text in Japanese encodings and Chinese text in Chinese encodings. (Of course this means that you need to be able to display text from multiple encodings in the same page if you want to display both languages on the same page, but all of the unicode consortium vaporware fixes require you to build something equivalently complex, and you wouldn't even be able to test it by using strings from two western encodings since it would be specific to Japanese and Chinese)
- amanwithnoplan 4y ago
  What's wrong with just wrapping UTF-8 text in lang spans? Browsers and Electron will just do the right thing, no?
  - lmm 4y ago
    Even assuming that works, getting it to make its whole way through your whole tech stack is no easier (and more HTML-specific) than having spans with their own encoding.
    - astrange 4y ago
      Since when can HTML pages have multiple encodings at the same time?
      - TheDong 4y ago
        This isn't about multiple encodings, this is about the one encoding, UTF-8, being able to represent multiple languages. Which is mostly can, except for han characters. In my reply here, I can type in english and russian at once. Привет, мир. Yet, if I try to type chinese on one line, and japanese on the next, I cannot do it. Hacker news does not let me enter "lang" tags, so I can only type either the chinese or the japanese variant of a kanji.

> unless and until unicode stops screwing over Japanese Can you say more about this? Are there still reasons to prefer shiftjis over unicode for japanese characters?

rtpg 4y ago

They might be alluding to issues from CJK unification. But I don’t believe that many practitioners working with Japanese text consider these concerns more important than the value that Unicode brings to the table. I think that CJK unification leads to awkwardness where some fonts will basically render some kanji the “Chinese way” or the “Japanese way”, so you can’t really have a good font that covers both at once. There’s a good aesthetic argument about not using the same font for both languages, but Unicode outright precludes it I think

lmm 4y ago
Virtually every Japanese organisation disagrees, you only have to look at a Japanese website to know as much (or are those somehow not "practitioners working with Japanese text"?). Most will use a non-Unicode encoding and/or resort to rendering important text as images, since it's impossible to rely on Unicode text rendering well.
- josephg 4y ago
  I wonder if this can ever be fixed. If basically every Japanese organization disagrees with the Unicode consortium to the point that its undermining the use of unicode in entire countries, it might be worth unwinding CJK normalization.
  - Freak_NL 4y ago
    This isn't the case obviously. Japanese organisations are involved in the Unicode process (something evidenced in the input for the several CJK Unified Ideograph Extensions). Thank to Unicode, many historical characters can be represented in digital text without resorting to placeholders or hacks. Anecdotally, I've visited a group enthusiastically exploring the possibilities of the then-new CJK Unified Ideograph Extension B in Tokyo over a decade ago. The CJK issue mentioned here was a bigger problem then than now (most developers know how to handle it properly, e.g. with language declarations in HTML etc.). People will use legacy encodings. That's just a fact of life. Sometimes out of inertia (i.e., it just works, so why bother changing it?), sometimes out of technical limitations of their pipeline, sometimes because of a lack of understanding, rarely out of principle. I'm stuck fixing an issue with some numb skull sending us SAML requests declared as UTF-8 containing some obsolete encoding instead, and this is for Dutch text! This just happens. There is also nothing wrong with text in Shift-JIS, if that encoding encompasses all characters used.
- squaresmile 4y ago
  From here [0], 94.1% of .jp websites use UTF-8 and 6.5% use Shift-JIS. You can find a list of popular sites still using Shift-JIS here [1]. If you change your PC or your phone locale to Japanese and access mostly Japanese website, the font display on most major OS works well. It's probably fair to assume that's a very common setup for Japanese users, and probably for Chinese and Korean users as well. I think the CJK "problem" is more apparent for international users who don't have the locale setup or dealing with multiple languages plain text at the same time. [0] https://w3techs.com/technologies/segmentation/tld-jp-/character_encoding https://w3techs.com/technologies/segmentation/tld-jp-/charac... [1] https://w3techs.com/technologies/details/en-shiftjis https://w3techs.com/technologies/details/en-shiftjis
  - lmm 4y ago
    Your own link shows that both Shift-JIS and EUC-JP are on an upward trend, which isn't what we'd expect if Unicode was succeeding. I can't see it listing examples of .jp sites using UTF-8; would be interesting to see how many of them are relying on text-as-images.
  - fomine3 4y ago
    Yeah what lmm says is not what happend/thought in Japan. Japanese still uses Shift_JIS because they love (/s) legacy software. Using Shift_JIS isn't a practical solution.
- rtpg 4y ago
  Random sample of some websites: - Rakuten Bank -> Shift JIS - NHK, the Ministry of Health and Labor Welfare, yahoo japan -> UTF-8 I don't really really know, but this random yahoo answers [0] page seems to agree with me that really you're gonna use UTF-8 for new stuff in general One point that answer brings up that I hadn't thought of: older flipphones would have good shift-JIS support but not necessarily good unicode support My guess is that "text as images" might partly be that, partly what you are saying in a legacy sense, and also partly for the same reason old sites in the US would do that: to make something look the way you want and you already have a photoshop render [0]: https://detail.chiebukuro.yahoo.co.jp/qa/question_detail/q1487980789 https://detail.chiebukuro.yahoo.co.jp/qa/question_detail/q14...
- GoblinSlayer 4y ago
  Ironically my company sends letters fully rendered as raster images, ugh.

lmm 4y ago

> Can you say more about this? Are there still reasons to prefer shiftjis over unicode for japanese characters? Yes. Unicode uses codepoints that are primarily for Chinese characters to represent Japanese characters that they consider equivalent, even when those Japanese characters have different appearences; as a result, Japanese text in unicode looks bad (readable, but ugly) unless displayed in a specifically Japanese font (in which case you'd have the converse problem of Chinese text looking wrong). The unicode consortium suggests various vaporware approaches to combat this, but the thing that actually works is keeping Japanese text in Japanese encodings and Chinese text in Chinese encodings. (Of course this means that you need to be able to display text from multiple encodings in the same page if you want to display both languages on the same page, but all of the unicode consortium vaporware fixes require you to build something equivalently complex, and you wouldn't even be able to test it by using strings from two western encodings since it would be specific to Japanese and Chinese)

amanwithnoplan 4y ago
What's wrong with just wrapping UTF-8 text in lang spans? Browsers and Electron will just do the right thing, no?
- lmm 4y ago
  Even assuming that works, getting it to make its whole way through your whole tech stack is no easier (and more HTML-specific) than having spans with their own encoding.
  - astrange 4y ago
    Since when can HTML pages have multiple encodings at the same time?
    - TheDong 4y ago
      This isn't about multiple encodings, this is about the one encoding, UTF-8, being able to represent multiple languages. Which is mostly can, except for han characters. In my reply here, I can type in english and russian at once. Привет, мир. Yet, if I try to type chinese on one line, and japanese on the next, I cannot do it. Hacker news does not let me enter "lang" tags, so I can only type either the chinese or the japanese variant of a kanji.