3 ms·

Well, I must nitpick: Shift JIS is actually just one of those out-of-band vaporware methods. Sure, it's better supported because of legacy, but new code that do

Well, I must nitpick: Shift JIS is actually just one of those out-of-band vaporware methods. Sure, it's better supported because of legacy, but new code that doesn't care about handling some lang metadata is not going to care about Shift JIS either. Of course, there is no correct solution (people in other comments seem to believe it exists a bit too quickly). Dynamic HTML page can load pieces of text written by people all across the world, it can't rely on any “main” language. Those people can't be automatically classified (based on some browser settings or location). Then there are always people who know both Chinese and Japanese (while using some other system locale, to make things more complex). It is wrong to assume that they should not be able to use Chinese forms and Japanese forms at once, even inside a single paragraph or phrase. I wonder why Unicode has not simply introduced some “combining language form character” to make user choice stick. After all, there's a whole subsystem for emoji modification, and those things were once “weird Japanese texting customs”. As for complexity of handling Unicode text, it asymptotically reaches its maximum anyway. Wait a second, there are “variation selectors” and some “Ideographic Variation Database”. Is it the solution? Can IMEs and converters simply stamp each variable character with invisible mark based on current input language? I suppose there's some catch…

lmm 2y ago
> Shift JIS is actually just one of those out-of-band vaporware methods. It's the opposite of vaporware; there's a whole bunch of well-known software that handles encodings correctly. > new code that doesn't care about handling some lang metadata is not going to care about Shift JIS either. If you support encodings the way almost every programming language tells you too, you'll handle Shift-JIS just fine. If you use the "legacy" encodings approach and test even one non-English language you'll handle Japanese fine. > Can IMEs and converters simply stamp each variable character with invisible mark based on current input language? I suppose there's some catch… They're officially deprecated, IMEs and converters don't use them, naïve search implementations (which is to say most search implementations, because people do not test for an obscure edge case) break, ....
- ogurechny 2y ago
  But there is no “naive” implementation of Unicode. It won't handle emoji, normalization, and thousands other things. People use The Library anyway. Adding another mutagen to it should be no different from the rest. As for official stance, maybe it's time for some group to agree on certain non-conflicting sequences, and implement them in some library/stack. A particularly evil solution would choose arbitrary existing combining characters that can't be declared “illegal” retroactively. By the way, is it possible to make proper quotes from other languages as described above when using Shift JIS?
  - lmm 2y ago
    > But there is no “naive” implementation of Unicode. I assure you there are thousands if not millions of apps that implement search the naive way, by comparing bytes. > It won't handle emoji, normalization, and thousands other things. Yeah. But it works well enough for Americans, so the maintainers don't care enough to fix it. > As for official stance, maybe it's time for some group to agree on certain non-conflicting sequences, and implement them in some library/stack. And then what? What do you do with documents in Japanese that have already been converted without using your new codepoints? What do you do with programs that ignore your new standard? What do you do about search being broken in most apps with no prospect of fixing it? There are already some PUA codepoints for a proper implementation of Japanese, but most app authors don't even understand the problem, never mind being willing to support something that's "non-standard". Asking them to support traditional non-unicode encodings, which is something that's at least relatively well-known and standardised, and something they can test without knowing Japanese, is much easier. > By the way, is it possible to make proper quotes from other languages as described above when using Shift JIS? No, not for arbitrary languages. If you want to mix arbitrary languages, you need some structure that's a rope of strings with an encoding per segment. But that's exactly the same thing that the unicode advocates claim is easy ("just have a sequence of spans with different lang tags"), and at least if you use traditional encodings then you don't have to do any such fancy business for the much more common case of a file that's entirely in Japanese (perhaps with some ASCII keywords).

lmm 2y ago

> Shift JIS is actually just one of those out-of-band vaporware methods. It's the opposite of vaporware; there's a whole bunch of well-known software that handles encodings correctly. > new code that doesn't care about handling some lang metadata is not going to care about Shift JIS either. If you support encodings the way almost every programming language tells you too, you'll handle Shift-JIS just fine. If you use the "legacy" encodings approach and test even one non-English language you'll handle Japanese fine. > Can IMEs and converters simply stamp each variable character with invisible mark based on current input language? I suppose there's some catch… They're officially deprecated, IMEs and converters don't use them, naïve search implementations (which is to say most search implementations, because people do not test for an obscure edge case) break, ....

ogurechny 2y ago
But there is no “naive” implementation of Unicode. It won't handle emoji, normalization, and thousands other things. People use The Library anyway. Adding another mutagen to it should be no different from the rest. As for official stance, maybe it's time for some group to agree on certain non-conflicting sequences, and implement them in some library/stack. A particularly evil solution would choose arbitrary existing combining characters that can't be declared “illegal” retroactively. By the way, is it possible to make proper quotes from other languages as described above when using Shift JIS?
- lmm 2y ago
  > But there is no “naive” implementation of Unicode. I assure you there are thousands if not millions of apps that implement search the naive way, by comparing bytes. > It won't handle emoji, normalization, and thousands other things. Yeah. But it works well enough for Americans, so the maintainers don't care enough to fix it. > As for official stance, maybe it's time for some group to agree on certain non-conflicting sequences, and implement them in some library/stack. And then what? What do you do with documents in Japanese that have already been converted without using your new codepoints? What do you do with programs that ignore your new standard? What do you do about search being broken in most apps with no prospect of fixing it? There are already some PUA codepoints for a proper implementation of Japanese, but most app authors don't even understand the problem, never mind being willing to support something that's "non-standard". Asking them to support traditional non-unicode encodings, which is something that's at least relatively well-known and standardised, and something they can test without knowing Japanese, is much easier. > By the way, is it possible to make proper quotes from other languages as described above when using Shift JIS? No, not for arbitrary languages. If you want to mix arbitrary languages, you need some structure that's a rope of strings with an encoding per segment. But that's exactly the same thing that the unicode advocates claim is easy ("just have a sequence of spans with different lang tags"), and at least if you use traditional encodings then you don't have to do any such fancy business for the much more common case of a file that's entirely in Japanese (perhaps with some ASCII keywords).