3 ms·

> Shift JIS is actually just one of those out-of-band vaporware methods. It's the opposite of vaporware; there's a whole bunch of well-known software that hand

by lmm 3y ago

> Shift JIS is actually just one of those out-of-band vaporware methods. It's the opposite of vaporware; there's a whole bunch of well-known software that handles encodings correctly. > new code that doesn't care about handling some lang metadata is not going to care about Shift JIS either. If you support encodings the way almost every programming language tells you too, you'll handle Shift-JIS just fine. If you use the "legacy" encodings approach and test even one non-English language you'll handle Japanese fine. > Can IMEs and converters simply stamp each variable character with invisible mark based on current input language? I suppose there's some catch… They're officially deprecated, IMEs and converters don't use them, naïve search implementations (which is to say most search implementations, because people do not test for an obscure edge case) break, ....

ogurechny 3y ago
But there is no “naive” implementation of Unicode. It won't handle emoji, normalization, and thousands other things. People use The Library anyway. Adding another mutagen to it should be no different from the rest. As for official stance, maybe it's time for some group to agree on certain non-conflicting sequences, and implement them in some library/stack. A particularly evil solution would choose arbitrary existing combining characters that can't be declared “illegal” retroactively. By the way, is it possible to make proper quotes from other languages as described above when using Shift JIS?
- lmm 3y ago
  > But there is no “naive” implementation of Unicode. I assure you there are thousands if not millions of apps that implement search the naive way, by comparing bytes. > It won't handle emoji, normalization, and thousands other things. Yeah. But it works well enough for Americans, so the maintainers don't care enough to fix it. > As for official stance, maybe it's time for some group to agree on certain non-conflicting sequences, and implement them in some library/stack. And then what? What do you do with documents in Japanese that have already been converted without using your new codepoints? What do you do with programs that ignore your new standard? What do you do about search being broken in most apps with no prospect of fixing it? There are already some PUA codepoints for a proper implementation of Japanese, but most app authors don't even understand the problem, never mind being willing to support something that's "non-standard". Asking them to support traditional non-unicode encodings, which is something that's at least relatively well-known and standardised, and something they can test without knowing Japanese, is much easier. > By the way, is it possible to make proper quotes from other languages as described above when using Shift JIS? No, not for arbitrary languages. If you want to mix arbitrary languages, you need some structure that's a rope of strings with an encoding per segment. But that's exactly the same thing that the unicode advocates claim is easy ("just have a sequence of spans with different lang tags"), and at least if you use traditional encodings then you don't have to do any such fancy business for the much more common case of a file that's entirely in Japanese (perhaps with some ASCII keywords).

by lmm 3y ago

ogurechny 3y ago

But there is no “naive” implementation of Unicode. It won't handle emoji, normalization, and thousands other things. People use The Library anyway. Adding another mutagen to it should be no different from the rest. As for official stance, maybe it's time for some group to agree on certain non-conflicting sequences, and implement them in some library/stack. A particularly evil solution would choose arbitrary existing combining characters that can't be declared “illegal” retroactively. By the way, is it possible to make proper quotes from other languages as described above when using Shift JIS?

lmm 3y ago
> But there is no “naive” implementation of Unicode. I assure you there are thousands if not millions of apps that implement search the naive way, by comparing bytes. > It won't handle emoji, normalization, and thousands other things. Yeah. But it works well enough for Americans, so the maintainers don't care enough to fix it. > As for official stance, maybe it's time for some group to agree on certain non-conflicting sequences, and implement them in some library/stack. And then what? What do you do with documents in Japanese that have already been converted without using your new codepoints? What do you do with programs that ignore your new standard? What do you do about search being broken in most apps with no prospect of fixing it? There are already some PUA codepoints for a proper implementation of Japanese, but most app authors don't even understand the problem, never mind being willing to support something that's "non-standard". Asking them to support traditional non-unicode encodings, which is something that's at least relatively well-known and standardised, and something they can test without knowing Japanese, is much easier. > By the way, is it possible to make proper quotes from other languages as described above when using Shift JIS? No, not for arbitrary languages. If you want to mix arbitrary languages, you need some structure that's a rope of strings with an encoding per segment. But that's exactly the same thing that the unicode advocates claim is easy ("just have a sequence of spans with different lang tags"), and at least if you use traditional encodings then you don't have to do any such fancy business for the much more common case of a file that's entirely in Japanese (perhaps with some ASCII keywords).