3 ms·

Has there been any attempt to solve this in Unicode, in-band? Let's say there was a control char for "this is chinese, start", "this is chinese, end" etc.

lmm 5y ago
There's an alternate set of codepoints, but existing software will "convert" SJIS into unicode in an information-destroying way so you have problems like "user enters a search string, software uses Chinese codepoints for that search string, doesn't find the matching phrase in the document". Stateful control characters make unicode mostly pointless - the whole point is to be self-synchronizing and have a universal representation for each character. (Granted emoji are busy destroying that already).
chrismorgan 5y ago
Unicode 3.1 introduced U+E0000–U+E007F for in-band language tagging, using what’s now called BCP 47 language tags. (https://www.unicode.org/reports/tr27/tr27-4.html https://www.unicode.org/reports/tr27/tr27-4.html, heading “13.7 Tag Characters (new section)”.) Right from their introduction, their use was “strongly discouraged”: they’re designed for use with special protocols, with out-of band tagging preferred. In Unicode 5.2, I think, this range was elevated from “strongly discouraged” to “deprecated”: https://www.unicode.org/versions/Unicode5.2.0/ch16.pdf#page=22 https://www.unicode.org/versions/Unicode5.2.0/ch16.pdf#page=.... In-band signalling in Unicode in general is fraught. The Unicode 5.2.0 specification linked goes on to show various of the reasons why this sort of tagging is generally problematic and should not be used in normal text. (And this is why they were strongly discouraged from the start.) Text direction signalling is another troublesome area of Unicode; there are multiple techniques, some strongly discouraged, and it’s a perennial source of interesting security bugs. The only reason direction signalling is supported at all is because it’s needed. Life would be easier with it gone.
ncmncm 5y ago
In-band signaling is always a reliable route to (typically slow-motion) disaster.

lmm 5y ago

There's an alternate set of codepoints, but existing software will "convert" SJIS into unicode in an information-destroying way so you have problems like "user enters a search string, software uses Chinese codepoints for that search string, doesn't find the matching phrase in the document". Stateful control characters make unicode mostly pointless - the whole point is to be self-synchronizing and have a universal representation for each character. (Granted emoji are busy destroying that already).

chrismorgan 5y ago

Unicode 3.1 introduced U+E0000–U+E007F for in-band language tagging, using what’s now called BCP 47 language tags. (https://www.unicode.org/reports/tr27/tr27-4.html https://www.unicode.org/reports/tr27/tr27-4.html, heading “13.7 Tag Characters (new section)”.) Right from their introduction, their use was “strongly discouraged”: they’re designed for use with special protocols, with out-of band tagging preferred. In Unicode 5.2, I think, this range was elevated from “strongly discouraged” to “deprecated”: https://www.unicode.org/versions/Unicode5.2.0/ch16.pdf#page=22 https://www.unicode.org/versions/Unicode5.2.0/ch16.pdf#page=.... In-band signalling in Unicode in general is fraught. The Unicode 5.2.0 specification linked goes on to show various of the reasons why this sort of tagging is generally problematic and should not be used in normal text. (And this is why they were strongly discouraged from the start.) Text direction signalling is another troublesome area of Unicode; there are multiple techniques, some strongly discouraged, and it’s a perennial source of interesting security bugs. The only reason direction signalling is supported at all is because it’s needed. Life would be easier with it gone.

ncmncm 5y ago

In-band signaling is always a reliable route to (typically slow-motion) disaster.