3 ms·
You cannot have sanity while handling international text as solely UTF-8 bytesequences (or any other encoding that treats sequences of unicode codepoints as alw
by lmm 5y ago
You cannot have sanity while handling international text as solely UTF-8 bytesequences (or any other encoding that treats sequences of unicode codepoints as always equivalent). Sooner or later you will have to deal with text that contains both Chinese and Japanese characters, so you will need a richer representation.
- chrismorgan 5y agoI presume you’re talking about Han unification? It’s true that sometimes you may need language annotations, which will sometimes also need to be applied to substrings. I don’t think that invalidates my claim that rigid UTF-8 allows you to have sanity, though I will tweak it to state that using UTF-8 is a necessary but not always sufficient condition.
- lmm 5y ago> It’s true that sometimes you may need language annotations, which will sometimes also need to be applied to substrings. I don’t think that invalidates my claim that rigid UTF-8 allows you to have sanity, though I will tweak it to state that using UTF-8 is a necessary but not always sufficient condition. Given that context, I don't see that UTF-8 actually helps much. Your fundamental structure has to look something like a rope of (byte sequence+annotation) entries. With that structure, using different encodings for different segments doesn't make things noticeably worse.
- TorKlingberg 5y agoI'd say having your Chinese and Japanese text in different encodings would make it worse. In a markup language the annotations can be inline. HTML has e.g. <span lang="ja">, which seems to work well enough.
- AnIdiotOnTheNet 5y agoSomeday programmers will have learned their lesson about in-band signaling, but apparently it won't be today.
- lmm 5y agoThat seems to have all the same problems as having parts of the document in different encodings - e.g. if you move a section of text from one part of the document to another it will suddenly change behaviour.
- kzrdude 5y agoHas there been any attempt to solve this in Unicode, in-band? Let's say there was a control char for "this is chinese, start", "this is chinese, end" etc.
- lmm 5y agoThere's an alternate set of codepoints, but existing software will "convert" SJIS into unicode in an information-destroying way so you have problems like "user enters a search string, software uses Chinese codepoints for that search string, doesn't find the matching phrase in the document". Stateful control characters make unicode mostly pointless - the whole point is to be self-synchronizing and have a universal representation for each character. (Granted emoji are busy destroying that already).
- chrismorgan 5y agoUnicode 3.1 introduced U+E0000–U+E007F for in-band language tagging, using what’s now called BCP 47 language tags. (https://www.unicode.org/reports/tr27/tr27-4.html https://www.unicode.org/reports/tr27/tr27-4.html, heading “13.7 Tag Characters (new section)”.) Right from their introduction, their use was “strongly discouraged”: they’re designed for use with special protocols, with out-of band tagging preferred. In Unicode 5.2, I think, this range was elevated from “strongly discouraged” to “deprecated”: https://www.unicode.org/versions/Unicode5.2.0/ch16.pdf#page=22 https://www.unicode.org/versions/Unicode5.2.0/ch16.pdf#page=.... In-band signalling in Unicode in general is fraught. The Unicode 5.2.0 specification linked goes on to show various of the reasons why this sort of tagging is generally problematic and should not be used in normal text. (And this is why they were strongly discouraged from the start.) Text direction signalling is another troublesome area of Unicode; there are multiple techniques, some strongly discouraged, and it’s a perennial source of interesting security bugs. The only reason direction signalling is supported at all is because it’s needed. Life would be easier with it gone.
- ncmncm 5y agoIn-band signaling is always a reliable route to (typically slow-motion) disaster.