5 ms·

I presume you’re talking about Han unification? It’s true that sometimes you may need language annotations, which will sometimes also need to be applied to sub

• lmm 5y ago
  > It’s true that sometimes you may need language annotations, which will sometimes also need to be applied to substrings. I don’t think that invalidates my claim that rigid UTF-8 allows you to have sanity, though I will tweak it to state that using UTF-8 is a necessary but not always sufficient condition. Given that context, I don't see that UTF-8 actually helps much. Your fundamental structure has to look something like a rope of (byte sequence+annotation) entries. With that structure, using different encodings for different segments doesn't make things noticeably worse.
  • TorKlingberg 5y ago
    I'd say having your Chinese and Japanese text in different encodings would make it worse. In a markup language the annotations can be inline. HTML has e.g. <span lang="ja">, which seems to work well enough.
    • AnIdiotOnTheNet 5y ago
      Someday programmers will have learned their lesson about in-band signaling, but apparently it won't be today.
    • lmm 5y ago
      That seems to have all the same problems as having parts of the document in different encodings - e.g. if you move a section of text from one part of the document to another it will suddenly change behaviour.