5 ms·

I didn't know about this, but can't help to think this sounds like a bug in unicode to me. If these characters are different then why does unicode assign one co

by hannob 5y ago

I didn't know about this, but can't help to think this sounds like a bug in unicode to me. If these characters are different then why does unicode assign one codepoint to them? Wasn't the promise of unicode to exactly not do this kind of thing? Can this be fixed? New character code for ambiguous characters could be assigned, of course this would require manual conversion (with knowledge of the variant) for existing data, but at least it would make this issue go away moving forward (and unconverted legacy data would be "just as bad" as it used to be, so no loss).

oleganza 5y ago
This issue with Han Unification was a big reason for stalled adoption of Unicode/UTF-8 in Ruby for years. UTF-8 by default came to Ruby after 1.9 where they've added thorough support for variety of encodings, so that UTF-8 is not the only option.
oleganza 5y ago
Han Unification started in the 90s when computers were big, memory small, UTF-8 did not exist and people were trying to fit all characters in a reasonable amount of codepoints. Today with variable-length encoding of UTF-8 and video streams over 5G, supporting all variants as distinct codepoints, and patching text search and sorting with more "normalization" algorithms would not be a problem at all.
- afiori 5y ago
  in my opinion utf8 should have been a bigger variable length encoding, today it is: 0xxxxxxx 110xxxxx 10xxxxxx 1110xxxx 10xxxxxx 10xxxxxx and 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx the only reason not to push those last bits and add 111110xx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 1111110x 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 11111110 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx and maybe even 11111111 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx is utf-32, they should have dropped it and solve the codepoint problem this way.
  - lifthrasiir 5y ago
    First I'd like to introduce you to this: http://ucsx.org/ http://ucsx.org/ But no, there is no particular reason to introduce a longer encoding than the modern UTF-8 (which is actually shortened from the original one-to-six-byte encoding). The current set of 1,114,112 Unicode characters is sufficient for at least the foreseeable future, because any new assignment requires a demonstrable historic or current use. (Emojis are slightly different, but they still require that the underlying concept is widespread and do not significantly overlap with existing emojis. See [1].) Han characters are the largest source of new assignments to this date and they are yet to reach two out of 17 full planes (that would equate to 131K characters). [1] https://news.ycombinator.com/item?id=26904980 https://news.ycombinator.com/item?id=26904980
jhanschoo 5y ago
This isn't a uniquely CJK problem, see my other comment https://news.ycombinator.com/item?id=29024422 https://news.ycombinator.com/item?id=29024422 The other approach is to assign to each language distinct codepoints, but I guess the current approach is better for backward compatibility with pre-Unicode, and less redundancy in Latin script documents.

by hannob 5y ago

oleganza 5y ago

This issue with Han Unification was a big reason for stalled adoption of Unicode/UTF-8 in Ruby for years. UTF-8 by default came to Ruby after 1.9 where they've added thorough support for variety of encodings, so that UTF-8 is not the only option.

oleganza 5y ago

Han Unification started in the 90s when computers were big, memory small, UTF-8 did not exist and people were trying to fit all characters in a reasonable amount of codepoints. Today with variable-length encoding of UTF-8 and video streams over 5G, supporting all variants as distinct codepoints, and patching text search and sorting with more "normalization" algorithms would not be a problem at all.

afiori 5y ago
in my opinion utf8 should have been a bigger variable length encoding, today it is: 0xxxxxxx 110xxxxx 10xxxxxx 1110xxxx 10xxxxxx 10xxxxxx and 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx the only reason not to push those last bits and add 111110xx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 1111110x 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 11111110 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx and maybe even 11111111 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx is utf-32, they should have dropped it and solve the codepoint problem this way.
- lifthrasiir 5y ago
  First I'd like to introduce you to this: http://ucsx.org/ http://ucsx.org/ But no, there is no particular reason to introduce a longer encoding than the modern UTF-8 (which is actually shortened from the original one-to-six-byte encoding). The current set of 1,114,112 Unicode characters is sufficient for at least the foreseeable future, because any new assignment requires a demonstrable historic or current use. (Emojis are slightly different, but they still require that the underlying concept is widespread and do not significantly overlap with existing emojis. See [1].) Han characters are the largest source of new assignments to this date and they are yet to reach two out of 17 full planes (that would equate to 131K characters). [1] https://news.ycombinator.com/item?id=26904980 https://news.ycombinator.com/item?id=26904980

jhanschoo 5y ago

This isn't a uniquely CJK problem, see my other comment https://news.ycombinator.com/item?id=29024422 https://news.ycombinator.com/item?id=29024422 The other approach is to assign to each language distinct codepoints, but I guess the current approach is better for backward compatibility with pre-Unicode, and less redundancy in Latin script documents.