3 ms·

In the Vietnamese example "Bạn khỏe không?" ạ and ỏ have tone markers, but (the way Vietnamese orthography was taught to me), ô does not. That is a totall

In the Vietnamese example "Bạn khỏe không?" ạ and ỏ have tone markers, but (the way Vietnamese orthography was taught to me), ô does not. That is a totally different letter than a naked o. I wonder if distinctions like this have any implications for unicode and normalisation. I imagine the answer is no, because missing a tone marker makes your orthography just as wrong as missing the caret from an ô. Maybe an actual Vietnamese speaker would disagree there.

bawolff 5y ago
Typically there is a big distinction when it comes to collation (e.g unicode collation algorithm) I'm not a unicode expert, but normalization is more about what characters are available in unicode, not the specific language, especially characters which are used in multiple languages (which i assume includes ô).
- WorldMaker 5y ago
  Collation algorithms are ultimately about sorting and normalization algorithms are ultimately about equality. There is a direct overlap because if your collation algorithm doesn't agree with your normalization algorithm you get subtle sorting bugs. I believe in this case things are handled by the collation algorithms all depending on a specific normalization form for input and the dependency arrow pointing that way. But there's a case to be made that the dependency arrow makes just as much sense pointing the other direction, and the proliferation of collation algorithms a likely indicator that language/collation-specific normalization is still a possible overlooked need. There's not really a specific right answer here, and the current dependency arrow makes sense from a simplicity standpoint, whether or not it is a perfect abstraction of the problem domain. There are known gaps, which is why Unicode is up to four different normalization algorithms today.
  - bawolff 5y ago
    I don't get why. UCA is defined over NFD. I suppose you could define it in terms of NFC, but why would you want to? It seems like it would be more annoying but otherwise exactly the same. > There are known gaps, which is why Unicode is up to four different normalization algorithms today. Like what? The four normalization algorithms serve different purposes, its not like the first one was broken so they made a second one.

bawolff 5y ago

Typically there is a big distinction when it comes to collation (e.g unicode collation algorithm) I'm not a unicode expert, but normalization is more about what characters are available in unicode, not the specific language, especially characters which are used in multiple languages (which i assume includes ô).

WorldMaker 5y ago
Collation algorithms are ultimately about sorting and normalization algorithms are ultimately about equality. There is a direct overlap because if your collation algorithm doesn't agree with your normalization algorithm you get subtle sorting bugs. I believe in this case things are handled by the collation algorithms all depending on a specific normalization form for input and the dependency arrow pointing that way. But there's a case to be made that the dependency arrow makes just as much sense pointing the other direction, and the proliferation of collation algorithms a likely indicator that language/collation-specific normalization is still a possible overlooked need. There's not really a specific right answer here, and the current dependency arrow makes sense from a simplicity standpoint, whether or not it is a perfect abstraction of the problem domain. There are known gaps, which is why Unicode is up to four different normalization algorithms today.
- bawolff 5y ago
  I don't get why. UCA is defined over NFD. I suppose you could define it in terms of NFC, but why would you want to? It seems like it would be more annoying but otherwise exactly the same. > There are known gaps, which is why Unicode is up to four different normalization algorithms today. Like what? The four normalization algorithms serve different purposes, its not like the first one was broken so they made a second one.