3 ms·
The official unicode documentation on normalization is good reading, and quite readable. It's actually an even more complicated topic than OP reveals, but the U
by jrochkind1 8y ago
The official unicode documentation on normalization is good reading, and quite readable. It's actually an even more complicated topic than OP reveals, but the Unicode Standard Annex #15 explains it well.
http://unicode.org/reports/tr15/ http://unicode.org/reports/tr15/
OP has a significant error:
> You can choose whatever form you’d like, as long as you’re consistent, so the same input always leads to the same result.
Not so much! Do _not_ use the "Compatibility" (rather than "Canonical") normalization forms unless you know what you are doing! UAX15 will explain why, but they are "lossy". In general, NFC is the one to use as a default.
- zokier 8y ago> In general, NFC is the one to use as a default. Why nfc instead of nfd?
- minitech 8y agoSaves space, is just what people expect, easier to work fast and loose with when you want to do that.
- rurban 8y agoBut needs a third more time than NFD. Apple choose the long form NFD in its old filesystem. NFC needs three steps, NFD only the first two. The size rarely matters that much.
- jrochkind1 8y agoNot really sure why NFC has become the general standard. UAX#15 notes that: > The W3C Character Model for the World Wide Web 1.0: Normalization [CharNorm] and other W3C Specifications (such as XML 1.0 5th Edition) recommend using Normalization Form C for all content, because this form avoids potential interoperability problems arising from the use of canonically equivalent, yet different, character sequences in document formats on the Web. See the W3C Character Model for the Word Wide Web: String Matching and Searching [CharMatch] for more background. One of the W3C documents cited says: > NFC has the advantage that almost all legacy data (if transcoded trivially, one-to-one, to a Unicode encoding), as well as data created by current software or entered by users on most (but not all) keyboards, is already in this form. NFC also has a slight compactness advantage and is a better match to user expectations in most languages with respect to the relationship between characters and graphemes. https://www.w3.org/TR/charmod-norm/ https://www.w3.org/TR/charmod-norm/
- jfk13 8y agoTo conform with W3C recommendations, for one thing: "Content authors SHOULD use Unicode Normalization Form C (NFC) wherever possible for content."[1] It's a SHOULD rather than a MUST. Non-NFC content is not a spec violation, but using NFC is strongly advised. [1] https://www.w3.org/TR/charmod-norm/#normalizationChoice https://www.w3.org/TR/charmod-norm/#normalizationChoice
- SloopJon 8y agoLigatures, which are mentioned in the article, are a good example of the distinction between canonical and compatibility forms. If your input contains "traffic", you don't necessarily want NFC to insert the "ffi" ligature and turn it into "tra\ufb03c". The ligature is generally a poor choice in a monospaced font, for example. Similarly, if you've gone to the trouble to insert the ligature, you don't necessarily want NFD to strip it out. However, with NFKC or NFKD, a search for "traffic" will find the string.
- lelf 8y agoLigatures are not a good example actually. They are obsolete by Unicode, i. e. there's no way to turn ffi⟨normal⟩ into ffi⟨ligature⟩. (But NFKD(ffi⟨ligarure⟩)≡ffi⟨normal⟩ of course.)
- jrochkind1 8y agoYou're right that there's no way to apply any specified unicode normalization to turn `ffi`(normal) into `ffi`(ligature). But they're a good example of the difference between "canonical" and "compatible" normalization anyway, in the other direction. Which is really the only direction that matters to illustrate the difference anyway. NFKD (or NFKC too I think?) will, as you say, turn the ligatures into their "individual" forms. It's different glyphs, and there is at that point no way to know which it was "originally", and no way to convert it in the other direction. The "compatibility" normalizations are "lossy". The "canonical" normalizations on the other hand are basically 'lossless' with regard to "glpyhs". Unless you actually _cared_ that it was represneted with a combining diacritic before, which in 99.9% of cases you don't. You should have exactly the same symbols on the screen after a 'canonical' normalization. And for any x, NFC(x) == NFC(NFD(NFC(x)). The compatibility normalizations are super useful, because as the person up there mentioned, you often want a search query for `ffi` to match on `ffi` (and vice versa). But they are intended to lose symbolic representation (ffi and ffi are now the same thing with no way to distinguish), where the canonical normalizations are not. I wouldn't say the `ffi` ligature is "obsoleted" in any way. People still use it all the time. We both just included it in our comments, unicode was happy to support that. Unicode is so happy for you to use it, that it provides compatibility normalization to make it _easier_ to recognize that it means the same thing as "ffi". :)