3 ms·
It's surprising how much software (mostly on Windows) doesn't properly handle Unicode in 2021. With something working in Unicode, it's not that big of a deal to
by dhosek 5y ago
It's surprising how much software (mostly on Windows) doesn't properly handle Unicode in 2021. With something working in Unicode, it's not that big of a deal to both handle letters like Ł or ż and also to run normalization on text strings so that you can (if desired) treat Łódź and Lodz as identical (e.g., for text searching).
- bobbylarrybobby 5y agoJust a note that normalization is not the same as diacritic insensitivity. Normalization is the process by which strings that are semantically equivalent (by some standard), yet have different underlying byte sequences, are transformed to have the same underlying byte sequence. For instance, replacing “e, combining acute accent” with “e with acute accent”.
- dhosek 5y agoYes, I should clarify that the process is to normalize to letter+combining accent and then strip out the accents, e.g., in Java: Normalizer.normalize(text, Normalizer.Form.NFD).replaceAll("\\p{M}", ""); (I hope I got this right—I adapted it from some code which actually strips out some other bits of text to turn, e.g., F. Janaček into FJanacek.)