12 ms·
Why to normalize Unicode strings
- mises 8y agoWhat is with the push to unicode? Why not ascii? It seems to give a lot less trouble, particularly wrt panagrams, normalization, etc.
- totoglazer 8y agoI think it might have something to do with the ~7 billion people whose native language isn’t expressible in ascii. Just maybe?
- happytoexplain 8y agoASCII is effectively just the keys on a US English keyboard. You couldn't even represent an accented 'e' or the price of British products. So even if you just wanted a character set defined as "very common characters in the English speaking world", ASCII falls miles short of even that tiny goal.
- mises 8y agoMost of the internet is based on American English. See programming for an example. For the goal of representing American English, it works fine. Why would anything more be necessary?
- function_seven 8y agoUnicode is not about the Internet. Just like ASCII or EBCDIC before it weren't about networking. The reason we use Unicode is because ASCII is very limited in its scope. It can't handle the majority of the world's languages. It can't even handle American English. No dashes, no open or close quotes, etc.
- mises 8y agoMost programming is in American english too. And most people who use computers end up picking up a decent bit anyway, according to some friends from a spanish-speaking country.
- function_seven 8y agoSo ASCII is good enough for programming, but fails at general text, unless we resign ourselves to kludges like -- instead of – and --- instead of —. And your friends from that Spanish-speaking country do UI work right? Can’t use ASCII for that. Ñ is out the window.
- mises 8y agoNo, they do c.
- happytoexplain 8y agoIf you're encoding data for a domain guaranteed to contain only US English words and US English symbols, and you're OK without the occasional non-ASCII bits in US English like diacritics in loan words, then sure. But such domains are vanishingly rare. As for your examples, "most of the internet" might still contain non-ASCII, I think, but even a little bit of the internet is still a gigantic and highly active part that's inseparable from the larger domain of "the internet". And, while the legal symbols in the syntax of a programming language might only be ASCII, programming languages must process arbitrary strings, which may be user-facing, or be in some other domain where ASCII is inadequate.
- profmonocle 8y ago> Most of the internet is based on American English. Emoji are rapidly becoming a staple of American English. (Edit: and apparently HN filters emoji out of posts!)
- kalleboo 8y agoそうですね、僕も分からない（´-`）⁉️
- NegativeLatency 8y agoGoogle Translate: "Well, I do not know"
- sevensor 8y agoSpeaking of which, how does Han Unification affect Unicode normalization. If I understand correctly, you can compose strokes into characters? Does that work?
- nabla9 8y agoI have really good news for you. UTF-8 encoding is compatible with ASCII. As long as you just use ASCII characters, they your strings are also valid UTF-8.
- 2038AD 8y agoASCII is pretty limited. For example your namesake's book Nationalökonomie could not properly by rendered in ASCII.
- 2038AD 8y ago*be
- bshipp 8y agoI'm currently processing a huge archive of files where someone was too lazy to bother using the proper encoding. out of the first 700K files I've run through I've dumped 3000 into a secondary queue because they've been so bastardized the mojibake is completely indecipherable. It continues to astonish me that programmers who spend hours arguing the relative merits of ECC RAM and database ACID implementations are so quick to destroy data by lazily defaulting to ASCII.
- js2 8y ago...in web apps (i.e. during presentation). Don’t do it at the storage layer: https://github.com/git/git/commit/76759c7dff53e8c84e975b88cb8245587c14c7ba https://github.com/git/git/commit/76759c7dff53e8c84e975b88cb... Edit: see comments below. My generalization is over broad. Maybe a fairer statement is that some forms of normalization lead to aliasing and sometimes you want that but sometimes not. So be aware of whether you want different strings to be treated the “same” or not. My thought was that you can always test for sameness after the fact, but once you’ve normalized into storage, you can’t undo it.
- rpedela 8y agoDepends on the context such as filenames on Mac which you link. There are many cases where doing it at the storage layer makes sense. Are you indexing text for search? Normalize. Do you want to de-duplicate textual data and store it in a database? Normalize. Are you going to run some sort of NLP analysis? Normalize.
- jrochkind1 8y agoEh, in general I think it makes a lot of sense to do it at the storage layer. That particular commit mentions _filenames_. I agree you should _never_ touch the bytes that are meant to be a filepath. file systems still do idiosyncratic things with non-ascii file paths, and most of us aren't filesystem experts. Leave the bytes of a filepath alone. Since git is all about filepaths, it makes sense that git would want to generally avoid this. But in general, "during presentation" is not enough to deal with the sorts of problems the OP talks about. If you're comparing strings somewhere, it's probably before "presentation". In general, I think it's quite reasonable to normalize your input, on the way in, to NFC. I think it's reasonable enough in most cases that normalizing input to NFC on the way in is a reasonable "default" to get started with, unless you know a reason you shouldn't. (For searching, you MIGHT want to normalize to NFKC, but that is "lossy" so I would never do that as a rule. I'd normally do it in some other field, and keep the original lossless copy too).
- magduf 8y agoSorry if this sounds a little clueless, but it sounds like the problem is because there's multiple Unicode standards (UTF-8, UTF-16, UTF-32). So it seems like if you just re-encode everything into one of these before committing to the storage layer, that you'd avoid this problem altogether, and you'd be able to do operations in the storage layer correctly too.
- athenot 8y agoJust tried this in Perl6; looks like string comparisons Do The Right Thing™. > "\x65\x301".contains("\xe9") True
- lelf 8y ago> "\c[latin small letter e]\c[combining acute accent]" eq "\c[latin small letter e with acute]" True Edit: And of course > "\c[dog face]".chars 1 and not 2 as in the article. PS: WTF? HN strips emojis :/ (and does it incorrectly when they are emoji sequences).
- deleted 8y ago[deleted]
- happytoexplain 8y agoSwift is another major language that has correctly solved this problem in this way - i.e. not representing/operating on strings as though they were naive arrays of bytes or code points - but rather as though they were arrays of characters, which Unicode thoroughly and intuitively defines in the same way that humans think about characters.
- lelf 8y agoSwift is headed in the right direction. But I think Perl6 is the only language that can do this magic: > 'Déjà vu' ~~ /:ignorecase:ignoremark deja \s vu/ ｢Déjà vu｣
- jfk13 8y ago> characters, which Unicode thoroughly and intuitively defines in the same way that humans think about characters "Character" is a somewhat vague term, and Unicode prefers to use more specific terms like "code unit", "code point", "abstract character", etc. In this case I think you may be referring to grapheme clusters, which come closer to how "humans think about characters" than Unicode abstract characters, which are building blocks of the technical encoding standard but in some cases don't really match a human concept of a graphical element of a writing system. See also “Characters” and Grapheme Clusters in section 2.11 of https://www.unicode.org/versions/Unicode12.0.0/ch02.pdf https://www.unicode.org/versions/Unicode12.0.0/ch02.pdf, for example.
- jrochkind1 8y agoThe official unicode documentation on normalization is good reading, and quite readable. It's actually an even more complicated topic than OP reveals, but the Unicode Standard Annex #15 explains it well. http://unicode.org/reports/tr15/ http://unicode.org/reports/tr15/ OP has a significant error: > You can choose whatever form you’d like, as long as you’re consistent, so the same input always leads to the same result. Not so much! Do _not_ use the "Compatibility" (rather than "Canonical") normalization forms unless you know what you are doing! UAX15 will explain why, but they are "lossy". In general, NFC is the one to use as a default.
- zokier 8y ago> In general, NFC is the one to use as a default. Why nfc instead of nfd?
- minitech 8y agoSaves space, is just what people expect, easier to work fast and loose with when you want to do that.
- rurban 8y agoBut needs a third more time than NFD. Apple choose the long form NFD in its old filesystem. NFC needs three steps, NFD only the first two. The size rarely matters that much.
- jrochkind1 8y agoNot really sure why NFC has become the general standard. UAX#15 notes that: > The W3C Character Model for the World Wide Web 1.0: Normalization [CharNorm] and other W3C Specifications (such as XML 1.0 5th Edition) recommend using Normalization Form C for all content, because this form avoids potential interoperability problems arising from the use of canonically equivalent, yet different, character sequences in document formats on the Web. See the W3C Character Model for the Word Wide Web: String Matching and Searching [CharMatch] for more background. One of the W3C documents cited says: > NFC has the advantage that almost all legacy data (if transcoded trivially, one-to-one, to a Unicode encoding), as well as data created by current software or entered by users on most (but not all) keyboards, is already in this form. NFC also has a slight compactness advantage and is a better match to user expectations in most languages with respect to the relationship between characters and graphemes. https://www.w3.org/TR/charmod-norm/ https://www.w3.org/TR/charmod-norm/
- s1mon 8y ago“The first of such conventions, or character encodings, was ASCII (American Standard Code for Information Interchange).” The author may know better and is glossing over history, but when I see statements like this that are obviously incorrect, I question everything else in the article.
- rpedela 8y agoThe article is largely correct. I do a lot of search and NLP work.
- doodpants 8y ago> Thankfully, there’s an easy solution, which is normalizing the string into the “canonical form”. Cool, problem solved! > There are four standard normalization forms: (╯°□°）╯︵ ┻━┻
- Zarel 8y agoI do understand the need for the difference between NFC and NFKC, but in hindsight NFD and NFKD seem entirely unnecessary.
- eridius 8y agoNFD is useful if you want to do a diacritic-insensitive search.
- lokedhs 8y agoThe problem is that diacritic-insensitive search is locale-dependent, so it doesn't do the right thing anyway. Specific example: in English, you'd want a search for ‘a’ to find ‘ä’ while this is the entirely wrong thing in Swedish where a and ä are distinct letters. An English speaker probably wouldn't want a search for ‘i’ to match ‘j’ even though the latter just has an extra hook on the bottom.
- eridius 8y agoThat's a very good point. There's still a use for locale-insensitive diacritic-insensitive searches, but you're absolutely right that in most cases you'd want it to be locale-aware and therefore NFD isn't sufficient (though it may still be easier to do this on NFD than NFC).
- ams6110 8y agoI actually laughed out loud when I got to that sentence. Glad that I still work in plain ASCII.
- gumby 8y agoBy the way the last letter of Zoë is e with a diaresis, not An umlaut. Like the second o in coöperate — it’s just an ordinary o with a marker to tell you to pronounce it rather than form a diphthong.
- WalterBright 8y agoThere shouldn't even be any such thing as normalized strings, i.e. two different Unicode sequences that are supposed to be the same character.
- timw4mail 8y agoSounds great in theory...but you've just vastly extended the number of "characters". While text parsing is a nightmare, you also get the flexibility of combining different characters together, such as emojis and skin tone modifiers, which outputs a different display character. On the other hand, I'm not sure its possible to have a font that represents every valid combination of unicode.
- WalterBright 8y agoUnicode can simply not recognize combining characters where a special code point exists, and vice versa, on a case by case basis. For example, the ä can remain its special code point, and a¨ can display as a¨. Then software that processes Unicode will become straightforward.
- int_19h 8y agoThat ship has sailed for backwards compatibility reasons.
- lozenge 8y agoIn that case you would lose information when transferring text from non-Unicode encoded files to Unicode encoded files. Not a great way for your standard to take off...
- ken 8y ago> Why use both [UTF-8 and UTF-16]? Western languages typically are most efficiently encoded with UTF-8 (since most characters would be represented with 1 byte only), while Asian languages can usually produce smaller files when using UTF-16 as encoding. The second sentence is technically correct, but it's a strange followup here because it's not why UTF-8 and UTF-16 exist today. I don't know any Asian webpages that use UTF-16 to save bandwidth, e.g., Japanese Wikipedia is still UTF-8. The major use of UTF-16 in 2019, AFAICT, is for legacy operating system interfaces.
- ameliaquining 8y agoAlso "legacy" language runtimes. "Legacy" being in scare quotes because JavaScript, the JVM, and the CLR all work this way and are all very much in widespread use today.
- ken 8y agoYes, indeed. I didn't mean "legacy" to imply that they aren't still useful. Simply that many of them predate the creation of codepoints which don't fit in 16 bits -- and it's harder to upgrade a binary API distributed in a client OS, compared to changing a web server's preferred encoding. JS/JVM/CLR all work fine, but I imagine if they were created today, their strings would not be based on UTF-16.
- sebazzz 8y agoUtf8String is in the works for C#/.NET. It would still be pretty much bolted on though.
- Someone 8y agoJDK 9 introduced “compact strings” (https://bugs.openjdk.java.net/browse/JDK-8054307 https://bugs.openjdk.java.net/browse/JDK-8054307). That stores a string’s characters in a byte array, with either the traditional two bytes per ‘char’ (encoding the entire string as UTF16) or, if possible, one (encoding the entire string as ISO-8859-1/Latin-1). They probably didn’t use UTF-8 because it would break the fact that string indexing is O(1).
- gwbas1c 8y agoI still don't understand why Unicode allows two different ways to represent the same thing. Naively, that appears as a major defect in Unicode. Perhaps someone reading this knows why this was the right thing to do?
- goto11 8y agoUnicode was created to unify a number of different pre-existing character sets so they all could be mapped directly to unicode code points. Some of these character-sets had pre-composed characters, e.g. a single code-point to represent 'ö'. Other used combining characters. Unicode therefore had to support both.
- eridius 8y agoIn particular, Unicode codepoints 128-255 were explicitly defined to be the same as ISO-8859-1, which has a bunch of precomposed characters in it.
- paulddraper 8y agoThe Unicode space is massive. It's odd they didn't just spend a bunch of code points for those existing (predominately Latin) character. Unless you are supposed to be able to put an unlaut on any character. CJK characters with umlauts.
- deleted 8y ago[deleted]
- paulddraper 8y agoYes, I am likewise confused. Text is one of those enormously complicated human conventions, so it's probably just ignorance speaking. But I would like to understand.
- zackmorris 8y agoNote that Apple's APFS doesn't normalize Unicode filenames: https://news.ycombinator.com/item?id=13953800 https://news.ycombinator.com/item?id=13953800 From what I understand, it stores them as-is but can read any (so is normalization insensitive): https://medium.com/@yorkxin/apfs-docker-unicode-6e9893c9385d https://medium.com/@yorkxin/apfs-docker-unicode-6e9893c9385d https://developer.apple.com/library/archive/documentation/FileManagement/Conceptual/APFS_Guide/FAQ/FAQ.html https://developer.apple.com/library/archive/documentation/Fi... This hit me a couple of years ago when I was working on a scraper and storing the title of the page as the filename. It looked fine, but would fail a Javascript string comparison. I can't remember if I was using HFS+ though, which I believe saved filenames as NFD: https://en.wikipedia.org/wiki/HFS_Plus#Criticisms https://en.wikipedia.org/wiki/HFS_Plus#Criticisms The same script might work today on APFS.
- watersb 8y agoHFS+ uses Form D. I had to remember to create my ZFS volumes with Form D enabled, as it isn't an attribute that can be changed afterwards. IIRC, ZFS on Mac OS X would set that by default so if you created the volumes from a Mac, then ok. But I was creating my ZFS array on a Linux or OpenSolaris server, where I would need to set Form D Normalization explicitly.