7 ms·
...in web apps (i.e. during presentation). Don’t do it at the storage layer: https://github.com/git/git/commit/76759c7dff53e8c84e975b88cb8245587c14c7ba https:/
by js2 8y ago
...in web apps (i.e. during presentation). Don’t do it at the storage layer:
https://github.com/git/git/commit/76759c7dff53e8c84e975b88cb8245587c14c7ba https://github.com/git/git/commit/76759c7dff53e8c84e975b88cb...
Edit: see comments below. My generalization is over broad. Maybe a fairer statement is that some forms of normalization lead to aliasing and sometimes you want that but sometimes not. So be aware of whether you want different strings to be treated the “same” or not.
My thought was that you can always test for sameness after the fact, but once you’ve normalized into storage, you can’t undo it.
- rpedela 8y agoDepends on the context such as filenames on Mac which you link. There are many cases where doing it at the storage layer makes sense. Are you indexing text for search? Normalize. Do you want to de-duplicate textual data and store it in a database? Normalize. Are you going to run some sort of NLP analysis? Normalize.
- jrochkind1 8y agoEh, in general I think it makes a lot of sense to do it at the storage layer. That particular commit mentions _filenames_. I agree you should _never_ touch the bytes that are meant to be a filepath. file systems still do idiosyncratic things with non-ascii file paths, and most of us aren't filesystem experts. Leave the bytes of a filepath alone. Since git is all about filepaths, it makes sense that git would want to generally avoid this. But in general, "during presentation" is not enough to deal with the sorts of problems the OP talks about. If you're comparing strings somewhere, it's probably before "presentation". In general, I think it's quite reasonable to normalize your input, on the way in, to NFC. I think it's reasonable enough in most cases that normalizing input to NFC on the way in is a reasonable "default" to get started with, unless you know a reason you shouldn't. (For searching, you MIGHT want to normalize to NFKC, but that is "lossy" so I would never do that as a rule. I'd normally do it in some other field, and keep the original lossless copy too).
- magduf 8y agoSorry if this sounds a little clueless, but it sounds like the problem is because there's multiple Unicode standards (UTF-8, UTF-16, UTF-32). So it seems like if you just re-encode everything into one of these before committing to the storage layer, that you'd avoid this problem altogether, and you'd be able to do operations in the storage layer correctly too.
- jrochkind1 8y agoYeah, no. That is actually a different issue. Curious if you read the OP? But, no, you do not avoid issues with unicode normalization by ensuring everything is in (eg) UTF-8. Unicode normalization is a thing within UTF-8.
- happytoexplain 8y agoA Unicode string is an array of Unicode's building blocks: Code points. Normalization/composition refers to how we use code points to represent characters. Encoding refers to how we use bytes to represent code points. The two concepts exist at two independent levels of abstraction.
- dummyfunnytoo 8y agoAccurate but so arrogant. HN purity achieved.
- happytoexplain 8y agoSorry, I didn't intend that. I honestly considered it a reasonable question.
- FabHK 8y agoNo worries, your reply was a concise clarification of the issue and terminology, and a reasonable answer; I didn't perceive it as arrogant in the least.