3 ms·
Eh, in general I think it makes a lot of sense to do it at the storage layer. That particular commit mentions _filenames_. I agree you should _never_ touch th
by jrochkind1 8y ago
Eh, in general I think it makes a lot of sense to do it at the storage layer.
That particular commit mentions _filenames_. I agree you should _never_ touch the bytes that are meant to be a filepath. file systems still do idiosyncratic things with non-ascii file paths, and most of us aren't filesystem experts. Leave the bytes of a filepath alone.
Since git is all about filepaths, it makes sense that git would want to generally avoid this.
But in general, "during presentation" is not enough to deal with the sorts of problems the OP talks about. If you're comparing strings somewhere, it's probably before "presentation".
In general, I think it's quite reasonable to normalize your input, on the way in, to NFC. I think it's reasonable enough in most cases that normalizing input to NFC on the way in is a reasonable "default" to get started with, unless you know a reason you shouldn't.
(For searching, you MIGHT want to normalize to NFKC, but that is "lossy" so I would never do that as a rule. I'd normally do it in some other field, and keep the original lossless copy too).
- magduf 8y agoSorry if this sounds a little clueless, but it sounds like the problem is because there's multiple Unicode standards (UTF-8, UTF-16, UTF-32). So it seems like if you just re-encode everything into one of these before committing to the storage layer, that you'd avoid this problem altogether, and you'd be able to do operations in the storage layer correctly too.
- jrochkind1 8y agoYeah, no. That is actually a different issue. Curious if you read the OP? But, no, you do not avoid issues with unicode normalization by ensuring everything is in (eg) UTF-8. Unicode normalization is a thing within UTF-8.
- happytoexplain 8y agoA Unicode string is an array of Unicode's building blocks: Code points. Normalization/composition refers to how we use code points to represent characters. Encoding refers to how we use bytes to represent code points. The two concepts exist at two independent levels of abstraction.
- dummyfunnytoo 8y agoAccurate but so arrogant. HN purity achieved.
- happytoexplain 8y agoSorry, I didn't intend that. I honestly considered it a reasonable question.
- FabHK 8y agoNo worries, your reply was a concise clarification of the issue and terminology, and a reasonable answer; I didn't perceive it as arrogant in the least.
- magduf 8y agoOP here; I didn't perceive it as arrogant either. I think the "arrogant" accuser might have a point about some stuff I see on HN, but this reply to my post just isn't it.
- ivanbakel 8y agoEven then, it's not just filepaths. Internal representations can range from "bytes that are sometimes string-ish" to "bytes which are valid strings", while in no way covering the domain where the string-like properties that normalization brings are desireable. I doubt there are hard-and-fast rules where normalization is appropriate, but if you start applying it to all input, you're going to break things. Email addresses immediately spring to mind, for example.
- jrochkind1 8y agoCan you even have unicode/non-ascii in email addresses? If you can, and your email address is josé@gmail.com, but whether it arrives to you depends on whether the address was entered as unicode codepoint 'LATIN SMALL LETTER E WITH ACUTE' or 'LATIN SMALL LETTER E'+'COMBINING ACUTE ACCENT'... you're going to have a lot of trouble getting email. Becauase people entering your address have no idea which they are entering. I know I can type an "é" on MacOS US keyboard layout by typing `option-e,e`, but I have _no idea_ what actual bytes (corresponding to which of these codepoints) are sent to HN when I enter that in an HN comment text box. And most people sending email, of course, don't even know any of this is a thing. If it works at all, it's actually gonna be because the software on either or both ends (sending and receiving) is already doing NFC normalization! So, yes, this illustrates that normalization _is a thing_ you have to pay attention to, and can get weird. But in general, for "text" input (rather than addresses or filepaths), normalizing to NFC is _usually_ a good call. If it's addresses or filepaths or what have you... it's gonna get confusing no matter what. If you know what you're doing, maybe NFC isn't right. If you don't know what you're doing, NFC is a lot better start-out-with call than not normalizing at all. Not normalizing at all will bite you more often than NFC will.
- ucarion 8y agoIs running normalization algorithms against UCS-2 -- i.e., where it's possible to have invalid surrogate pairs -- well-defined?
- jrochkind1 8y ago
- paulddraper 8y ago> That particular commit mentions _filenames_. This basically just comes down to the (sometime unpleasant) reality that on most Unix systems, file paths are not text; they are byte strings that don't have 00 or 2F bytes.