2 ms·

Im pretty sure we both agree that equality should not depend on the look of the character, as you said you prefer byte comparison. Treating different byte enco

Im pretty sure we both agree that equality should not depend on the look of the character, as you said you prefer byte comparison. Treating different byte encodings of the same character as equal makes sense to me, because the different possibilities of representing the characters are just an artifact of the encoding. It’s like comparing negative and positive zero floats. It’s the same value, but the encoding allows multiple equivalent representations. I wouldn’t call the matching after normalization fuzzy, because the thing you want to compare is the actual characters, not the way they are encoded. You’re right that doing normalization when diffing files would be wrong, but that’s because you don’t want to compare strings when doing diffs, you want to compare binary representation. Edit: to make my point more clear: If I give my string comparison function different strings containing the same characters, but encoded in different ways (for example UTF-8 and UTF-16), I would expect the function to tell me they are equal, because the function is supposed to compare the string content, not the way it is encoded in memory.

tsimionescu 2y ago
> Im pretty sure we both agree that equality should not depend on the look of the character, as you said you prefer byte comparison. My position is that "string equality" is a meaningless term on its own, that different contexts require different notions of equality and that none of these is fundamentally better than any other. "the strings look the same to a human", "the strings are represented by the same bytes on a wire/disk", "the strings represent the same characters even if encoded differently", "the strings represent the same word to a native", "the strings represent the same street name to a native", "the strings represent the same human name", etc are all just as valid concepts of "string equality" and are useful in different more or less narrow contexts. Your point that the need for normalization is just an artifact of the encoding is fair enough, but there are still valid contexts, even beyond pure byte comparisons, that require these to be considered different - if nothing else, then at least in posts explaining this very difference. > If I give my string comparison function different strings containing the same characters, but encoded in different ways (for example UTF-8 and UTF-16), I would expect the function to tell me they are equal, because the function is supposed to compare the string content, not the way it is encoded in memory. A perfectly valid equality, just like many others I mentioned.

tsimionescu 2y ago

> Im pretty sure we both agree that equality should not depend on the look of the character, as you said you prefer byte comparison. My position is that "string equality" is a meaningless term on its own, that different contexts require different notions of equality and that none of these is fundamentally better than any other. "the strings look the same to a human", "the strings are represented by the same bytes on a wire/disk", "the strings represent the same characters even if encoded differently", "the strings represent the same word to a native", "the strings represent the same street name to a native", "the strings represent the same human name", etc are all just as valid concepts of "string equality" and are useful in different more or less narrow contexts. Your point that the need for normalization is just an artifact of the encoding is fair enough, but there are still valid contexts, even beyond pure byte comparisons, that require these to be considered different - if nothing else, then at least in posts explaining this very difference. > If I give my string comparison function different strings containing the same characters, but encoded in different ways (for example UTF-8 and UTF-16), I would expect the function to tell me they are equal, because the function is supposed to compare the string content, not the way it is encoded in memory. A perfectly valid equality, just like many others I mentioned.