4 ms·

You mentioned quite a lot of things here. Not really a spectrum, but on one side you have a comparison of bits (bytes), where only maybe byte order may cause is

You mentioned quite a lot of things here. Not really a spectrum, but on one side you have a comparison of bits (bytes), where only maybe byte order may cause issues, and on the other you have such high-level problems like comparing synonyms "begin" == "start" (or you could go further and compare sentences if they convey the same information). Somewhere between those extremes you have characters, glyphs, "confusables", normalization... I don't follow on your CJK unification argument... How are strings different if they share the same codepoint? Is there software that analyzes the language used, or otherwise context, and prints the codepoint differently based on the context? Or is there a recommendation to do so?

tsimionescu 2y ago
> Not really a spectrum, but on one side you have a comparison of bits (bytes), where only maybe byte order may cause issues, and on the other you have such high-level problems like [...] Yes, and my point is that various real-world problems will require various solutions from this spectrum. > How are strings different if they share the same codepoint? Is there software that analyzes the language used, or otherwise context, and prints the codepoint differently based on the context? Or is there a recommendation to do so? Yes, any renderer is supposed to choose a (slightly) different glyph for those codepoints based on the user's chosen locale. Wikipedia has a table of examples [0] of various codepoints that are supposed to be slightly different in Chinese (Simplified), Chinese (Traditional), Japanese, Korean, or Vietnamese. It of course depends on your use case if you'd care beyond that. For example, if the problem you're solving is "are these two strings going to render identically", then the code points don't help, you also need to know that you need to take into account locale. I can imagine some (very niche, to be sure) use cases like this in the area of OCR or of web page comparison. [0] https://en.m.wikipedia.org/wiki/Han_unification#Examples_of_language-dependent_glyphs https://en.m.wikipedia.org/wiki/Han_unification#Examples_of_...

tsimionescu 2y ago

> Not really a spectrum, but on one side you have a comparison of bits (bytes), where only maybe byte order may cause issues, and on the other you have such high-level problems like [...] Yes, and my point is that various real-world problems will require various solutions from this spectrum. > How are strings different if they share the same codepoint? Is there software that analyzes the language used, or otherwise context, and prints the codepoint differently based on the context? Or is there a recommendation to do so? Yes, any renderer is supposed to choose a (slightly) different glyph for those codepoints based on the user's chosen locale. Wikipedia has a table of examples [0] of various codepoints that are supposed to be slightly different in Chinese (Simplified), Chinese (Traditional), Japanese, Korean, or Vietnamese. It of course depends on your use case if you'd care beyond that. For example, if the problem you're solving is "are these two strings going to render identically", then the code points don't help, you also need to know that you need to take into account locale. I can imagine some (very niche, to be sure) use cases like this in the area of OCR or of web page comparison. [0] https://en.m.wikipedia.org/wiki/Han_unification#Examples_of_language-dependent_glyphs https://en.m.wikipedia.org/wiki/Han_unification#Examples_of_...