3 ms·

Incorrect. You CAN concatenate, assuming your inputs are valid the outputs will also be valid; BUT normalization is now ONLY correct if all inputs were of the

Incorrect. You CAN concatenate, assuming your inputs are valid the outputs will also be valid; BUT normalization is now ONLY correct if all inputs were of the same format. This can be fixed with libraries if you care, and if you don't, it often doesn't matter. Edit-Additional: If something DOES care, it SHOULD enforce the normalization format it wants on the input boundary. UTF-16 cannot be split any differently than any other encoding, including UTF-32; all must pass through a library that understands compositing characters and sequences. There is no escaping this (other than your own code becoming such a library). Most of the other issues you fault are shared by _any_ encoding of Unicode. However a notable thing is that for very specific functions, E.G. searching for a given valid Unicode sequence and replacing it with another... (E.G. replace someone's name) you'll nearly always be able to do this in without issue. To always do it without issue additional checks must be made around combining characters at the boundary edges. (Which I wouldn't want to maintain, so I'd still call a library unless matching against known will never be such a value, such as quotes or other configuration file control characters.)

tsimionescu 6y ago
> You CAN concatenate, assuming your inputs are valid the outputs will also be valid Is that true even if you mix LTR and RTL text? I have a suspicion there could be problems there, but I'm happy to be wrong. I would still say that safe concatenation is a minor boon for the cost of enforced UTF-8 compliance. > Most of the other issues you fault are shared by _any_ encoding of Unicode. That is exactly my point - that UTF-8, UTF-16, UTF-32, byte strings that could be invalid UTF-8 are all basically equally bad if you want to guarantee meaningful text. A text consisting of two valid UTF-8 code points representing combining characters is no more meaningful than a text consisting of 4 bytes that are invalid UTF-8 code points. > searching for a given valid Unicode sequence and replacing it with another... (E.G. replace someone's name) you'll nearly always be able to do this in without issue. I don't agree, unless you are talking about something extremely 'programmatic' like someone's username or email. For real text apps, you'd want something even more complex than Unicode, that could recognize that 'Ștefan' and 'Stefan' are the same name.
- mjevans 6y ago
  Blindly concatenating any two byte-streams, irrespective of encodings, would also be an invalid assumption; however at that point you've got massive design issues. You cannot concatenate any random 16 or 32 bit character sequences as the underlying byte forms might be Little or Big Endian stored. UTF-8 does not need that consideration, BUT you must know (or at least contractually expect) valid UTF-8 input; anything else is garbage in garbage out. The LTR RTL issue... depends on if all RTL strings _must_ end with an LTR character; in your use case. https://en.wikipedia.org/wiki/Bidirectional_text https://en.wikipedia.org/wiki/Bidirectional_text Your suggestion of delegating all concatenation operations to that is a safe default. Other options might also be valid, depending on the exact context of inputs and use case.

tsimionescu 6y ago

> You CAN concatenate, assuming your inputs are valid the outputs will also be valid Is that true even if you mix LTR and RTL text? I have a suspicion there could be problems there, but I'm happy to be wrong. I would still say that safe concatenation is a minor boon for the cost of enforced UTF-8 compliance. > Most of the other issues you fault are shared by _any_ encoding of Unicode. That is exactly my point - that UTF-8, UTF-16, UTF-32, byte strings that could be invalid UTF-8 are all basically equally bad if you want to guarantee meaningful text. A text consisting of two valid UTF-8 code points representing combining characters is no more meaningful than a text consisting of 4 bytes that are invalid UTF-8 code points. > searching for a given valid Unicode sequence and replacing it with another... (E.G. replace someone's name) you'll nearly always be able to do this in without issue. I don't agree, unless you are talking about something extremely 'programmatic' like someone's username or email. For real text apps, you'd want something even more complex than Unicode, that could recognize that 'Ștefan' and 'Stefan' are the same name.

mjevans 6y ago
Blindly concatenating any two byte-streams, irrespective of encodings, would also be an invalid assumption; however at that point you've got massive design issues. You cannot concatenate any random 16 or 32 bit character sequences as the underlying byte forms might be Little or Big Endian stored. UTF-8 does not need that consideration, BUT you must know (or at least contractually expect) valid UTF-8 input; anything else is garbage in garbage out. The LTR RTL issue... depends on if all RTL strings _must_ end with an LTR character; in your use case. https://en.wikipedia.org/wiki/Bidirectional_text https://en.wikipedia.org/wiki/Bidirectional_text Your suggestion of delegating all concatenation operations to that is a safe default. Other options might also be valid, depending on the exact context of inputs and use case.