4 ms·
What can you actually do with a known-valid UTF-8 (but otherwise of unkown structure) string that you can't do with a UTF-16 string or even byte-based string?
by tsimionescu 6y ago
What can you actually do with a known-valid UTF-8 (but otherwise of unkown structure) string that you can't do with a UTF-16 string or even byte-based string?
You can't concatenate, split, enumerate, assume they are valid human text, capitalize, count the number of characters, turn to lower/uppercase etc.
In general, you need a text handling library to do anything meaningful with text if you want to handle internationalization, and then the text handling library can also handle all of the encoding problems easily, all that matters is having a known encoding so you don't need to guess.
It's nice to have specific types for specific encodings of strings, but otherwise I think there is little to be gained by representing strings as anything other than either byte arrays or text.
- mjevans 6y agoIncorrect. You CAN concatenate, assuming your inputs are valid the outputs will also be valid; BUT normalization is now ONLY correct if all inputs were of the same format. This can be fixed with libraries if you care, and if you don't, it often doesn't matter. Edit-Additional: If something DOES care, it SHOULD enforce the normalization format it wants on the input boundary. UTF-16 cannot be split any differently than any other encoding, including UTF-32; all must pass through a library that understands compositing characters and sequences. There is no escaping this (other than your own code becoming such a library). Most of the other issues you fault are shared by _any_ encoding of Unicode. However a notable thing is that for very specific functions, E.G. searching for a given valid Unicode sequence and replacing it with another... (E.G. replace someone's name) you'll nearly always be able to do this in without issue. To always do it without issue additional checks must be made around combining characters at the boundary edges. (Which I wouldn't want to maintain, so I'd still call a library unless matching against known will never be such a value, such as quotes or other configuration file control characters.)
- tsimionescu 6y ago> You CAN concatenate, assuming your inputs are valid the outputs will also be valid Is that true even if you mix LTR and RTL text? I have a suspicion there could be problems there, but I'm happy to be wrong. I would still say that safe concatenation is a minor boon for the cost of enforced UTF-8 compliance. > Most of the other issues you fault are shared by _any_ encoding of Unicode. That is exactly my point - that UTF-8, UTF-16, UTF-32, byte strings that could be invalid UTF-8 are all basically equally bad if you want to guarantee meaningful text. A text consisting of two valid UTF-8 code points representing combining characters is no more meaningful than a text consisting of 4 bytes that are invalid UTF-8 code points. > searching for a given valid Unicode sequence and replacing it with another... (E.G. replace someone's name) you'll nearly always be able to do this in without issue. I don't agree, unless you are talking about something extremely 'programmatic' like someone's username or email. For real text apps, you'd want something even more complex than Unicode, that could recognize that 'Ștefan' and 'Stefan' are the same name.
- mjevans 6y agoBlindly concatenating any two byte-streams, irrespective of encodings, would also be an invalid assumption; however at that point you've got massive design issues. You cannot concatenate any random 16 or 32 bit character sequences as the underlying byte forms might be Little or Big Endian stored. UTF-8 does not need that consideration, BUT you must know (or at least contractually expect) valid UTF-8 input; anything else is garbage in garbage out. The LTR RTL issue... depends on if all RTL strings _must_ end with an LTR character; in your use case. https://en.wikipedia.org/wiki/Bidirectional_text https://en.wikipedia.org/wiki/Bidirectional_text Your suggestion of delegating all concatenation operations to that is a safe default. Other options might also be valid, depending on the exact context of inputs and use case.