3 ms·

Unicode Transformation Formats are designed to be equivalent in what they can encode. This is simple fact, and sane. Yes, I hate how UTF-16 ruined Unicode, and

Unicode Transformation Formats are designed to be equivalent in what they can encode. This is simple fact, and sane. Yes, I hate how UTF-16 ruined Unicode, and the typical absence of validation of UTF-16 is a problem, but given the abomination that is UTF-16, surrogates not being part of valid Unicode strings is certainly a better course of action than UTF-16 being unable to represent certain Unicode strings. Just about everything should validate UTF-8, because if you don’t, various invariants that you depend on will break. As a simple example, if you operate on unvalidated UTF-8, then all kinds of operations that care about code points will have to check that they’re in-bounds at every single code unit, because to do otherwise would be memory-unsafe, risking crashing or possibly worse. It’s faster and much more peaceful if you validate UTF-8 in all strings at time of ingress, once for all. Given Rust’s preferences for correctness and performance, it is very clearly in the right in validating UTF-8 in all strings.

kazinator 4y ago
> surrogates not being part of valid Unicode strings is certainly a better course of action than UTF-16 being unable to represent certain Unicode strings. I'm okay with people not being able to paste emojis into the Windows port of an application; get yourself a better OS whose wide characters are 32 bits. The Basic Multilingual Plane is good enough for business purposes in the developed world. Surrogates are just some nonsense for Windows and some languages that start with J.
account42 4y ago
> surrogates not being part of valid Unicode strings is certainly a better course of action than UTF-16 being unable to represent certain Unicode strings. I disagree. Better would be to consider UTF-16 a legacy encoding that cannot handle all code points and agressively replace it with a better one (UTF-8 in pretty much all cases) just like we did with UCS-2 and ASCII before it. > It’s faster and much more peaceful if you validate UTF-8 in all strings at time of ingress, once for all. If you don't want bounds checks for every byte then all need is to add a buffer zone to the end of the string anyway (or parse the last < 4 bytes separately) which is exactly what vectorized UTF-8 parsers do already. But many operations on UTF-8 strings don't need to parse them anyway and those that do probably need to do much heaver unicode processing such as normalization or case folding anyway. Being able to preserve all input is much more important than any theoretical concerns about additional bound checking performance.

kazinator 4y ago

> surrogates not being part of valid Unicode strings is certainly a better course of action than UTF-16 being unable to represent certain Unicode strings. I'm okay with people not being able to paste emojis into the Windows port of an application; get yourself a better OS whose wide characters are 32 bits. The Basic Multilingual Plane is good enough for business purposes in the developed world. Surrogates are just some nonsense for Windows and some languages that start with J.

account42 4y ago

> surrogates not being part of valid Unicode strings is certainly a better course of action than UTF-16 being unable to represent certain Unicode strings. I disagree. Better would be to consider UTF-16 a legacy encoding that cannot handle all code points and agressively replace it with a better one (UTF-8 in pretty much all cases) just like we did with UCS-2 and ASCII before it. > It’s faster and much more peaceful if you validate UTF-8 in all strings at time of ingress, once for all. If you don't want bounds checks for every byte then all need is to add a buffer zone to the end of the string anyway (or parse the last < 4 bytes separately) which is exactly what vectorized UTF-8 parsers do already. But many operations on UTF-8 strings don't need to parse them anyway and those that do probably need to do much heaver unicode processing such as normalization or case folding anyway. Being able to preserve all input is much more important than any theoretical concerns about additional bound checking performance.