4 ms·
> surrogates not being part of valid Unicode strings is certainly a better course of action than UTF-16 being unable to represent certain Unicode strings. I di
by account42 4y ago
> surrogates not being part of valid Unicode strings is certainly a better course of action than UTF-16 being unable to represent certain Unicode strings.
I disagree. Better would be to consider UTF-16 a legacy encoding that cannot handle all code points and agressively replace it with a better one (UTF-8 in pretty much all cases) just like we did with UCS-2 and ASCII before it.
> It’s faster and much more peaceful if you validate UTF-8 in all strings at time of ingress, once for all.
If you don't want bounds checks for every byte then all need is to add a buffer zone to the end of the string anyway (or parse the last < 4 bytes separately) which is exactly what vectorized UTF-8 parsers do already. But many operations on UTF-8 strings don't need to parse them anyway and those that do probably need to do much heaver unicode processing such as normalization or case folding anyway.
Being able to preserve all input is much more important than any theoretical concerns about additional bound checking performance.