4 ms·
You can also do that with invalid UTF8. You simply treat the first invalid byte of a codepoint as the first byte of a new codepoint, terminating the previous on
by jakobnissen 2y ago
You can also do that with invalid UTF8. You simply treat the first invalid byte of a codepoint as the first byte of a new codepoint, terminating the previous one. In this way, the self-syncronization (self-dilimiting) of UTF8 is preserved. This is what Julia does.
- LegionMammal978 2y agoSure, that's one way to do it, but if the invalid byte were logically tied to the byte before it in its original encoding, then now your substring split will mangle that character.
- Etherlord87 2y agoIf it's invalid then you don't mangle anything, it's mangled already for whatever reason, no?