4 ms·
The author seems to assume that the bit patterns representing commas or quotation marks won't show up in any multibyte (not just unicode) patterns and each byte
by nieve 5y ago
The author seems to assume that the bit patterns representing commas or quotation marks won't show up in any multibyte (not just unicode) patterns and each byte can be interpreted in isolation, but is that actually safe?
- Const-me 5y agoYes, assuming the input is valid UTF-8. UTF-8 bytes less than 0x80 are only used for the first 128 Unicode codepoints. Characters who need multiple bytes in UTF8 representation encode into bytes with high bits set, i.e. >= 0x80: https://en.wikipedia.org/wiki/UTF-8#Encoding https://en.wikipedia.org/wiki/UTF-8#Encoding
- CuriousCosmic 5y agoFor any multibyte representation no however I'd say it's safe to just say "all encodings must be valid UTF-8" and require the user/service to validate that first. There are a number of algorithms out there that can validate UTF-8 with significantly less than 1 instruction per byte. I'd imagine the overhead for pipelining the two is significantly cheaper than trying to handle the cases in the same pass.