4 ms·
If you're reading something in pieces, like a buffer that fills 256 bytes at a time, you have to be careful. UTF-8 is a multi-byte encoding so the last byte in
by makecheck 14y ago
If you're reading something in pieces, like a buffer that fills 256 bytes at a time, you have to be careful. UTF-8 is a multi-byte encoding so the last byte in your buffer may not completely finish a code point. Unlike older code that can just read a bunch of bytes and use them, with multi-byte encodings you have to have a way to deal with "left-overs" until new bytes show up.
Fortunately the UTF-8 encoding (e.g. see the Wikipedia page) makes it clear when a byte is the beginning of a new point and it tells you how many intermediate bytes should follow.