3 ms·

I'll freely admit to being one of those mystified folk. As an easy example of which, can you roughly outline which n-bit values are not valid UTF-n? (are they

I'll freely admit to being one of those mystified folk. As an easy example of which, can you roughly outline which n-bit values are not valid UTF-n? (are they reserved for shifting to other systems? and if so, isn't that still a valid, even an important value, within UTF-n?)

mikeash 11y ago
The article covers this for the case of UTF-16. To recap, UTF-16 has surrogate pairs, which are a sequence of two 16-bit values in certain ranges. A high surrogate is in the range 0xD800–0xDBFF and a low surrogate is in the range 0xDC00–0xDFFF. This is used to encode values beyond 65,535. If a high surrogate appears without a following low surrogate, or a low surrogate appears without a leading high surrogate, the sequence is not valid UTF-16. For UTF-32, only valid code points are allowed. Values above 0x10FFFF are not allowed, nor are values in the range used for UTF-16 surrogates. UTF-8 is more complex. It's ASCII compatible, which means all values with the top bit set to 0 are allowed. Bytes with the top bit set to 1 must exist as part of a multi-byte sequence. The first byte in such a sequence must contain 2-4 leading 1s followed by a 0. Subsequent bytes must be of the form 10xxxxxx. The total length of the sequence is equal to the number of leading 1s in the first byte. Any deviation from this is invalid UTF-8. Further, any sequence must be minimal, meaning that the value it encodes can't be one that would fit into a smaller sequence.