13 ms·

Why encode the length at all in the extension? Either the next byte is a continuation byte or not. Are there real world use cases where the length encoding is u

Why encode the length at all in the extension? Either the next byte is a continuation byte or not. Are there real world use cases where the length encoding is used in UTF-8 where individual bytes don’t have to be read anyway (for example, codepoint counting, but in practice what good is that without knowing if the code points are combining)? Even in the case where you just want a byte count, an invalid UTF-8 stream won’t respect the first count marker. The arbitrarily sized count is no better than an arbitrarily sized code point - you’d still have to handle all the same resource and sizing issues for safety. And then what, are you going to allocate a buffer that can fit a length that needs more than 64-bits to represent?

kragen 14d ago
It's a brilliant hack. It means that when you're dealing with valid UTF-8 strings, various kinds of string relationships enjoy a homomorphism between byte strings and Unicode strings. Specifically, where S and T are Unicode strings and E is the UTF-8 encoding operation: • E(S concatenated with T) = E(S) concatenated with E(T)† • S starts with T iff E(S) starts with E(T) • S ends with T iff E(S) ends with E(T) • S contains T iff E(T) contains E(T) This means that, as long as you know your encoded strings don't contain invalid UTF-8, you can do a great deal of your string processing on the byte-encoded form, which is enormously faster than decoding the strings before doing the string processing, permits efficient radix-256 tries, and is much smaller in many common cases. It also bounds the work you have to do if you're processing a string starting from the end, while certain other encodings require looking back in the string arbitrarily far to figure out how to interpret the bytes you're looking at. This is particularly important for Boyer–Moore string search, but in many cases it's also an express trip to getting your code featured on https://www.tumblr.com/accidentallyquadratic https://www.tumblr.com/accidentallyquadratic. In short, yes. ______ †Surprisingly, even this most basic homomorphism is not true of many other character encodings, which may need extra bytes to be inserted in between.
faithful_droog 14d ago
Having the length of encoded in each codepoint means that you know if a stream ends at codepoint boundary or if it has a truncated final codepoint. You could probably do away with the length and maintain this property if you had distinct forms for start, continuer-mid and continuer-final bytes.

kragen 14d ago

It's a brilliant hack. It means that when you're dealing with valid UTF-8 strings, various kinds of string relationships enjoy a homomorphism between byte strings and Unicode strings. Specifically, where S and T are Unicode strings and E is the UTF-8 encoding operation: • E(S concatenated with T) = E(S) concatenated with E(T)† • S starts with T iff E(S) starts with E(T) • S ends with T iff E(S) ends with E(T) • S contains T iff E(T) contains E(T) This means that, as long as you know your encoded strings don't contain invalid UTF-8, you can do a great deal of your string processing on the byte-encoded form, which is enormously faster than decoding the strings before doing the string processing, permits efficient radix-256 tries, and is much smaller in many common cases. It also bounds the work you have to do if you're processing a string starting from the end, while certain other encodings require looking back in the string arbitrarily far to figure out how to interpret the bytes you're looking at. This is particularly important for Boyer–Moore string search, but in many cases it's also an express trip to getting your code featured on https://www.tumblr.com/accidentallyquadratic https://www.tumblr.com/accidentallyquadratic. In short, yes. ______ †Surprisingly, even this most basic homomorphism is not true of many other character encodings, which may need extra bytes to be inserted in between.

faithful_droog 14d ago

Having the length of encoded in each codepoint means that you know if a stream ends at codepoint boundary or if it has a truncated final codepoint. You could probably do away with the length and maintain this property if you had distinct forms for start, continuer-mid and continuer-final bytes.