6 ms·

This is a cool post, but I'm still not convinced I want to do UTF-8 validation in the first place. UTF-8 text, even invalid, has the great property of failing g

This is a cool post, but I'm still not convinced I want to do UTF-8 validation in the first place. UTF-8 text, even invalid, has the great property of failing gracefully in display that a boolean "valid" or "invalid" can never come close to.

vardump 6y ago
It always depends, but most of the time it's better to fail fast. Invalid UTF-8 is a good indication something went wrong. The earlier the failure occurs, the easier it'll be to troubleshoot it. Of course there are situations where you want to make do with your input and try to interpret it the best you can. Sometimes you need to be lenient in what you accept [0]. That's how I'd expect, say, a web browser to operate. Default to not accepting invalid input. Just fail fast. [0]: Be liberal in what you accept, and conservative in what you send - Jon Postel
- jrochkind1 6y ago
  > Of course there are situations where you want to make do with your input and try to interpret it the best you can. Sometimes you need to be lenient in what you accept The best way to do this is to TURN the input into valid UTF8 though. Like replacing any uninterpretable-as-utf8 bytes with the unicode replacement character (U+FFFD). Which is what most well-behaved software I've seen does with bad UTF-8. I work with lots of text processing, including processing old legacy files etc. Just refusing to go forward with bad input ("just fail fast") would work a lot less well. Can you imagine if you tried to open up something in UTF8 in a text editor, and it just refused to open it, telling you nothing but "Bad UTF8" or maybe "Bad UTF8 at byte 19734"? Not super helpful. But yeah, you want to TURN it into valid UTF8 as early in the pipeline as possible, while logging/flagging/notifying what was done. You definitely don't want to let the bad UTF8 just continue through the pipeline like that unless you really really have to, that way lies madness.
  - vardump 6y ago
    > Can you imagine if you tried to open up something in UTF8 in a text editor, and it just refused to open it, telling you nothing but "Bad UTF8" or maybe "Bad UTF8 at byte 19734"? Not super helpful. I think text editor is a good example where you need to be lenient. Furthermore, in that use case, any unnecessary modification of the data in any way is a bad idea. This includes replacing invalid code points with something else, unless they're edited by the user. The data might not even be UTF-8 in the first place.
- TwoBit 6y ago
  > Be liberal in what you accept, and conservative in what you send Terrible advice, with a long history proving why it's terrible, starting with IE6.
samatman 6y ago
The self-synchronizing aspect of utf-8 is one of the best things about the encoding, for sure. But there are a lot of reasons to validate utf-8. The article mentions security vulnerabilities, and as a langsec devotee, I approve of always parsing inputs to prevent that sort of thing. But even if you're reasonably confident that an invalid string isn't going to power a weird machine in your codebase (and are you, really?), validating is useful, because at that point you can trust algorithms that work on codepoints to actually work on codepoints, and don't have to constantly insert logic to detect malformation and re-synchronize your read, when all you want to be doing is working with your string. An even simpler reason: It's easy to write something to work on some utf-8, and be quite sure that, for your application, it's going to be fed valid utf-8 or 'close enough'. And then some wire gets crossed and you're feeding it an mp3 or something.
kwillets 6y ago
There have apparently been some injection attacks where an overlength ";" or something has been missed by a string sanitizer. Overlength means eg a 7-bit ASCII character is encoded in two or more bytes, neither of which would be noticed by an 8-bit delimiter checker. The only flag in this case is that bitfields in the two bytes have 0's in the high bits beyond 7.
rectang 6y ago
Routines that manipulate UTF-8 strings can produce unsafe results on invalid UTF-8. For example, finding the end of a string may place you into invalid memory if there are missing trailing bytes. To achieve safety, one option is to use routines that always assume that the data they are operating on may be invalid — but such routines are always slow, because they are essentially validating the UTF-8 data continually. Therefore, what most systems do is establish the validity of UTF-8 data as an invariant by running a sanity check once before supplying the data to unsafe-but-fast internal routines. And that's where validators like the one described in this article come in.
- Joker_vD 6y ago
  > finding the end of a string may place you into invalid memory if there are missing trailing bytes I struggle to imagine such a function. If its signature is "const char * utf8end(const char * s, size_t len)", then the implementation is obviously "return s + len". If the len is missing then, uh, you can't find the end unless there is a terminator character, in which case the implementation is "return s + strlen(s)". And if you mean finding the beginning of the last codepoint, then again, you simply go to the very last byte of the string and rewind back until you see a non-trailing byte (or you've run off the beginning of the string, or you've seen more than 4 bytes in which case you have an invalid string).
  - zamadatix 6y ago
    > If the len is missing then, uh, you can't find the end unless there is a terminator character, in which case the implementation is "return s + strlen(s)". Sure, and if you do strlen(s) without checking if there is actually a NUL first then: > finding the end of a string may place you into invalid memory if there are missing trailing bytes. As was originally said. This doesn't mean it's impossible to make such functions but <rest of original comment>.
    - labawi 6y ago
      Function finding end of a string that breaks on invalid utf8 would be silly and is a bad example. A function that counts the number of characters/codepoints or maybe decodes utf8 -> ucs4, may wish to operate on entire codepoints instead of always checking for potential end of buffer mid-decoding. Or perhaps avoid handling of invalid codepoints at all.
bloak 6y ago
I mostly agree. In many, perhaps most cases I'd rather not specify that the input to a function is UTF-8; I'd prefer to say it's an array of bytes that will usually be valid UTF-8. Even valid UTF-8 can turn out to be fairly unusable because it requires unimplemented features such as some corner case of bidirectional text, unexpected combinations of diacritics, or just plain characters that were added to the Unicode standard after your font was designed, so leaving validation till the point of rendering can be a good strategy, and it also means that your validation will be even more ridiculously fast in the case where the string never gets rendered at all.
josephg 6y ago
I think you’re confusing two concepts. There’s two meanings of “invalid” here - UTF-8 strings which contain invalid / unknown sequences, and strings which are packed in an invalid format. (Eg where the last byte is missing from a multi-byte code point). I agree that usually you should often allow and preserve unknown code point sequences, because rendering degrades gracefully and Unicode keeps changing. But the algorithm this article is talking about checks that a UTF8 string is ‘valid’ in the context of the raw byte encoding format. These checks are much more simple, and they’re needed in basically any software that interacts with strings.
kazinator 6y ago
I'm not convinced I should be coding my own UTF-8 validation in the year 2020, unless I'm working on the run-time of a programming language. I expect the I/O facility of a modern programming language to either reject bad UTF-8 or "quarantine" it somehow (map the bad bytes to a safe representation according to some documented rules).
umanwizard 6y ago
Regardless of whether it’s a good idea, there is enough inertia behind it by now that making it faster is a worthwhile project. In Rust and probably many other languages, if you want to work with non-utf8-validated text you’ll be fighting the ecosystem every step of the way.
smsm42 6y ago
I've seen serious bugs and even security issues due to invalid utf-8 and code just assuming it's valid (e.g. if there's half of the surrogate present the code just assumes the other half is there too). Display is not the only thing done with utf-8 text, and when parsing utf-8 strings being able to quickly validate utf-8 is a great capability.

vardump 6y ago

It always depends, but most of the time it's better to fail fast. Invalid UTF-8 is a good indication something went wrong. The earlier the failure occurs, the easier it'll be to troubleshoot it. Of course there are situations where you want to make do with your input and try to interpret it the best you can. Sometimes you need to be lenient in what you accept [0]. That's how I'd expect, say, a web browser to operate. Default to not accepting invalid input. Just fail fast. [0]: Be liberal in what you accept, and conservative in what you send - Jon Postel

jrochkind1 6y ago
> Of course there are situations where you want to make do with your input and try to interpret it the best you can. Sometimes you need to be lenient in what you accept The best way to do this is to TURN the input into valid UTF8 though. Like replacing any uninterpretable-as-utf8 bytes with the unicode replacement character (U+FFFD). Which is what most well-behaved software I've seen does with bad UTF-8. I work with lots of text processing, including processing old legacy files etc. Just refusing to go forward with bad input ("just fail fast") would work a lot less well. Can you imagine if you tried to open up something in UTF8 in a text editor, and it just refused to open it, telling you nothing but "Bad UTF8" or maybe "Bad UTF8 at byte 19734"? Not super helpful. But yeah, you want to TURN it into valid UTF8 as early in the pipeline as possible, while logging/flagging/notifying what was done. You definitely don't want to let the bad UTF8 just continue through the pipeline like that unless you really really have to, that way lies madness.
- vardump 6y ago
  > Can you imagine if you tried to open up something in UTF8 in a text editor, and it just refused to open it, telling you nothing but "Bad UTF8" or maybe "Bad UTF8 at byte 19734"? Not super helpful. I think text editor is a good example where you need to be lenient. Furthermore, in that use case, any unnecessary modification of the data in any way is a bad idea. This includes replacing invalid code points with something else, unless they're edited by the user. The data might not even be UTF-8 in the first place.
TwoBit 6y ago
> Be liberal in what you accept, and conservative in what you send Terrible advice, with a long history proving why it's terrible, starting with IE6.

samatman 6y ago

The self-synchronizing aspect of utf-8 is one of the best things about the encoding, for sure. But there are a lot of reasons to validate utf-8. The article mentions security vulnerabilities, and as a langsec devotee, I approve of always parsing inputs to prevent that sort of thing. But even if you're reasonably confident that an invalid string isn't going to power a weird machine in your codebase (and are you, really?), validating is useful, because at that point you can trust algorithms that work on codepoints to actually work on codepoints, and don't have to constantly insert logic to detect malformation and re-synchronize your read, when all you want to be doing is working with your string. An even simpler reason: It's easy to write something to work on some utf-8, and be quite sure that, for your application, it's going to be fed valid utf-8 or 'close enough'. And then some wire gets crossed and you're feeding it an mp3 or something.

kwillets 6y ago

There have apparently been some injection attacks where an overlength ";" or something has been missed by a string sanitizer. Overlength means eg a 7-bit ASCII character is encoded in two or more bytes, neither of which would be noticed by an 8-bit delimiter checker. The only flag in this case is that bitfields in the two bytes have 0's in the high bits beyond 7.

rectang 6y ago

Routines that manipulate UTF-8 strings can produce unsafe results on invalid UTF-8. For example, finding the end of a string may place you into invalid memory if there are missing trailing bytes. To achieve safety, one option is to use routines that always assume that the data they are operating on may be invalid — but such routines are always slow, because they are essentially validating the UTF-8 data continually. Therefore, what most systems do is establish the validity of UTF-8 data as an invariant by running a sanity check once before supplying the data to unsafe-but-fast internal routines. And that's where validators like the one described in this article come in.

Joker_vD 6y ago
> finding the end of a string may place you into invalid memory if there are missing trailing bytes I struggle to imagine such a function. If its signature is "const char * utf8end(const char * s, size_t len)", then the implementation is obviously "return s + len". If the len is missing then, uh, you can't find the end unless there is a terminator character, in which case the implementation is "return s + strlen(s)". And if you mean finding the beginning of the last codepoint, then again, you simply go to the very last byte of the string and rewind back until you see a non-trailing byte (or you've run off the beginning of the string, or you've seen more than 4 bytes in which case you have an invalid string).
- zamadatix 6y ago
  > If the len is missing then, uh, you can't find the end unless there is a terminator character, in which case the implementation is "return s + strlen(s)". Sure, and if you do strlen(s) without checking if there is actually a NUL first then: > finding the end of a string may place you into invalid memory if there are missing trailing bytes. As was originally said. This doesn't mean it's impossible to make such functions but <rest of original comment>.
  - labawi 6y ago
    Function finding end of a string that breaks on invalid utf8 would be silly and is a bad example. A function that counts the number of characters/codepoints or maybe decodes utf8 -> ucs4, may wish to operate on entire codepoints instead of always checking for potential end of buffer mid-decoding. Or perhaps avoid handling of invalid codepoints at all.

bloak 6y ago

I mostly agree. In many, perhaps most cases I'd rather not specify that the input to a function is UTF-8; I'd prefer to say it's an array of bytes that will usually be valid UTF-8. Even valid UTF-8 can turn out to be fairly unusable because it requires unimplemented features such as some corner case of bidirectional text, unexpected combinations of diacritics, or just plain characters that were added to the Unicode standard after your font was designed, so leaving validation till the point of rendering can be a good strategy, and it also means that your validation will be even more ridiculously fast in the case where the string never gets rendered at all.

josephg 6y ago

I think you’re confusing two concepts. There’s two meanings of “invalid” here - UTF-8 strings which contain invalid / unknown sequences, and strings which are packed in an invalid format. (Eg where the last byte is missing from a multi-byte code point). I agree that usually you should often allow and preserve unknown code point sequences, because rendering degrades gracefully and Unicode keeps changing. But the algorithm this article is talking about checks that a UTF8 string is ‘valid’ in the context of the raw byte encoding format. These checks are much more simple, and they’re needed in basically any software that interacts with strings.

kazinator 6y ago

I'm not convinced I should be coding my own UTF-8 validation in the year 2020, unless I'm working on the run-time of a programming language. I expect the I/O facility of a modern programming language to either reject bad UTF-8 or "quarantine" it somehow (map the bad bytes to a safe representation according to some documented rules).

umanwizard 6y ago

Regardless of whether it’s a good idea, there is enough inertia behind it by now that making it faster is a worthwhile project. In Rust and probably many other languages, if you want to work with non-utf8-validated text you’ll be fighting the ecosystem every step of the way.

smsm42 6y ago

I've seen serious bugs and even security issues due to invalid utf-8 and code just assuming it's valid (e.g. if there's half of the surrogate present the code just assumes the other half is there too). Display is not the only thing done with utf-8 text, and when parsing utf-8 strings being able to quickly validate utf-8 is a great capability.