4 ms·

It always depends, but most of the time it's better to fail fast. Invalid UTF-8 is a good indication something went wrong. The earlier the failure occurs, the e

It always depends, but most of the time it's better to fail fast. Invalid UTF-8 is a good indication something went wrong. The earlier the failure occurs, the easier it'll be to troubleshoot it. Of course there are situations where you want to make do with your input and try to interpret it the best you can. Sometimes you need to be lenient in what you accept [0]. That's how I'd expect, say, a web browser to operate. Default to not accepting invalid input. Just fail fast. [0]: Be liberal in what you accept, and conservative in what you send - Jon Postel

jrochkind1 6y ago
> Of course there are situations where you want to make do with your input and try to interpret it the best you can. Sometimes you need to be lenient in what you accept The best way to do this is to TURN the input into valid UTF8 though. Like replacing any uninterpretable-as-utf8 bytes with the unicode replacement character (U+FFFD). Which is what most well-behaved software I've seen does with bad UTF-8. I work with lots of text processing, including processing old legacy files etc. Just refusing to go forward with bad input ("just fail fast") would work a lot less well. Can you imagine if you tried to open up something in UTF8 in a text editor, and it just refused to open it, telling you nothing but "Bad UTF8" or maybe "Bad UTF8 at byte 19734"? Not super helpful. But yeah, you want to TURN it into valid UTF8 as early in the pipeline as possible, while logging/flagging/notifying what was done. You definitely don't want to let the bad UTF8 just continue through the pipeline like that unless you really really have to, that way lies madness.
- vardump 6y ago
  > Can you imagine if you tried to open up something in UTF8 in a text editor, and it just refused to open it, telling you nothing but "Bad UTF8" or maybe "Bad UTF8 at byte 19734"? Not super helpful. I think text editor is a good example where you need to be lenient. Furthermore, in that use case, any unnecessary modification of the data in any way is a bad idea. This includes replacing invalid code points with something else, unless they're edited by the user. The data might not even be UTF-8 in the first place.
TwoBit 6y ago
> Be liberal in what you accept, and conservative in what you send Terrible advice, with a long history proving why it's terrible, starting with IE6.

jrochkind1 6y ago

> Of course there are situations where you want to make do with your input and try to interpret it the best you can. Sometimes you need to be lenient in what you accept The best way to do this is to TURN the input into valid UTF8 though. Like replacing any uninterpretable-as-utf8 bytes with the unicode replacement character (U+FFFD). Which is what most well-behaved software I've seen does with bad UTF-8. I work with lots of text processing, including processing old legacy files etc. Just refusing to go forward with bad input ("just fail fast") would work a lot less well. Can you imagine if you tried to open up something in UTF8 in a text editor, and it just refused to open it, telling you nothing but "Bad UTF8" or maybe "Bad UTF8 at byte 19734"? Not super helpful. But yeah, you want to TURN it into valid UTF8 as early in the pipeline as possible, while logging/flagging/notifying what was done. You definitely don't want to let the bad UTF8 just continue through the pipeline like that unless you really really have to, that way lies madness.

vardump 6y ago
> Can you imagine if you tried to open up something in UTF8 in a text editor, and it just refused to open it, telling you nothing but "Bad UTF8" or maybe "Bad UTF8 at byte 19734"? Not super helpful. I think text editor is a good example where you need to be lenient. Furthermore, in that use case, any unnecessary modification of the data in any way is a bad idea. This includes replacing invalid code points with something else, unless they're edited by the user. The data might not even be UTF-8 in the first place.

TwoBit 6y ago

> Be liberal in what you accept, and conservative in what you send Terrible advice, with a long history proving why it's terrible, starting with IE6.