3 ms·

Why do we need to check utf-8 validity fast? If you have invalid utf-8, go fix whatever code it is that is producing invalid utf-8...

PhilipRoman 3y ago
Reminds me of a developer who insisted that his service crashing when someone entered an address longer than 1024 characters in a form was no big deal since "no one has an address that long"
Limeray 3y ago
Man i would love to live in your perfect world where you never have to interact with 3rd party data.
- kevincox 3y ago
  Let alone malicious third parties.
kitd 3y ago
How do you know if you have invalid utf8?
- ape4 3y ago
  From the Wikipedia article... Not all sequences of bytes are valid UTF-8. A UTF-8 decoder should be prepared for: invalid bytes an unexpected continuation byte a non-continuation byte before the end of the character the string ending before the end of the character (which can happen in simple string truncation) an overlong encoding a sequence that decodes to an invalid code point https://en.wikipedia.org/wiki/UTF-8 https://en.wikipedia.org/wiki/UTF-8
  - jraph 3y ago
    I think the question of your parent commenter is rather "how do you know if you have invalid UTF-8 if you don't check?"
    - Dylan16807 3y ago
      OP is arguing against checking as a matter of course, on the hot path, not saying that you shouldn't ever test ever. That particular counterargument is a bad one.
      - jraph 3y ago
        Clearly, that's not what they wrote. They told people to fix their code that generate broken utf-8 instead, implying that validating fast is pointless since you don't need to validate at all. I don't understand how you came up with this interpretation. And I'm also not sure what you call out as a bad counterargument and how you would do what you suggest. And even if you don't validate on the hot path, it doesn't hurt to do it faster, so I don't understand why one would be against a more efficient computation. But that's not what I'm answering to, and I was not taking sides so I don't know why you would reply to my comment with this. I was just clarifying something. I'm answering a comment that might have misread what they answered too, that's it.
        Dylan16807 3y ago
        I replied to you because you made the implication much more direct in your wording, but the implication is wrong. I'm not saying it's your opinion, you just made it easier to reply to. OP was saying not to validate as part of your normal pipeline. They weren't saying not to validate ever, such as while poking around or testing, which is how you could still figure out that bad utf-8 exists and where it is coming from. > And even if you don't validate on the hot path, it doesn't hurt to do it faster, so I don't understand why one would be against a more efficient computation. It's not about being for or against the speed of calculation. It's that removing unnecessary calculations is even faster. So questioning the need is good. If 'hot path' was unclear and you think I'm making things up, let me try a different wording. As a hypothetical: If I don't think something needs to be validated in production, I might question a library dedicated to doing it really fast, and say your producers should be fixed instead. Me saying that doesn't mean I think you should be stumbling around blind, never testing those producers.
stephc_int13 3y ago
This is about input validation, something you can't trust to be valid. And obviously you don't want to spend more time doing this than necessary.
jeffbee 3y ago
There are various contexts where the inputs are defined to be UTF-8 and nonconforming inputs are rejected. For example, a `string` field in a protocol buffer.
- geraldcombs 3y ago
  This could be generalized to any `string` field in any network packet.
  - jeffbee 3y ago
    I don't see how. A user of a conforming protobuf library can assume, after a successful decode, that a string field contains valid UTF-8. That doesn't generalize to everything called "string".
    - geraldcombs 3y ago
      I was trying to make the point that string validation isn't limited to protobuf. You need to do it for any network protocol that uses strings, which is most of them.
layer8 3y ago
Your question is valid, but your subsequent statement is silly and dangerous.

PhilipRoman 3y ago

Reminds me of a developer who insisted that his service crashing when someone entered an address longer than 1024 characters in a form was no big deal since "no one has an address that long"

Limeray 3y ago

Man i would love to live in your perfect world where you never have to interact with 3rd party data.

kevincox 3y ago
Let alone malicious third parties.

kitd 3y ago

How do you know if you have invalid utf8?

ape4 3y ago
From the Wikipedia article... Not all sequences of bytes are valid UTF-8. A UTF-8 decoder should be prepared for: invalid bytes an unexpected continuation byte a non-continuation byte before the end of the character the string ending before the end of the character (which can happen in simple string truncation) an overlong encoding a sequence that decodes to an invalid code point https://en.wikipedia.org/wiki/UTF-8 https://en.wikipedia.org/wiki/UTF-8
- jraph 3y ago
  I think the question of your parent commenter is rather "how do you know if you have invalid UTF-8 if you don't check?"
  - Dylan16807 3y ago
    OP is arguing against checking as a matter of course, on the hot path, not saying that you shouldn't ever test ever. That particular counterargument is a bad one.
    - jraph 3y ago
      Clearly, that's not what they wrote. They told people to fix their code that generate broken utf-8 instead, implying that validating fast is pointless since you don't need to validate at all. I don't understand how you came up with this interpretation. And I'm also not sure what you call out as a bad counterargument and how you would do what you suggest. And even if you don't validate on the hot path, it doesn't hurt to do it faster, so I don't understand why one would be against a more efficient computation. But that's not what I'm answering to, and I was not taking sides so I don't know why you would reply to my comment with this. I was just clarifying something. I'm answering a comment that might have misread what they answered too, that's it.
      - Dylan16807 3y ago
        I replied to you because you made the implication much more direct in your wording, but the implication is wrong. I'm not saying it's your opinion, you just made it easier to reply to. OP was saying not to validate as part of your normal pipeline. They weren't saying not to validate ever, such as while poking around or testing, which is how you could still figure out that bad utf-8 exists and where it is coming from. > And even if you don't validate on the hot path, it doesn't hurt to do it faster, so I don't understand why one would be against a more efficient computation. It's not about being for or against the speed of calculation. It's that removing unnecessary calculations is even faster. So questioning the need is good. If 'hot path' was unclear and you think I'm making things up, let me try a different wording. As a hypothetical: If I don't think something needs to be validated in production, I might question a library dedicated to doing it really fast, and say your producers should be fixed instead. Me saying that doesn't mean I think you should be stumbling around blind, never testing those producers.

stephc_int13 3y ago

This is about input validation, something you can't trust to be valid. And obviously you don't want to spend more time doing this than necessary.

jeffbee 3y ago

There are various contexts where the inputs are defined to be UTF-8 and nonconforming inputs are rejected. For example, a `string` field in a protocol buffer.

geraldcombs 3y ago
This could be generalized to any `string` field in any network packet.
- jeffbee 3y ago
  I don't see how. A user of a conforming protobuf library can assume, after a successful decode, that a string field contains valid UTF-8. That doesn't generalize to everything called "string".
  - geraldcombs 3y ago
    I was trying to make the point that string validation isn't limited to protobuf. You need to do it for any network protocol that uses strings, which is most of them.

layer8 3y ago

Your question is valid, but your subsequent statement is silly and dangerous.