4 ms·

I think the question of your parent commenter is rather "how do you know if you have invalid UTF-8 if you don't check?"

by jraph 3y ago

Dylan16807 3y ago
OP is arguing against checking as a matter of course, on the hot path, not saying that you shouldn't ever test ever. That particular counterargument is a bad one.
- jraph 3y ago
  Clearly, that's not what they wrote. They told people to fix their code that generate broken utf-8 instead, implying that validating fast is pointless since you don't need to validate at all. I don't understand how you came up with this interpretation. And I'm also not sure what you call out as a bad counterargument and how you would do what you suggest. And even if you don't validate on the hot path, it doesn't hurt to do it faster, so I don't understand why one would be against a more efficient computation. But that's not what I'm answering to, and I was not taking sides so I don't know why you would reply to my comment with this. I was just clarifying something. I'm answering a comment that might have misread what they answered too, that's it.
  - Dylan16807 3y ago
    I replied to you because you made the implication much more direct in your wording, but the implication is wrong. I'm not saying it's your opinion, you just made it easier to reply to. OP was saying not to validate as part of your normal pipeline. They weren't saying not to validate ever, such as while poking around or testing, which is how you could still figure out that bad utf-8 exists and where it is coming from. > And even if you don't validate on the hot path, it doesn't hurt to do it faster, so I don't understand why one would be against a more efficient computation. It's not about being for or against the speed of calculation. It's that removing unnecessary calculations is even faster. So questioning the need is good. If 'hot path' was unclear and you think I'm making things up, let me try a different wording. As a hypothetical: If I don't think something needs to be validated in production, I might question a library dedicated to doing it really fast, and say your producers should be fixed instead. Me saying that doesn't mean I think you should be stumbling around blind, never testing those producers.
    - ncruces 3y ago
      You don't think JSON should be validated in production? Presumably you think you should always be able to trust anyone who sends you JSON? I mean Go already does a bunch of UTF-8 validations in its JSON encoder/decoder [1]. No need to make those faster. [1] https://pkg.go.dev/encoding/json#Marshal https://pkg.go.dev/encoding/json#Marshal
      - Dylan16807 3y ago
        Hi, what? I said hypothetical, and I did not use the word JSON.
        ncruces 3y ago
        Both JSON encoding and decoding require UTF-8 validation. If there's a workload that makes sense to optimize because it's used in production, it's UTF-8 validation. So why are we even arguing whether UTF-8 validation is necessary/useful in the "hot path" or in "production"?
        Dylan16807 3y ago
        So you asked what I thought, but I never gave an opinion on UTF-8, I gave a generic hypothetical. If you're asking about OP's idea, then I agree that validating the UTF-8 inside of JSON is needed sometimes, though it's not required in all situations. But I wasn't commenting on that, I was commenting on the specific objection of "how do you know if you don't check?", because I don't think that specific objection is very strong.
    - jraph 3y ago
      > you just made it easier to reply to Ah, makes sense. Thanks! Well, the implication is not wrong. If you validate only during testing, you never validate some inputs, the ones that matter: user input. I now see what you mean by hot path, I was imagining something like validating asynchronously, later, when not serving a request for instance.

I think the question of your parent commenter is rather "how do you know if you have invalid UTF-8 if you don't check?"

by jraph 3y ago

I think the question of your parent commenter is rather "how do you know if you have invalid UTF-8 if you don't check?"

OP is arguing against checking as a matter of course, on the hot path, not saying that you shouldn't ever test ever. That particular counterargument is a bad one.

jraph 3y ago
Clearly, that's not what they wrote. They told people to fix their code that generate broken utf-8 instead, implying that validating fast is pointless since you don't need to validate at all. I don't understand how you came up with this interpretation. And I'm also not sure what you call out as a bad counterargument and how you would do what you suggest. And even if you don't validate on the hot path, it doesn't hurt to do it faster, so I don't understand why one would be against a more efficient computation. But that's not what I'm answering to, and I was not taking sides so I don't know why you would reply to my comment with this. I was just clarifying something. I'm answering a comment that might have misread what they answered too, that's it.
- Dylan16807 3y ago
  I replied to you because you made the implication much more direct in your wording, but the implication is wrong. I'm not saying it's your opinion, you just made it easier to reply to. OP was saying not to validate as part of your normal pipeline. They weren't saying not to validate ever, such as while poking around or testing, which is how you could still figure out that bad utf-8 exists and where it is coming from. > And even if you don't validate on the hot path, it doesn't hurt to do it faster, so I don't understand why one would be against a more efficient computation. It's not about being for or against the speed of calculation. It's that removing unnecessary calculations is even faster. So questioning the need is good. If 'hot path' was unclear and you think I'm making things up, let me try a different wording. As a hypothetical: If I don't think something needs to be validated in production, I might question a library dedicated to doing it really fast, and say your producers should be fixed instead. Me saying that doesn't mean I think you should be stumbling around blind, never testing those producers.
  - ncruces 3y ago
    You don't think JSON should be validated in production? Presumably you think you should always be able to trust anyone who sends you JSON? I mean Go already does a bunch of UTF-8 validations in its JSON encoder/decoder [1]. No need to make those faster. [1] https://pkg.go.dev/encoding/json#Marshal https://pkg.go.dev/encoding/json#Marshal
    - Dylan16807 3y ago
      Hi, what? I said hypothetical, and I did not use the word JSON.
      - ncruces 3y ago
        Both JSON encoding and decoding require UTF-8 validation. If there's a workload that makes sense to optimize because it's used in production, it's UTF-8 validation. So why are we even arguing whether UTF-8 validation is necessary/useful in the "hot path" or in "production"?
        Dylan16807 3y ago
        So you asked what I thought, but I never gave an opinion on UTF-8, I gave a generic hypothetical. If you're asking about OP's idea, then I agree that validating the UTF-8 inside of JSON is needed sometimes, though it's not required in all situations. But I wasn't commenting on that, I was commenting on the specific objection of "how do you know if you don't check?", because I don't think that specific objection is very strong.
  - jraph 3y ago
    > you just made it easier to reply to Ah, makes sense. Thanks! Well, the implication is not wrong. If you validate only during testing, you never validate some inputs, the ones that matter: user input. I now see what you mean by hot path, I was imagining something like validating asynchronously, later, when not serving a request for instance.