5 ms·

Compare with https://lexi-lambda.github.io/blog/2019/11/05/parse-don-t-validate/ https://lexi-lambda.github.io/blog/2019/11/05/parse-don-t-va...

by wnoise 5y ago

dnautics 5y ago
I think the domain models are different. "parse-don't-validate" is great when your users are internal and trusted (e.g. a library that does codegen - the operators of the parser are already in the codebase). When your users are potentially hostile, you should at some level have a separate validate and eject strategy.
- mjw1007 5y ago
  I think "parse, don't validate" is an improvement on what the author of this article recommends: « So in cases where you do need to “echo” raw user input, carefully filter input based on a restrictive whitelist, and store the result in the database. When you come to output it, output it as stored without escaping. » I think the "parse, don't validate" approach comes out as follows: - take the list of things you would have included on your whitelist - add nodes for them to your internal representation for parsed markdown - extend your markdown parser to convert html-like input into those nodes - implement output for those nodes in a similar way to normal markdown This way, given the "escape output" they recommend, it's harder for any variant of the input that you hadn't considered to have harmful effects.
  - yakshaving_jgt 5y ago
    No, the parse, don’t validate idea is completely unrelated. It’s about leveraging a type system like in Haskell. It’s about parsing a value into a type with a narrower domain which in turn minimises the amount of control flow needed to implement a sufficiently correct program.
    - mjw1007 5y ago
      I agree with "It’s about parsing a value into a type with a narrower domain", but I don't see how you get to the first sentence. In their example of a markdown renderer, the internal-representation node is the type with the narrower domain.
- yakshaving_jgt 5y ago
  The idea of parsing over validation is just as applicable with untrusted input as with trusted input. The idea is more about system design rather than the prevention of security vulnerabilities.
  - dnautics 5y ago
    come again? I didn't say you can't validate while parsing for untrusted input, I said for untrusted input you will probably STILL need additional separate validation methods. Key emphasis on negating don't absolute imperative in the original aphorism.
    - yakshaving_jgt 5y ago
      I didn't say that that's what you said. I tried to communicate that whether or not the input is trusted is besides the point. Unless I'm deeply confused, the idea in parse, don't validate is about doing something like this parseFoo :: Text -> Maybe Foo parseFoo t = if textIsAFoo t then Just (Foo t) else Nothing f :: Foo -> IO () f = _ Rather than something like this (which seems to be more common) f :: Text -> IO () f t = when (textIsAFoo t) g where g = _
      - dnautics 5y ago
        I thought the idea of parse, don't validate is that your parser should contain validation logic. So instead of text -> generic json parser -> validate json -> ... you would do text -> custom json parser that stops if encounters "incorrect content" -> ...
        yakshaving_jgt 5y ago
        You can think of the parser as containing validation logic — it can parse the value into a more constrained type if it conforms to validation rules, or it can fail. The point is that once your value is in a more principled type, the rest of the system is free from having to make assumptions (and guard against potential failures) about the breadth of that value's domain. As the article mentions, this is really only relevant to languages with proper type systems like Haskell.
marcosdumay 5y ago
Two different advises for two different things. That one is about data validation, making sure it is coherent and fits your data quality rules. This one is about data encoding, making sure it fits a different system's rules.
kevincox 5y ago
These are both good advise. I have seen really funny bugs where Java accepted non-ascii numbers in an IP address but the C++ control plane very much did not. If the re-serialized version was sent to the backend this wouldn't have been an issue. But the domains are different. Data validation is ensuring that the information is something that your system accepts. Data encoding is used when you are serializing information. You should very likely validate on input, but not "sanitize" or encode. You do your encoding on output.

Compare with https://lexi-lambda.github.io/blog/2019/11/05/parse-don-t-validate/ https://lexi-lambda.github.io/blog/2019/11/05/parse-don-t-va...

by wnoise 5y ago

Compare with https://lexi-lambda.github.io/blog/2019/11/05/parse-don-t-validate/ https://lexi-lambda.github.io/blog/2019/11/05/parse-don-t-va...

dnautics 5y ago

I think the domain models are different. "parse-don't-validate" is great when your users are internal and trusted (e.g. a library that does codegen - the operators of the parser are already in the codebase). When your users are potentially hostile, you should at some level have a separate validate and eject strategy.

mjw1007 5y ago
I think "parse, don't validate" is an improvement on what the author of this article recommends: « So in cases where you do need to “echo” raw user input, carefully filter input based on a restrictive whitelist, and store the result in the database. When you come to output it, output it as stored without escaping. » I think the "parse, don't validate" approach comes out as follows: - take the list of things you would have included on your whitelist - add nodes for them to your internal representation for parsed markdown - extend your markdown parser to convert html-like input into those nodes - implement output for those nodes in a similar way to normal markdown This way, given the "escape output" they recommend, it's harder for any variant of the input that you hadn't considered to have harmful effects.
- yakshaving_jgt 5y ago
  No, the parse, don’t validate idea is completely unrelated. It’s about leveraging a type system like in Haskell. It’s about parsing a value into a type with a narrower domain which in turn minimises the amount of control flow needed to implement a sufficiently correct program.
  - mjw1007 5y ago
    I agree with "It’s about parsing a value into a type with a narrower domain", but I don't see how you get to the first sentence. In their example of a markdown renderer, the internal-representation node is the type with the narrower domain.
yakshaving_jgt 5y ago
The idea of parsing over validation is just as applicable with untrusted input as with trusted input. The idea is more about system design rather than the prevention of security vulnerabilities.
- dnautics 5y ago
  come again? I didn't say you can't validate while parsing for untrusted input, I said for untrusted input you will probably STILL need additional separate validation methods. Key emphasis on negating don't absolute imperative in the original aphorism.
  - yakshaving_jgt 5y ago
    I didn't say that that's what you said. I tried to communicate that whether or not the input is trusted is besides the point. Unless I'm deeply confused, the idea in parse, don't validate is about doing something like this parseFoo :: Text -> Maybe Foo parseFoo t = if textIsAFoo t then Just (Foo t) else Nothing f :: Foo -> IO () f = _ Rather than something like this (which seems to be more common) f :: Text -> IO () f t = when (textIsAFoo t) g where g = _
    - dnautics 5y ago
      I thought the idea of parse, don't validate is that your parser should contain validation logic. So instead of text -> generic json parser -> validate json -> ... you would do text -> custom json parser that stops if encounters "incorrect content" -> ...
      - yakshaving_jgt 5y ago
        You can think of the parser as containing validation logic — it can parse the value into a more constrained type if it conforms to validation rules, or it can fail. The point is that once your value is in a more principled type, the rest of the system is free from having to make assumptions (and guard against potential failures) about the breadth of that value's domain. As the article mentions, this is really only relevant to languages with proper type systems like Haskell.

marcosdumay 5y ago

Two different advises for two different things. That one is about data validation, making sure it is coherent and fits your data quality rules. This one is about data encoding, making sure it fits a different system's rules.

kevincox 5y ago

These are both good advise. I have seen really funny bugs where Java accepted non-ascii numbers in an IP address but the C++ control plane very much did not. If the re-serialized version was sent to the backend this wouldn't have been an issue. But the domains are different. Data validation is ensuring that the information is something that your system accepts. Data encoding is used when you are serializing information. You should very likely validate on input, but not "sanitize" or encode. You do your encoding on output.