4 ms·

There's not as big a niche as you might think. A lot of the software that needs valid UTF-8 don't validate upfront, but do it as part of a parsing process. An e

There's not as big a niche as you might think. A lot of the software that needs valid UTF-8 don't validate upfront, but do it as part of a parsing process. An example would be something like an XML or JSON parser. For more mundane uses like opening a big text file, you often want to tolerate invalid bytes and show a replacement character. You could use the technique, but you'd have to modify it to work for that usage.

pdkl95 7y ago
[re: upfront parsing - a PSA about validating input] > A lot of the software ... don't validate upfront They almost always should be validating all input up front. Deferring validation tends to become many different pieces of code all informally parsing fragments of the input that are needed locally. Without upfront validation of the complete unit of input, the resulting fragmented parsers are just a weird machine waiting to be programmed by a malicious attacker. For a much better explanation, I strongly recommend Meredith and Sergey's 28c3 talk[1] about The Science of Insecurity. > you often want to tolerate invalid bytes and show a replacement character While this validator wouldn't be useful in that situation, it is still important to validate the input upfront. When showing replacement characters, the "invalid bytes" that will be supported with a replacement character should be formally defined and added to the validation grammar, because they are no longer "invalid", but instead will be handled as a special case. Postel's robustness principle shouldn't be used as an excuse to skip validation. Being "liberal in what you accept" should still be well-defined and validated. [1] https://media.ccc.de/v/28c3-4763-en-the_science_of_insecurity https://media.ccc.de/v/28c3-4763-en-the_science_of_insecurit...
- nitwit005 7y ago
  You trimmed my quote in a way that completly changes the meaning, so I'm not even going to bother responding to that. How unicode replacement characters should work is already well defined. Just look it up.
zwegner 7y ago
In case anyone sees this: I initially agreed with you, thinking that pure UTF-8 validation is a bit more of a niche than might be expected. But if I'm not mistaken, I think at least for JSON validation, the UTF-8 validation can happen completely independently of the parsing. I don't think any control characters (braces, commas, quotes, escapes, etc) will change the validity of the UTF, or vice versa. So it might be beneficial (both for speed and security, as the sibling comment notes) to validate chunks of input as UTF-8, then pass them off to the parser, which now doesn't need to deal with validation. Maybe this could be applied to XML, but that's quite a behemoth of a standard--I really wouldn't be surprised if there was a way to switch encodings mid-stream. I have no idea though...
- nitwit005 7y ago
  Sure, you can validate and then parse the JSON, but parsing it is also effectively validating it as UTF-8, except for the string values in the JSON, so it's heavily going to be duplicated effort. And unfortunately, the strings can contain things like invalid surrogate pairs by using unicode escapes, so you may want to validate after escaping. You can't normally blindly parse XML as UTF-8. The encoding has rules for detecting the character set.

pdkl95 7y ago

[re: upfront parsing - a PSA about validating input] > A lot of the software ... don't validate upfront They almost always should be validating all input up front. Deferring validation tends to become many different pieces of code all informally parsing fragments of the input that are needed locally. Without upfront validation of the complete unit of input, the resulting fragmented parsers are just a weird machine waiting to be programmed by a malicious attacker. For a much better explanation, I strongly recommend Meredith and Sergey's 28c3 talk[1] about The Science of Insecurity. > you often want to tolerate invalid bytes and show a replacement character While this validator wouldn't be useful in that situation, it is still important to validate the input upfront. When showing replacement characters, the "invalid bytes" that will be supported with a replacement character should be formally defined and added to the validation grammar, because they are no longer "invalid", but instead will be handled as a special case. Postel's robustness principle shouldn't be used as an excuse to skip validation. Being "liberal in what you accept" should still be well-defined and validated. [1] https://media.ccc.de/v/28c3-4763-en-the_science_of_insecurity https://media.ccc.de/v/28c3-4763-en-the_science_of_insecurit...

nitwit005 7y ago
You trimmed my quote in a way that completly changes the meaning, so I'm not even going to bother responding to that. How unicode replacement characters should work is already well defined. Just look it up.

zwegner 7y ago

In case anyone sees this: I initially agreed with you, thinking that pure UTF-8 validation is a bit more of a niche than might be expected. But if I'm not mistaken, I think at least for JSON validation, the UTF-8 validation can happen completely independently of the parsing. I don't think any control characters (braces, commas, quotes, escapes, etc) will change the validity of the UTF, or vice versa. So it might be beneficial (both for speed and security, as the sibling comment notes) to validate chunks of input as UTF-8, then pass them off to the parser, which now doesn't need to deal with validation. Maybe this could be applied to XML, but that's quite a behemoth of a standard--I really wouldn't be surprised if there was a way to switch encodings mid-stream. I have no idea though...

nitwit005 7y ago
Sure, you can validate and then parse the JSON, but parsing it is also effectively validating it as UTF-8, except for the string values in the JSON, so it's heavily going to be duplicated effort. And unfortunately, the strings can contain things like invalid surrogate pairs by using unicode escapes, so you may want to validate after escaping. You can't normally blindly parse XML as UTF-8. The encoding has rules for detecting the character set.