5 ms·

I'm not sure the post (from 2022) is/was correct. I've looked into it too, and I expect this was reachable by the existing x509 fuzzer. There's a fallacy in a

I'm not sure the post (from 2022) is/was correct. I've looked into it too, and I expect this was reachable by the existing x509 fuzzer. There's a fallacy in assuming that a fuzzer will solve for all reachable code paths in a reasonable time, and that if it doesn't then there must be a problem with the harness. The harness is a reasonable top-level x509 parsing harness, but starting all the way from the network input makes solving those deep constraints unlikely to happen by (feedback-driven) chance, which is what I think happened here. Of course, a harness that started from the punycode parsing -- instead of the top-level X509 parsing -- finds this vulnerability immediately.

hedora 2y ago
I've found that writing randomized unit tests for each small part of a system finds this sort of stuff immediately. In this case, a test that just generated 1,000,000 random strings and passed them to punycode would probably have found the problem (maybe not in the first run, but after a week or two in continuous integration). I try to structure the tests so they can run with dumb random input or with coverage-guided input from a fuzzer. The former usually finds >90% of the bugs the fuzzer would, and does so 100x faster, so it's better than fuzzing during development, but worse during nightly testing. One other advantage for dumb random input is that it works with distributed systems and things written in multiple languages (where coverage information isn't readily available).
- mattgreenrocks 2y ago
  I really like this idea because it avoids the issue of fuzzers needing to burn tons of CPU just to get down to the actual domain logic, which can have really thorny bugs. Per usual, the idea of "unit" gets contentious quickly, but with appropriate tooling I could foresee adding annotations to code that leverage random input, property-based testing, and a user-defined dictionary of known weird inputs.
- paulddraper 2y ago
  > maybe not in the first run, but after a week or two in continuous integration You'd use a different seed for each CI run?? That sounds like a nightmare of non-determinism, and lost of trust in CI system in general.
  - forty 2y ago
    Not if you log the seed of the failing runs
    - hedora 2y ago
      Yep; I definitely log the seed and re-seed every few seconds. Most software I work on these days is non-deterministic anyway (it involves the network, etc.), so CI is fundamentally going to fail some runs and not others. Even stuff like deterministic simulation has this property: Those test suites rely on having a large number of randomly generated schedules, so there's always a chance that running the test one more time (with a new seed) will find a new bug.
      - jhardy54 2y ago
        If you’re using Git, I’d strongly recommend the hash of the current tree (not the commit). That way your tests are deterministic based on the contents of your tree. For example, if you add a commit and then revert, you’ll end up with the same test seed as if you hadn’t committed.
        hedora 2y ago
        The important thing to log is the seed passed to the rng. (It’s usually wallclock time measured at nanosecond granularity.) In a typical night for a production quality system, 100-1000+ hours of such tests would run, all with different seeds, and in diverse configurations, etc, so the seed isn’t derived from the git sha, or the source code.
    - paulddraper 2y ago
      Or someone just punches retry
      - hedora 2y ago
        Ignoring test failures is a different issue. It can range from known bugs and rational tradeoffs to a hiring/hr issue. Establishing a healthy dev/engineering culture is hard.
- Arnavion 2y ago
  aka property testing.
cryptonector 2y ago
> Of course, a harness that started from the punycode parsing -- instead of the top-level X509 parsing -- finds this vulnerability immediately. Yes, it's nice to have a fuzzer that can start from a very high-level entry point, but it's even nicer to fuzz lots of entry points (one for every parser).
capitainenemo 2y ago
If finding it required generating a valid signed certificate from random string generation that seems essentially equivalent to "unreachable" to me....
- gravescale 2y ago
  Not necessarily because the fuzzer has visibility into the execution pathways triggered, so it can "feel" its way through the maze. For example, say any certificate that starts with 0x00 is invalid (and there's an up-front test and return). Once the fuzzer has tried 0x00 and concluded that the pathway is a dead end, it won't try again, even though there are squillions of data blocks beginning 0x00. It's not a highly efficient way to generate a valid certificate (especially when you have to navigate the network code as well), but it's also not just rolling 'n' 256-sided dice until the end of the universe.
  - cjbprime 2y ago
    I think the parent is pointing out that if the signature has to be both syntactically and cryptographically valid, then this would defeat the fuzzer for obvious reasons. But I don't think it does, for this vulnerability. The signature is checked last.
    - capitainenemo 2y ago
      Yep. That was what I was thinking. Didn't realise the signature was checked later. Thanks for the clarification.

hedora 2y ago

I've found that writing randomized unit tests for each small part of a system finds this sort of stuff immediately. In this case, a test that just generated 1,000,000 random strings and passed them to punycode would probably have found the problem (maybe not in the first run, but after a week or two in continuous integration). I try to structure the tests so they can run with dumb random input or with coverage-guided input from a fuzzer. The former usually finds >90% of the bugs the fuzzer would, and does so 100x faster, so it's better than fuzzing during development, but worse during nightly testing. One other advantage for dumb random input is that it works with distributed systems and things written in multiple languages (where coverage information isn't readily available).

mattgreenrocks 2y ago
I really like this idea because it avoids the issue of fuzzers needing to burn tons of CPU just to get down to the actual domain logic, which can have really thorny bugs. Per usual, the idea of "unit" gets contentious quickly, but with appropriate tooling I could foresee adding annotations to code that leverage random input, property-based testing, and a user-defined dictionary of known weird inputs.
paulddraper 2y ago
> maybe not in the first run, but after a week or two in continuous integration You'd use a different seed for each CI run?? That sounds like a nightmare of non-determinism, and lost of trust in CI system in general.
- forty 2y ago
  Not if you log the seed of the failing runs
  - hedora 2y ago
    Yep; I definitely log the seed and re-seed every few seconds. Most software I work on these days is non-deterministic anyway (it involves the network, etc.), so CI is fundamentally going to fail some runs and not others. Even stuff like deterministic simulation has this property: Those test suites rely on having a large number of randomly generated schedules, so there's always a chance that running the test one more time (with a new seed) will find a new bug.
    - jhardy54 2y ago
      If you’re using Git, I’d strongly recommend the hash of the current tree (not the commit). That way your tests are deterministic based on the contents of your tree. For example, if you add a commit and then revert, you’ll end up with the same test seed as if you hadn’t committed.
      - hedora 2y ago
        The important thing to log is the seed passed to the rng. (It’s usually wallclock time measured at nanosecond granularity.) In a typical night for a production quality system, 100-1000+ hours of such tests would run, all with different seeds, and in diverse configurations, etc, so the seed isn’t derived from the git sha, or the source code.
  - paulddraper 2y ago
    Or someone just punches retry
    - hedora 2y ago
      Ignoring test failures is a different issue. It can range from known bugs and rational tradeoffs to a hiring/hr issue. Establishing a healthy dev/engineering culture is hard.
Arnavion 2y ago
aka property testing.

cryptonector 2y ago

> Of course, a harness that started from the punycode parsing -- instead of the top-level X509 parsing -- finds this vulnerability immediately. Yes, it's nice to have a fuzzer that can start from a very high-level entry point, but it's even nicer to fuzz lots of entry points (one for every parser).

capitainenemo 2y ago

If finding it required generating a valid signed certificate from random string generation that seems essentially equivalent to "unreachable" to me....

gravescale 2y ago
Not necessarily because the fuzzer has visibility into the execution pathways triggered, so it can "feel" its way through the maze. For example, say any certificate that starts with 0x00 is invalid (and there's an up-front test and return). Once the fuzzer has tried 0x00 and concluded that the pathway is a dead end, it won't try again, even though there are squillions of data blocks beginning 0x00. It's not a highly efficient way to generate a valid certificate (especially when you have to navigate the network code as well), but it's also not just rolling 'n' 256-sided dice until the end of the universe.
- cjbprime 2y ago
  I think the parent is pointing out that if the signature has to be both syntactically and cryptographically valid, then this would defeat the fuzzer for obvious reasons. But I don't think it does, for this vulnerability. The signature is checked last.
  - capitainenemo 2y ago
    Yep. That was what I was thinking. Didn't realise the signature was checked later. Thanks for the clarification.