4 ms·
tl;dr: because ossl_a2ulabel had no unit tests until a few days ago, the fuzzer could not have reached it through any combination of other tests. That fuzzing
by jeffbee 4y ago
tl;dr: because ossl_a2ulabel had no unit tests until a few days ago, the fuzzer could not have reached it through any combination of other tests.
That fuzzing is tricky was not the problem here. The problem is the culture that allowed ossl_a2ulabel to exist without unit tests. And before some weird nerd jumps in to say that openssl is so old we can't apply modern standards of project health, please note that the vulnerable function was committed from scratch in August 2020. Without unit tests.
- MuffinFlavored 4y ago> because ossl_a2ulabel had no unit tests until a few days ago it's not realistic to enforce unit test coverage % with a project at the scale of OpenSSL, right?
- pca006132 4y agoIt should be realistic, line coverage isn't really that hard. The hard thing is that high line coverage alone is usually not enough for numerical stuff...
- jeffbee 4y agoIt is trivial to enforce that new functions have new unit tests and fuzz tests. You are the reviewer of https://github.com/openssl/openssl/pull/9654 https://github.com/openssl/openssl/pull/9654 and you just say "Please add unit tests and fuzz tests for foo and bar" and you don't approve it. I don't know what the deal is with their testing culture but in year 27 of the project they demonstrably haven't learned this lesson. It's nice that they added integration tests (testing given encoded certs) but as the article points out that was insufficient.
- dralley 4y agoIMO, one of the biggest benefits of "modern" systems languages like Rust, D, Zig is how much easier they make writing and running tests compared to C and C++. Yes, you can write tests for those languages, but it's nowhere near as trivial. And that makes a difference.
- fisf 4y agoI am not buying it. Writing unit tests for c/c++ is trivial. There are perfectly fine test frameworks, used by developers every day, integrated in any major IDE or runnable as one-liner from the command line. This is absolutely a cultural problem.
- edgyquant 4y agoIn modern languages you either have built in testing or it’s a single package install away. It is nowhere near this simple with C or C++
- pjmlp 4y agoI was writing unit tests for C in 1996, naturally we still haven't coined the term, so we just called them automatic tests. It was part of our data structures and algorithms project, failure to execute the automatic tests meant no admission to the final exam. We had three sets of tests, those provided initially at the begin of the term, those that we were expected to write ourselves, and a surprise set on the integration week at the end of the semester.
- bluGill 4y agoLast week in a code review I got a "please add unit tests for this code" comment. The person who wrote that comment wasn't aware that this was a refactoring where the functionality was well tested. There is no substitute for reviewers who really understand the code in question. The problem is they are the ones writing the code and so are biased and not able to give a good review.
- masklinn 4y ago> it's not realistic to enforce unit test coverage % with a project at the scale of OpenSSL, right? Why not? You can enforce that all new files should be covered (at the very least line-covered). It requires some setup effort (collecting code coverage and either sending it to a tool which perform the correlation or correlating yourself), but once that's done... it does its thing. Then you can work on increasing coverage for existing files, and ratcheting requirements.
- stefan_ 4y agoNo one is asking for that. But this is code that did one thing: punycode decoding, with millions of well documented test vectors. The code had zero dependencies on anything OpenSSL related. It is a very simple "text in, text out" problem, the most trivial thing to write unit tests for. At the same time, it's code that parses externally provided buffers and has to deal with things like unicode in C - there should be a massive red flashing warning light in every developers head here.
- duped 4y agoIt's realistic to reject a CR for a new parsing function without proof that it works, which usually comes in the form of a unit test
- planede 4y agoNit: a unit test never proves that it works. At best it can prove that it doesn't work. Otherwise, I agree.
- hardware2win 4y agoNit: "never"? Even if you unit test all cases?
- tz18 4y agoYes, never. You are assuming that the implementation of all unit test cases are themselves correct (that they would fail if there was any error in the case they cover). In fact unit tests are often wrong. In that context a unit test can't even prove code incorrect, unless we know that the unit test is correct. IMO to prove that code is correct requires a proof; a unit test can only provide evidence suggestive of correctness.
- planede 4y agoMistakes in proofs are just as probable as mistakes in exhaustive tests. An exhaustive test is just one type of a machine verified proof.
- nicoburns 4y ago> An exhaustive test is just one type of a machine verified proof. Not entirely sure I agree with this. A proof by construction is a very different beast to empirical unit tests that only cover a subset of inputs. The equivalent would be units tests that cover every single possible input.
- asveikau 4y agoIt's worth noting that coverage can be a deceptive metric sometimes. You can have coverage on code that divides - it won't tell you if you ever divide by zero. You can have coverage on code that follows a pointer - it won't tell you if you ever pass a bad pointer.
- adql 4y agoSeems OpenSSL as an organization is just irreparably broken if they haven't still learned the lesson
- Someone1234 4y agoOpenSSL still receives minimum funding. Until Heartbleed they had nearly no funding, and now it is two full time people. https://en.wikipedia.org/wiki/Core_Infrastructure_Initiativev https://en.wikipedia.org/wiki/Core_Infrastructure_Initiative...
- pizza234 4y agoI'm of the opinion that in cases like this, it'd be better for the organization to close, and allow the gap to be filled naturaly. If the current OpenSSL maintainers closed the project, given its importance, there would be a rush to follow up maintenance. Chances are, it'd be better funded; even in worst case, it'll hardly be assigned less than two devs. This a case of the general dynamic where a barely-sufficient-but-arguably-insufficient solution prevents actors from finding and executing a proper one.
- AdamJacobMuller 4y agoYou don't even have to close. You can just refuse to merge code which does not include 100% test coverage. If someone wants the feature badly enough, they will figure out a way to fill the gap. Alternatively, someone can always fork the code and release "OpenSSL-but-with-lots-of-untested-code" variant.
- SoftTalker 4y agoFor a project like OpenSSL, it's not just having "enough" developers (whatever that is) it's having qualified developers. Writing good crypto code requires deep expertise. There aren't a lot of people with such expertise whose time is not already fully committed.
- coder543 4y agoI think at this point we've established that it's C which is just irreparably broken. Blaming the OpenSSL developers for writing bad C is just a "no true scotsman" at this point, since there is no large, popular C codebase in existence that I'm aware of that avoids running into vulnerabilities like this; vulnerabilities that just about every other language (mainly excluding C++) would have prevented from becoming an RCE, and likely prevented from even being a DoS. Memory safe languages obviously can't prevent all vulnerabilities, since the developer can still intentionally or unintentionally write code that simply does the wrong thing, but memory safe languages can prevent a lot of dumb vulnerabilities, including this one. No feasible amount of funding would have prevented this, since it continues to happen to much better funded projects also written in C. On the other hand, I guess we could blame the OpenSSL developers for writing C at all, being unwilling to start writing new code in a memory safe language of some kind, and ideally rewriting particularly risks code paths like parsers as well. We've learned this lesson the hard way a thousand times. C isn't going away any time soon (unfortunately), but that doesn't mean we have to continue writing new vulnerabilities like this one, which was written in the last two years.