3 ms·
I mean, branch coverage could have fingered this hole in the unit tests, couldn't it? The problem is X509_V_FLAG_X509_STRICT was added in 2004 without any corre
by jeffbee 6y ago
I mean, branch coverage could have fingered this hole in the unit tests, couldn't it? The problem is X509_V_FLAG_X509_STRICT was added in 2004 without any corresponding tests. The project still has decades of debt from past poor practices to pay down.
- makomk 6y agoI don't see how it could. If they had a test that X509_V_FLAG_X509_STRICT carried out the intended additional checks, and a test that non-CA certificates could not issue other certificates, and a test that valid certificate chains were accepted, wouldn't that cover all the branches involved without ever detecting the particular problematic combination involved in this bug?
- jeffbee 6y agoI dunno, at least the new release slightly uh, changes?, the coverage. ~/openssl-1.1.1i % gcov crypto/x509/x509_vfy.c -bm File 'crypto/x509/x509_vfy.c' Lines executed:63.20% of 1625 Branches executed:73.70% of 1194 Taken at least once:53.85% of 1194 Calls executed:58.47% of 496 Creating 'x509_vfy.c.gcov' ~/openssl-1.1.1k % gcov crypto/x509/x509_vfy.c -bm File 'crypto/x509/x509_vfy.c' Lines executed:63.21% of 1628 Branches executed:73.75% of 1196 Taken at least once:53.93% of 1196 Calls executed:58.47% of 496 Creating 'x509_vfy.c.gcov' There are still hundreds of never-taken branches in this file alone. ~/openssl-1.1.1k % grep -c never\ executed x509_vfy.c.gcov 520
- tptacek 6y agoI 100% buy that OpenSSL hasn't dug itself out of the hole it was in back in 2012. But building a new TLS library puts you in the same hole with respect to exhaustive unit test coverage, right? You're better off starting from a place where you already have a fairly extensive (if incomplete) unit test infrastructure than one where you're starting from scratch. This is a weakly held opinion. I'm mostly motivated by the knee-jerk response OpenSSL gets on threads like these; I think those responses are based on a reputation that doesn't capture the level of work that has gone into the project in the years since 2012. That doesn't mean it's the apogee of what we can do with a TLS library.
- jeffbee 6y agoI agree that it is in much better shape today and any competing project starting from scratch would be at a disadvantage with respect to the size of their test suite. However, I also like the BoringSSL approach of forking the project and deleting all the dumb features that nobody needs. They deleted the option flag involved in this advisory, years ago. By deleting it they avoided having to add tests for it.
- tialaramex 6y agoUltimately I agree with you about what the right approach is, but the route there is convoluted. BoringSSL is able to be simpler by defining pretty tight criteria for what it cares about and everything else either doesn't work or isn't guaranteed to exist at all. I happen to think the safe way forward is to agree that OK then, we can't do all these other weird things, too bad. But this is an extremely unpopular position because lots of people have at least one weird thing they very much want to do.
- est31 6y agoIdeally at least the integration tests are interchangeable between implementations of the same thing.