4 ms·

OpenSSL was thought to be "bulletproof" and the "go to" option. And look where we are now

nathell 9y ago
OpenSSL has always been nowhere near the level of robustness and bulletproofness that SQLite is at. Just look at how it's tested: https://www.sqlite.org/testing.html https://www.sqlite.org/testing.html
- vbezhenar 9y ago
  It's too bad some part of tests are proprietary. I think it someone would want to rewrite SQLite (keeping compatibility), those tests would be of huge value.
  - krylon 9y ago
    Yes, but that is also a source of funding for the project.
lghh 9y ago
But it was bulletproof until proven otherwise.
d33 9y ago
I suspect there's a huge code quality difference between OpenSSL and SQLite. Also, keep in mind that SQLite was much easier to fuzz - it's basically one of the libraries that AFL has specific features for (dictionary-based coverage-guided fuzzing).
Sir_Cmpwn 9y ago
I don't think this comparison is fair. SQLite is widely regarded as one of the best codebases in the world and it has 100% test coverage.
- sidlls 9y ago
  That level of test coverage isn't as meaningful as people think. I'd bet that a significant fraction (> 10%) of those tests are redundant or cost more (in terms of maintenance, for one example) than the value they provide.
  - Sir_Cmpwn 9y ago
    I think you should read the full write-up on sqlite testing: https://www.sqlite.org/testing.html https://www.sqlite.org/testing.html There's no question that sqlite is one of the most well understood and reliable codebases in the world.
IncRnd 9y ago
OpenSSL was never thought to be bulletproof. It is, however, the best of the worst for many scenarios.
jeremiep 9y ago
And how many more critical flaws would OpenSSL have if people kept rewriting it in the trending language of the week? Rewriting just for the sake of rewriting or because you like another language better is an almost certain recipe for disaster.
jagger27 9y ago
I have a hard time believing this. It was common knowledge and lamented before Heartbleed that OpenSSL was bloated an unwieldy. People were surprised by the severity, not that it happened.
jcranmer 9y ago
SQLite has a much more thorough testing process than OpenSSL did. Their acceptance tests have 100% MC/DC coverage [1], for example. While previous things like the IOC tester did find issues with SQLite, SQLite is very aggressive about including these sorts of tools into their ongoing testing process. [1] MC/DC coverage is a slightly more rigorous form than branch coverage. A condition if (a && b) requires that you only test one of (a false, b true) and (a false, b false) for 100% coverage, whereas MC/DC coverage would insist on both being tested.
saagarjha 9y ago
OpenSSL is known to have a poorly documented, spaghetti codebase. The same is not true for SQLite.
- protomyth 9y ago
  Bob Beck during his LibreSSL talk had a good overview of what was thought of OpenSSL https://youtu.be/GnBbhXBDmwU?t=2m11s https://youtu.be/GnBbhXBDmwU?t=2m11s "we are all guilty"

OpenSSL has always been nowhere near the level of robustness and bulletproofness that SQLite is at. Just look at how it's tested: https://www.sqlite.org/testing.html https://www.sqlite.org/testing.html

vbezhenar 9y ago
It's too bad some part of tests are proprietary. I think it someone would want to rewrite SQLite (keeping compatibility), those tests would be of huge value.
- krylon 9y ago
  Yes, but that is also a source of funding for the project.

lghh 9y ago

But it was bulletproof until proven otherwise.

d33 9y ago

I suspect there's a huge code quality difference between OpenSSL and SQLite. Also, keep in mind that SQLite was much easier to fuzz - it's basically one of the libraries that AFL has specific features for (dictionary-based coverage-guided fuzzing).

Sir_Cmpwn 9y ago

I don't think this comparison is fair. SQLite is widely regarded as one of the best codebases in the world and it has 100% test coverage.

sidlls 9y ago
That level of test coverage isn't as meaningful as people think. I'd bet that a significant fraction (> 10%) of those tests are redundant or cost more (in terms of maintenance, for one example) than the value they provide.
- Sir_Cmpwn 9y ago
  I think you should read the full write-up on sqlite testing: https://www.sqlite.org/testing.html https://www.sqlite.org/testing.html There's no question that sqlite is one of the most well understood and reliable codebases in the world.

IncRnd 9y ago

OpenSSL was never thought to be bulletproof. It is, however, the best of the worst for many scenarios.

jeremiep 9y ago

And how many more critical flaws would OpenSSL have if people kept rewriting it in the trending language of the week? Rewriting just for the sake of rewriting or because you like another language better is an almost certain recipe for disaster.

jagger27 9y ago

I have a hard time believing this. It was common knowledge and lamented before Heartbleed that OpenSSL was bloated an unwieldy. People were surprised by the severity, not that it happened.

jcranmer 9y ago

SQLite has a much more thorough testing process than OpenSSL did. Their acceptance tests have 100% MC/DC coverage [1], for example. While previous things like the IOC tester did find issues with SQLite, SQLite is very aggressive about including these sorts of tools into their ongoing testing process. [1] MC/DC coverage is a slightly more rigorous form than branch coverage. A condition if (a && b) requires that you only test one of (a false, b true) and (a false, b false) for 100% coverage, whereas MC/DC coverage would insist on both being tested.

saagarjha 9y ago

OpenSSL is known to have a poorly documented, spaghetti codebase. The same is not true for SQLite.

protomyth 9y ago
Bob Beck during his LibreSSL talk had a good overview of what was thought of OpenSSL https://youtu.be/GnBbhXBDmwU?t=2m11s https://youtu.be/GnBbhXBDmwU?t=2m11s "we are all guilty"