12 ms·

It is surprising that a project that is quite mission critical is completely at the bottom of the scale when it comes to how much the development process is ori

by stiff 13y ago

It is surprising that a project that is quite mission critical is completely at the bottom of the scale when it comes to how much the development process is oriented toward reliability. There are no systematic unit tests, no systematic documentation, the best you get is a bunch of disorganized integration tests, so it is not even at the level you would expect for a decently maintained business project: https://github.com/openssl/openssl https://github.com/openssl/openssl Perhaps the open source model of development is just not very good for software of this kind. Of course it's good that the source is open for everyone to look and potentially contribute, but without funding and without having a real process and a full time team it seems to me it is hard to get the level of quality required. I also wonder how much in the end the big institutions care about this stuff. Intel hires a bunch of guys to do formal models of their processors to ensure bugs aren't shipped to millions of customers, why is nobody funding a formally specified version of SSL? For other mission critical systems, like what goes into spacecrafts, or hospitals, or gets developed in the military there are rigorous processes in use to prevent stupid mistakes, so it's somewhat disappointing that the major infrastructure pieces don't receive this kind of treatment.

zmguy 13y ago
Stiff, I think you answered your own question. Intel HIRES them, open source projects don't generally hire people. They sit around and wait for someone to contribute. Are you truly surprised that a volunteer created software is not as rigorously tested as software created by Intel?
- icebraining 13y ago
  Being FOSS doesn't have to mean relying on volunteers. Linux is mostly written by paid developers; why isn't OpenSSL, considering its reach in the commercial world?
  - hrjet 13y ago
    Slightly off topic, but do you know whether the paid developers of Linux work on the core kernel, or on device drivers?
    - abrahamsen 13y ago
      Both, as they provide the bulk of the code. It would be more illuminating to examine where the unpaid volunteers contribute. My guess would be device drivers, but I don't know.
    - rwmj 13y ago
      This has some information although not exactly what you wanted to know: http://lwn.net/Articles/579081/ http://lwn.net/Articles/579081/
- stiff 13y ago
  I had some problems expressing what bothers me clearly and edited the comment heavily, perhaps it makes more sense now. You're right it is not that surprising, but it's still disappointing that even the most rudimentary best practices are not adopted. Have a look at sqllite for comparison, also an open source project, also in C, certainly less mission critical, and what a difference: https://github.com/smparkes/sqlite/tree/master/test https://github.com/smparkes/sqlite/tree/master/test https://github.com/smparkes/sqlite/blob/master/src/rowset.c https://github.com/smparkes/sqlite/blob/master/src/rowset.c
  - andreasvc 13y ago
    sqlite does not seem less mission critical to me, and definitely relied on funding: "D. Richard Hipp designed SQLite in the spring of 2000 while working for General Dynamics on contract with the United States Navy.[7] Hipp was designing software used on board guided missile destroyers" -- http://en.wikipedia.org/wiki/Sqlite#History http://en.wikipedia.org/wiki/Sqlite#History
    - wadetandy 13y ago
      I work for a very large company that relies on a fork (with contributions back upstream) of SQLite for a majority of its massive enterprise SOA. It is not just unpaid volunteers keeping that project going.
      - zmguy 13y ago
        What do you consider when choosing SQLite issues to assigns resources? I would guess it would start with issues relevant to your roadmap. If that's the case with most enterprise FOSS contributors, they most likely trusted the features of OpenSSL they were using. Thus no reason to go poking around that section of the code. It's understandable why a team might choose to not perform an ad-hoc security audit of features that pass specs, even more so when such an audit requires niche expertise. We can hope this bug changes that attitude and more enterprises with the resources and knowledge start performing security and encryption audits. Just as your buildings have security guards, we need proactive and preemptive audits of at least the most common libraries is use, flagging of software that implement unaudited encryption libraries. A Travis CI like badge on GitHub for these audit metrics would bring attention to the problem. We could call it EncryptCI. Maybe this already exists?
- ig1 13y ago
  A number of the OpenSSL team are employed by large tech companies to work on OpenSSL. For example Bodo Möller and Ben Laurie work for Google.
  - zmguy 13y ago
    This bug was in fact discovered by a Google employee: https://www.openssl.org/news/secadv_20140407.txt https://www.openssl.org/news/secadv_20140407.txt
deleted 13y ago
[deleted]
motters 13y ago
My attitude towards complaints about open source projects is that if you think the errors are rudimentary then just submit some patches. If there is not enough test coverage then add one. This applies especially if you believe that the project is critically important. I think what might be happening here is expert syndrome. People may be told that if they're not experts then they shouldn't be reviewing or changing the code.
- stiff 13y ago
  Even if you work on a commercial project, where there is a reasonably stable core team, people just committing stuff, even if it's in itself pretty good stuff, won't result in good overall code quality. You need to have a systematic process, including a detailed coding standard, requirements about test and documentation coverage for submitted code, precise guidelines for contributing, rules of code review, and you need people who understand the whole project, review pretty much all of the code changes, have an overall development schedule in mind including refactorings and technical improvements, and all the time putting work into maintaining overall integrity of the project: http://c2.com/cgi/wiki?ConceptualIntegrity http://c2.com/cgi/wiki?ConceptualIntegrity This is pretty much a must for any project where there is more than one person working, and the more people contributing or the more mission-critical the project the more of this kind of high-level coordination is required. Some open source projects with strong leadership do get to this kind of integrity, but most don't. It's easier when there is a relatively small team of very dedicated individuals, but some large projects have succeeded to some extent in building a real development culture in an open source setting, like Linux.
- JabavuAdams 13y ago
  The failure mode with this approach is that it disallows coaching. There may be some bug in a library that's similar to what I do for my day job, and I spot it immediately. Or perhaps I've seen the inevitable results of certain design decisions play out in multiple organizations. I may not have time to code up a patch. Frankly, I've got about 5 projects on the go besides work and kids. So, FOSS projects may lose out on that kind of expertise that could easily be crowd-sourced if it didn't ask so much of contributors.
- 13y ago
belorn 13y ago
Generalization in this case is really unhelpful, since there is almost no diversity in SSL libraries. There is 3 of them, but almost everyone uses openssl. One case do not make a trend at all, and two cases is the worst statistical proof there is for trends. On the positive side, open source model of development do allow projects to use not only what is common, but also the exciting fronter regarding security models. My project has used gnutls with pgp for 6 years now, nicely avoiding all those 2 major vulnerabilities, and that was only possible because someone found themselves wanting to implement the new TLS standard that supported pgp keys.
- lucian1900 13y ago
  gnutls has also had vulnerabilities and it gets less attention, I guess you mostly got lucky. Sadly, it'll never become popular because of the license.
  - belorn 13y ago
    The same license that some libraries inside Starcraft 2 has. But blizzard must be one of those small companies that do not care about profits, or have no lawyers. No "serious company" or serious software would dare to use LGPL, right? But instead of arguing license religion, I suggest we move back to the topic of OpenSSL and vulnerabilities.
    - lucian1900 13y ago
      You know as well as I do that LGPL is not always an option. A very visible case is console games. But yes, let's go back to lamenting the state of crypto libraries, we can at least agree on that.
      - boolt 13y ago
        I would actually be interested in hearing some of the issues related to LPGL in video games and other projects. Are there stories / write ups on this that you could link to or would you mind explaining it some?
joshuak 13y ago
I agree this is incredibly surprising. Even beyond typical production testing it seems to me that critical infrastructure crypto should have a formalized testing structure based on logic and information theory (as I had assumed OpenSSL did). Build trust from the bottom up. Test that only known affirmatively tested primitives are used for memory allocation, and other known sensitive operations. Things like buffer overflow, range checking, executable code in data, etc. can all be easily tested. There is a lot of work in crypto research about trust in the logical and mathematical sense, why is this work not applied to software testing at least for infrastructure crypto? P.S. By way of incentivizing this work... seems like a pretty good dissertation topic.
return0 13y ago
It's hardly a problem of open-source, it's a problem of open-incentives. While nobody disagrees the project is critical for everyone, are you (or I) willing to make it a priority? It's a essentially a "who will build the roads" issue.
- jordigh 13y ago
  Yeah, free software isn't a development model and it doesn't mean it's made by amateurs. As another prominent free project with a much better track record for security and stability, consider OpenSSH, the crown jewel of the OpenBSD crowd, a free software distribution itself popularly renowned for its security. Theo's criticism here of OpenSSL carries a lot of weight.
  - gpvos 13y ago
    Note that despite being the crown jewel, OpenSSH's security page lists about two dozen vulnerabilities it has had at some time in the past.
    - jordigh 13y ago
      Of course, they are disclosing their vulneratiblities, they disclose everything, and they have an established audit process. Quite the leaders in security in the free software world: http://www.openbsd.org/security.html http://www.openbsd.org/security.html
oggy 13y ago
There is a formally verified implementation of TLS in F# (comes from Microsoft Research, if anybody cares): http://www.mitls.org http://www.mitls.org However, there are gotchas. At this point the TLS protocol has gotten pretty complex. Even the act of just stating what the desired security property are is not simple. Furthermore, understanding what their proof means requires you to go through at least two of their papers (and probably both of their tech reports to grok it fully). Then you need to look at the code itself and the different assertions/assumptions spread throughout, and make sure that those conform to the properties you expect. Even if you are familiar with the field, starting from scratch would probably take you a couple of weeks, just to gain confidence that what they prove is correct. And it might be possible that there are some features of TLS that they don't implement (as you can tell, I haven't taken the couple of weeks to fully understand their stuff).
- JabavuAdams 13y ago
  Sounds complicated. Let's just build an AI that can program, and trust it when it says it's not spying on us.
- kyllo 13y ago
  My assumption would be that this formally verified implementation is not optimized, and therefore runs rather slowly compared to the C implementations. Plus it requires the .NET framework or Mono to run it. Am I off base?
  - chriswarbo 13y ago
    I would imagine it is 'optimised for verification' rather than for speed, ie. the code will be written to follow the structure of the specification. However, the nice thing about having a verified implementation is that it can be used as the specification (reference implementation) for any other optimisations you want to make. Such a two-step process is usually much more tractable than trying to verify optimised code directly. This is used extensively by BedRock ( http://plv.csail.mit.edu/bedrock/ http://plv.csail.mit.edu/bedrock/ ) which uses functional programs as specifications and an assembly-like language for the "real"/optimised implementation. The resulting verification problems are straightforward enough to be mostly automated.
jameshart 13y ago
There are certainly large commercial entities who have sufficient incentive to keep OpenSSL secure that they really ought to be contributing actively to the project; I wonder whether Amazon might step up given how badly (and publicly - witness Mojang's taking minecraft services offline and pointing the finger at Amazon as the vendor they were waiting on to fix things - https://twitter.com/notch/status/453529143121309696 https://twitter.com/notch/status/453529143121309696) they got bitten by this vulnerability. Cloud hosting companies clearly have a responsibility to deliver trustworthy services, and that means they need to be deploying software stacks they can rely on not to screw them; either they will need to step up and support OpenSSL or switch to different SSL solutions that have stronger guarantees. Network appliance vendors like F5 and Cisco also have a clear interest in fixing this stuff. Question is, will they? (disclosure - I work for IBM, though not in cloud infrastructure. What I say for Amazon applies to our cloud services guys too.)
- ak217 13y ago
  I'm not sure that OpenSSL is the project they ought to be contributing to. It looks to be beyond repair architecturally (as a project as well as codebase).
  - mikevm 13y ago
    How in the heck is this mess of a project so popular?
    - lern_too_spel 13y ago
      It was developed outside the US at a time when the US had export restrictions on strong crypto. Now that those restrictions are gone, anybody can just use NSS instead.
    - aaronblohowiak 13y ago
      Age, momentum; "network effects."
    - delinka 13y ago
tomphoolery 13y ago
> I also wonder how much in the end the big institutions care about this stuff. Well, if they did care more, you'd see engineers from Intel, IBM, et. al., contributing to the project in droves like they do with the Linux kernel.
- tankenmate 13y ago
  Or maybe not after the current maintainers roll our their not so welcome mat.
- endeavor 13y ago
  Chrome and Firefox both use NSS. I would assume that the big institutions looked at OpenSSL and decided their money would be better invested in other projects.
lern_too_spel 13y ago
"Big institutions" already did that. It's called NSS.

by stiff 13y ago

zmguy 13y ago

Stiff, I think you answered your own question. Intel HIRES them, open source projects don't generally hire people. They sit around and wait for someone to contribute. Are you truly surprised that a volunteer created software is not as rigorously tested as software created by Intel?

icebraining 13y ago
Being FOSS doesn't have to mean relying on volunteers. Linux is mostly written by paid developers; why isn't OpenSSL, considering its reach in the commercial world?
- hrjet 13y ago
  Slightly off topic, but do you know whether the paid developers of Linux work on the core kernel, or on device drivers?
  - abrahamsen 13y ago
    Both, as they provide the bulk of the code. It would be more illuminating to examine where the unpaid volunteers contribute. My guess would be device drivers, but I don't know.
  - rwmj 13y ago
    This has some information although not exactly what you wanted to know: http://lwn.net/Articles/579081/ http://lwn.net/Articles/579081/
stiff 13y ago
I had some problems expressing what bothers me clearly and edited the comment heavily, perhaps it makes more sense now. You're right it is not that surprising, but it's still disappointing that even the most rudimentary best practices are not adopted. Have a look at sqllite for comparison, also an open source project, also in C, certainly less mission critical, and what a difference: https://github.com/smparkes/sqlite/tree/master/test https://github.com/smparkes/sqlite/tree/master/test https://github.com/smparkes/sqlite/blob/master/src/rowset.c https://github.com/smparkes/sqlite/blob/master/src/rowset.c
- andreasvc 13y ago
  sqlite does not seem less mission critical to me, and definitely relied on funding: "D. Richard Hipp designed SQLite in the spring of 2000 while working for General Dynamics on contract with the United States Navy.[7] Hipp was designing software used on board guided missile destroyers" -- http://en.wikipedia.org/wiki/Sqlite#History http://en.wikipedia.org/wiki/Sqlite#History
  - wadetandy 13y ago
    I work for a very large company that relies on a fork (with contributions back upstream) of SQLite for a majority of its massive enterprise SOA. It is not just unpaid volunteers keeping that project going.
    - zmguy 13y ago
      What do you consider when choosing SQLite issues to assigns resources? I would guess it would start with issues relevant to your roadmap. If that's the case with most enterprise FOSS contributors, they most likely trusted the features of OpenSSL they were using. Thus no reason to go poking around that section of the code. It's understandable why a team might choose to not perform an ad-hoc security audit of features that pass specs, even more so when such an audit requires niche expertise. We can hope this bug changes that attitude and more enterprises with the resources and knowledge start performing security and encryption audits. Just as your buildings have security guards, we need proactive and preemptive audits of at least the most common libraries is use, flagging of software that implement unaudited encryption libraries. A Travis CI like badge on GitHub for these audit metrics would bring attention to the problem. We could call it EncryptCI. Maybe this already exists?
ig1 13y ago
A number of the OpenSSL team are employed by large tech companies to work on OpenSSL. For example Bodo Möller and Ben Laurie work for Google.
- zmguy 13y ago
  This bug was in fact discovered by a Google employee: https://www.openssl.org/news/secadv_20140407.txt https://www.openssl.org/news/secadv_20140407.txt

deleted 13y ago

[deleted]

motters 13y ago

My attitude towards complaints about open source projects is that if you think the errors are rudimentary then just submit some patches. If there is not enough test coverage then add one. This applies especially if you believe that the project is critically important. I think what might be happening here is expert syndrome. People may be told that if they're not experts then they shouldn't be reviewing or changing the code.

stiff 13y ago
Even if you work on a commercial project, where there is a reasonably stable core team, people just committing stuff, even if it's in itself pretty good stuff, won't result in good overall code quality. You need to have a systematic process, including a detailed coding standard, requirements about test and documentation coverage for submitted code, precise guidelines for contributing, rules of code review, and you need people who understand the whole project, review pretty much all of the code changes, have an overall development schedule in mind including refactorings and technical improvements, and all the time putting work into maintaining overall integrity of the project: http://c2.com/cgi/wiki?ConceptualIntegrity http://c2.com/cgi/wiki?ConceptualIntegrity This is pretty much a must for any project where there is more than one person working, and the more people contributing or the more mission-critical the project the more of this kind of high-level coordination is required. Some open source projects with strong leadership do get to this kind of integrity, but most don't. It's easier when there is a relatively small team of very dedicated individuals, but some large projects have succeeded to some extent in building a real development culture in an open source setting, like Linux.
JabavuAdams 13y ago
The failure mode with this approach is that it disallows coaching. There may be some bug in a library that's similar to what I do for my day job, and I spot it immediately. Or perhaps I've seen the inevitable results of certain design decisions play out in multiple organizations. I may not have time to code up a patch. Frankly, I've got about 5 projects on the go besides work and kids. So, FOSS projects may lose out on that kind of expertise that could easily be crowd-sourced if it didn't ask so much of contributors.
13y ago

belorn 13y ago

Generalization in this case is really unhelpful, since there is almost no diversity in SSL libraries. There is 3 of them, but almost everyone uses openssl. One case do not make a trend at all, and two cases is the worst statistical proof there is for trends. On the positive side, open source model of development do allow projects to use not only what is common, but also the exciting fronter regarding security models. My project has used gnutls with pgp for 6 years now, nicely avoiding all those 2 major vulnerabilities, and that was only possible because someone found themselves wanting to implement the new TLS standard that supported pgp keys.

lucian1900 13y ago
gnutls has also had vulnerabilities and it gets less attention, I guess you mostly got lucky. Sadly, it'll never become popular because of the license.
- belorn 13y ago
  The same license that some libraries inside Starcraft 2 has. But blizzard must be one of those small companies that do not care about profits, or have no lawyers. No "serious company" or serious software would dare to use LGPL, right? But instead of arguing license religion, I suggest we move back to the topic of OpenSSL and vulnerabilities.
  - lucian1900 13y ago
    You know as well as I do that LGPL is not always an option. A very visible case is console games. But yes, let's go back to lamenting the state of crypto libraries, we can at least agree on that.
    - boolt 13y ago
      I would actually be interested in hearing some of the issues related to LPGL in video games and other projects. Are there stories / write ups on this that you could link to or would you mind explaining it some?

joshuak 13y ago

I agree this is incredibly surprising. Even beyond typical production testing it seems to me that critical infrastructure crypto should have a formalized testing structure based on logic and information theory (as I had assumed OpenSSL did). Build trust from the bottom up. Test that only known affirmatively tested primitives are used for memory allocation, and other known sensitive operations. Things like buffer overflow, range checking, executable code in data, etc. can all be easily tested. There is a lot of work in crypto research about trust in the logical and mathematical sense, why is this work not applied to software testing at least for infrastructure crypto? P.S. By way of incentivizing this work... seems like a pretty good dissertation topic.

return0 13y ago

It's hardly a problem of open-source, it's a problem of open-incentives. While nobody disagrees the project is critical for everyone, are you (or I) willing to make it a priority? It's a essentially a "who will build the roads" issue.

jordigh 13y ago
Yeah, free software isn't a development model and it doesn't mean it's made by amateurs. As another prominent free project with a much better track record for security and stability, consider OpenSSH, the crown jewel of the OpenBSD crowd, a free software distribution itself popularly renowned for its security. Theo's criticism here of OpenSSL carries a lot of weight.
- gpvos 13y ago
  Note that despite being the crown jewel, OpenSSH's security page lists about two dozen vulnerabilities it has had at some time in the past.
  - jordigh 13y ago
    Of course, they are disclosing their vulneratiblities, they disclose everything, and they have an established audit process. Quite the leaders in security in the free software world: http://www.openbsd.org/security.html http://www.openbsd.org/security.html

oggy 13y ago

There is a formally verified implementation of TLS in F# (comes from Microsoft Research, if anybody cares): http://www.mitls.org http://www.mitls.org However, there are gotchas. At this point the TLS protocol has gotten pretty complex. Even the act of just stating what the desired security property are is not simple. Furthermore, understanding what their proof means requires you to go through at least two of their papers (and probably both of their tech reports to grok it fully). Then you need to look at the code itself and the different assertions/assumptions spread throughout, and make sure that those conform to the properties you expect. Even if you are familiar with the field, starting from scratch would probably take you a couple of weeks, just to gain confidence that what they prove is correct. And it might be possible that there are some features of TLS that they don't implement (as you can tell, I haven't taken the couple of weeks to fully understand their stuff).

JabavuAdams 13y ago
Sounds complicated. Let's just build an AI that can program, and trust it when it says it's not spying on us.
kyllo 13y ago
My assumption would be that this formally verified implementation is not optimized, and therefore runs rather slowly compared to the C implementations. Plus it requires the .NET framework or Mono to run it. Am I off base?
- chriswarbo 13y ago
  I would imagine it is 'optimised for verification' rather than for speed, ie. the code will be written to follow the structure of the specification. However, the nice thing about having a verified implementation is that it can be used as the specification (reference implementation) for any other optimisations you want to make. Such a two-step process is usually much more tractable than trying to verify optimised code directly. This is used extensively by BedRock ( http://plv.csail.mit.edu/bedrock/ http://plv.csail.mit.edu/bedrock/ ) which uses functional programs as specifications and an assembly-like language for the "real"/optimised implementation. The resulting verification problems are straightforward enough to be mostly automated.

jameshart 13y ago

There are certainly large commercial entities who have sufficient incentive to keep OpenSSL secure that they really ought to be contributing actively to the project; I wonder whether Amazon might step up given how badly (and publicly - witness Mojang's taking minecraft services offline and pointing the finger at Amazon as the vendor they were waiting on to fix things - https://twitter.com/notch/status/453529143121309696 https://twitter.com/notch/status/453529143121309696) they got bitten by this vulnerability. Cloud hosting companies clearly have a responsibility to deliver trustworthy services, and that means they need to be deploying software stacks they can rely on not to screw them; either they will need to step up and support OpenSSL or switch to different SSL solutions that have stronger guarantees. Network appliance vendors like F5 and Cisco also have a clear interest in fixing this stuff. Question is, will they? (disclosure - I work for IBM, though not in cloud infrastructure. What I say for Amazon applies to our cloud services guys too.)

ak217 13y ago
I'm not sure that OpenSSL is the project they ought to be contributing to. It looks to be beyond repair architecturally (as a project as well as codebase).
- mikevm 13y ago
  How in the heck is this mess of a project so popular?
  - lern_too_spel 13y ago
    It was developed outside the US at a time when the US had export restrictions on strong crypto. Now that those restrictions are gone, anybody can just use NSS instead.
  - aaronblohowiak 13y ago
    Age, momentum; "network effects."
  - delinka 13y ago

tomphoolery 13y ago

> I also wonder how much in the end the big institutions care about this stuff. Well, if they did care more, you'd see engineers from Intel, IBM, et. al., contributing to the project in droves like they do with the Linux kernel.

tankenmate 13y ago
Or maybe not after the current maintainers roll our their not so welcome mat.
endeavor 13y ago
Chrome and Firefox both use NSS. I would assume that the big institutions looked at OpenSSL and decided their money would be better invested in other projects.

lern_too_spel 13y ago

"Big institutions" already did that. It's called NSS.