10 ms·

Of Money, Responsibility, and Pride

112 points by silenteh 12y ago

runn1ng 12y ago
Google Cache version (site is 404ing for me) http://webcache.googleusercontent.com/search?q=cache:http://veridicalsystems.com/blog/of-money-responsibility-and-pride/&strip=1 http://webcache.googleusercontent.com/search?q=cache:http://...
tptacek 12y ago
Could someone involved with the OpenSSL Foundation and the OpenSSL project maybe pitch in with a quick description of how the project is managed? * Who owns which subsystems? * Is there a board of governors or a BDFL or something like that effectively overseeing the whole project? * What is the process for screening commits from people new to the project? This whole post seems to be tinged with a bit of defensiveness on behalf of the most active committers to the project. But it wasn't the active committers who introduced this most recent bug.
wbl 12y ago
OpenBSD has had two holes in a heck of a long time. By contrast OpenSSL has had a remote execute in 2010, and another 4 in 2002, and is regularly patching DOS's resulting from memory corruption that turns out not to be exploitable. It is 453,000 or so lines, more than five times the size of xv6. It is ten times as big as PolarSSL. Documentation and internal structuring is wildly inconsistent. Features that make static analysis annoying are widely used. The API is far too low level. Do you believe this is acceptable in a security library? Do believe that aspiring to the security of qmail or OpenSSH is a reasonable goal, even at the cost of features? Why should I use OpenSSL for TLS termination when formally validated alternatives exist?
- wfn 12y ago
  > Why should I use OpenSSL for TLS termination when formally validated alternatives exist? Oh please do share! (spoiler: alternatives which enable side channels because the pretty compiled-optimized-code (that is generated from source code that itself may feature immutable data etc. with no side effects) is ripe with leaks via cpu branching, caching, etc etc do not count.) This is not a spiteful rhetorical inquiry, by the way!
  - AaronFriel 12y ago
    Those side-channel attacks are largely theoretical. OpenSSL and RSA in general are vulnerable to side channel attacks, because many of the fundamental operations are not constant-time. That can change eventually, but RSA is difficult to write in a constant-time manner. OpenSSL certainly has had its fair of lab-demonstrated side-channel attacks, but I don't think anyone has been able to demonstrate their use in virtualized hosting environments against arbitrary other tenants. Side channel attacks once you've already got code running on the same operating system as the target are much easier. But if you can get arbitrary code to execute on the same machine running OpenSSL, you probably already have their keys. So, I think the criticism to OpenSSL is valid. Why use it when it seems there are less bad alternatives? A lot of it comes down to network effects, inertia, and licensing. That's a much better answer than side-channel attack surface area, which all RSA implementations share :)
    - wbl 12y ago
      Nope: you can exploit the timing attacks from across a network link in AES. DJB did this long, long ago. Furthermore, I believe OpenSSL has constant-time RSA, but you can always use ECDSA for which constant time implementations in C exist. (I wrote one, but I still need to switch the hash function to SHA256 and add the unnecessary encoding steps to the output, so consider it a PoC).
      - AaronFriel 12y ago
        Are you sure? Searching for "side channel OpenSSL" reveals a majority of the attacks are against ECDSA. Of course, searches aren't the best measure of vulnerability, it's just an indication of popularity.
        wbl 12y ago
        http://cr.yp.to/antiforgery/cachetiming-20050414.pdf http://cr.yp.to/antiforgery/cachetiming-20050414.pdf ECDSA is nice because you can use partial nonce recovery+lattice reduction. Furthermore recent Intel chips have AES instructions.
- riffraff 12y ago
  I believe the formally validated alternatives have licenses that don't make them acceptable for a large part of the possible users.
  - judk 12y ago
    SSL is important. Proprietary users should buy proprietary licenses
- dfc 12y ago
  I am not defending OpenSSL but I am not sure your comparisons are very informative and quite frankly some of your sloccounts seem to wander far away from fact. > OpenBSD has had two holes in a heck of a long time Two remote holes in the default install. The default install is configured in such a way as to minimize the attack surface area. Do you know what services are configured to respond to public internet traffic in the default install? OpenSSL on the other hand essentially is always interacting with public internet traffic. Do you use OpenBSD on your desktop and laptop? > 453,000 or so lines, more than five times the size of xv6 Are you referring to Xv6, "the simple Unix-like teaching operating system" a project designed for education and not commercial service offerings? How does this comparison of the size of xv6/OpenSSL inform the debate? The Linux kernel has 12 million lines of code. What should I conclude from this number compared to OpenSSL? Do you think some of PolarSSL's failures with frankencerts can be explained by how small the code base is? Please see the addendum for more sloccount discussion. > Do believe that aspiring to the security...even at the cost of features? I don't mean to be difficult but I have no idea what you are asking here. Which features are you referring to? Without some specificity of "cost of features" this seems to be more about showmanship and rhetorical style than fostering meaningful debate. Do you run OpenBSD on your desktop/laptop? > Why should I use OpenSSL for TLS termination when formally validated alternatives exist? What are these alternatives? I did not realize there were multiple formally validated alternatives to OpenSSL. Maybe we have different opinions about "alternatives" versus "implementations." SLOCCOUNT Addendum: I did not want to inject a pile of sloccount data in the main body of my comment but I do have some questions about your counts. After a little research it seems that your numbers might be somewhat hand wavy which is unfortunate because you gave no indication of this in your comment. What versions of OpenSSL and PolarSSL are you referring to? My count for OpenSSL 1.0.1g puts it at 6.5 times the size of PolarSSL and not the 10 times that you stated in your comment. I could not connect to OpenSSL.org so I do not know if they have released a version since April 7th but I must say I am surprised that that the OpenSSL dev team added 91,344 lines (25% increase) in the last seven days. And even if they did that still puts them 90,000 lines short of "ten times the size of PolarSSL." My sloccount data: # OpenSSL 1.0.1g: Totals grouped by language (dominant language first): ansic: 273820 (75.71%) perl: 69192 (19.13%) asm: 11015 (3.05%) cpp: 4367 (1.21%) sh: 3238 (0.90%) lisp: 24 (0.01%) Total Physical Source Lines of Code (SLOC) = 361,656 # PolarSSL 1.3.6-gpl Totals grouped by language (dominant language first): ansic: 52212 (94.71%) sh: 2165 (3.93%) perl: 748 (1.36%) tcl: 4 (0.01%) Total Physical Source Lines of Code (SLOC) = 55,129 # Linux kernel 3.14 Totals grouped by language (dominant language first): ansic: 11828087 (97.00%) asm: 275047 (2.26%) !!!TRIMMED counts < 0.5%!!! Total Physical Source Lines of Code (SLOC) = 12,193,312 # Xv6 Totals grouped by language (dominant language first): ansic: 7022 (87.81%) sh: 481 (6.01%) asm: 301 (3.76%) perl: 189 (2.36%) lisp: 4 (0.05%) Total Physical Source Lines of Code (SLOC) = 7,997
AaronFriel 12y ago
This is a mistake. The Apache foundation doesn't sell $250/hour consulting gigs for its primary source of revenue. Neither does the Linux Foundation, the SQLite Consortium, or other massive, mission-critical open source products. This is the wrong funding model. It keeps money in OpenSSL developer's pockets, but there is no financial incentive for any OpenSSL developer to work on foundational improvements to OpenSSL. He said himself: there is over $100,000 in open contracts for competent developers to work on non-foundational improvements to the project. If you are an enterprising developer with good C skills and a knack for crypto projects and you apply to work for the OpenSSL foundation, are you going to start servicing that $100,000 pool of contracts or are you going to pretend that money doesn't exist and live on ramen? If nearly all of OpenSSL's revenue comes from clients that want OpenSSL to meet their particular needs, then none of that money is going to developers to strengthen OpenSSL's foundation. This is why OpenSSL looks like a hodgepodge of hacks upon hacks in order to accomplish narrow goals with limited impact testing. It should be no surprise to anyone else: clients are literally paying OpenSSL developers for this, and nothing else. Who is paying OpenSSL for developers to clean up the code base and remove ancient #IFDEFs? Who is paying OpenSSL for developers to analyze code paths and do case analysis? Who is paying OpenSSL for developers to write unit tests or even have a test harness at all? No one will pay an hourly rate to accomplish these tasks. Google is not going to pay by the hour for a developer to stare at a function until they grok it; they want a feature. Joe Company will not pay for developers to write unit tests, they want OpenSSL to handle $QUIRK from a vendor's system, or to know how to make their code handle it. This model needs to go away. Competent OpenSSL developers time is too valuable to waste on client asks. Their project is too important, and as long as the money is flowing only for novel features and not structural improvement, then that money will dictate that only new features are developed.
- deleted 12y ago
  [deleted]
- uuid_to_string 12y ago
  This is one of the better comments I have seen on OpenSSL in the past week. Well said. "This is why OpenSSL looks like a hodgepodge of hacks upon hacks in order to accomplish narrow goals with limited impact testing." It doesn't just look like a hodgepodege of accumulated hacks, it is a hodgepodge of accumulated hacks. :) "It should be no surprise to anyone else: clients are literally paying OpenSSL developers for this, and nothing else." One could say this with respect to many popular open source projects, including ones with corporate sponsorship. The complexity just keeps building over time and there is no such thing as "finished, accepting bug fixes only". "Who is paying OpenSSL for developers to clean up the code base and remove ancient #IFDEFs? Who is paying OpenSSL for developers to analyze code paths and do case analysis? Who is paying OpenSSL for developers to write unit tests or even have a test harness at all?" Those are rhetorical questions. We know the answers. Alas, when the people who pay for (open source) software and consulting pay to have "features" removed instead of added, "pigs will fly". Doug McIllroy is quoted as saying, "The hero is the negative coder". (Just in case this need explanation: Prof. McIllroy is the mind behind UNIX pipes and one of computer science's most prominant contributors. "Negative coder" means someone who removes code instead of constantly adding, or "committing", new code.) We could really use some more heros. And as we switch away from OpenSSL there will be a lot of links to libssl to remove. Meanwhile some people have been writing and testing small, auditable and usable open source crypto, more or less for "free". http://tweetnacl.cr.yp.to http://tweetnacl.cr.yp.to My guess (and hope) is that pathological requests for "features" to be added would be met with heavy scrutiny. The authors already have day jobs in academia.
deleted 12y ago
[deleted]
x0x0 12y ago
It's well and fine that Stephen lives very cheaply, but all of this is an attempt to distract from the OpenSSL project's very real issues by wearing a cilice then bitching about it. The fundamental facts are these: openssl contains a large quantity of code that, if I where to check into my company's repo, I would have at best a rough conversation with the cto and at worst I'd get fired. Plus a lack of good tests. These combine to create more than hypothetical problems; we've seen some severe security holes and there's almost certainly more to come. The question that should be discussed is if openssl is, ala sendmail, unsuitable for purpose and, if so, what should it be replaced with.
dfc 12y ago
The tor project is a great example of how open source software (OSS) projects can work with sponsors. Trying to find more information on Qualys and PSW Groups sponsorship of openssl is a nightmare compared to tor project sponsors.[^1][^2] Without the tor project's emphasis on transparency and professionalism I doubt they could post numbers like this: Since meeting the revenue milestones of $1,253,241 in 2009, $1,574,119 in 2010 and $1,681,101 in 2011, Tor has reached new heights in 2012 with over $2 million in revenue (unaudited).[^3] My comment should not be read as a criticism of OpenSSL, it should be interpreted as cause for optimism. The tor project has demonstrated that OSS projects can get sponsored to solve complicated security problems that are difficult to explain to the general public. [^1]: List of sponsors/projects: https://trac.torproject.org/projects/tor/wiki/org/sponsors https://trac.torproject.org/projects/tor/wiki/org/sponsors [^2]: Example monthly report for SponsorF's project: https://lists.torproject.org/pipermail/tor-reports/2014-March/000484.html https://lists.torproject.org/pipermail/tor-reports/2014-Marc... [^3]: Tor Project Annual Report 2012, pg 8, https://www.torproject.org/about/findoc/2012-TorProject-Annual-Report.pdf https://www.torproject.org/about/findoc/2012-TorProject-Annu...
piercebot 12y ago
Reading about Dr. Stephen Henson reminded me of the article written about Tarn Adams, the creator of Dwarf Fortress. http://www.nytimes.com/2011/07/24/magazine/the-brilliance-of-dwarf-fortress.html?pagewanted=all&_r=0 http://www.nytimes.com/2011/07/24/magazine/the-brilliance-of...
conductr 12y ago
Not hard. Just change the license and require commercial use to be paid

112 points by silenteh 12y ago

Google Cache version (site is 404ing for me) http://webcache.googleusercontent.com/search?q=cache:http://veridicalsystems.com/blog/of-money-responsibility-and-pride/&strip=1 http://webcache.googleusercontent.com/search?q=cache:http://...

tptacek 12y ago

Could someone involved with the OpenSSL Foundation and the OpenSSL project maybe pitch in with a quick description of how the project is managed? * Who owns which subsystems? * Is there a board of governors or a BDFL or something like that effectively overseeing the whole project? * What is the process for screening commits from people new to the project? This whole post seems to be tinged with a bit of defensiveness on behalf of the most active committers to the project. But it wasn't the active committers who introduced this most recent bug.

wbl 12y ago

OpenBSD has had two holes in a heck of a long time. By contrast OpenSSL has had a remote execute in 2010, and another 4 in 2002, and is regularly patching DOS's resulting from memory corruption that turns out not to be exploitable. It is 453,000 or so lines, more than five times the size of xv6. It is ten times as big as PolarSSL. Documentation and internal structuring is wildly inconsistent. Features that make static analysis annoying are widely used. The API is far too low level. Do you believe this is acceptable in a security library? Do believe that aspiring to the security of qmail or OpenSSH is a reasonable goal, even at the cost of features? Why should I use OpenSSL for TLS termination when formally validated alternatives exist?

wfn 12y ago
> Why should I use OpenSSL for TLS termination when formally validated alternatives exist? Oh please do share! (spoiler: alternatives which enable side channels because the pretty compiled-optimized-code (that is generated from source code that itself may feature immutable data etc. with no side effects) is ripe with leaks via cpu branching, caching, etc etc do not count.) This is not a spiteful rhetorical inquiry, by the way!
- AaronFriel 12y ago
  Those side-channel attacks are largely theoretical. OpenSSL and RSA in general are vulnerable to side channel attacks, because many of the fundamental operations are not constant-time. That can change eventually, but RSA is difficult to write in a constant-time manner. OpenSSL certainly has had its fair of lab-demonstrated side-channel attacks, but I don't think anyone has been able to demonstrate their use in virtualized hosting environments against arbitrary other tenants. Side channel attacks once you've already got code running on the same operating system as the target are much easier. But if you can get arbitrary code to execute on the same machine running OpenSSL, you probably already have their keys. So, I think the criticism to OpenSSL is valid. Why use it when it seems there are less bad alternatives? A lot of it comes down to network effects, inertia, and licensing. That's a much better answer than side-channel attack surface area, which all RSA implementations share :)
  - wbl 12y ago
    Nope: you can exploit the timing attacks from across a network link in AES. DJB did this long, long ago. Furthermore, I believe OpenSSL has constant-time RSA, but you can always use ECDSA for which constant time implementations in C exist. (I wrote one, but I still need to switch the hash function to SHA256 and add the unnecessary encoding steps to the output, so consider it a PoC).
    - AaronFriel 12y ago
      Are you sure? Searching for "side channel OpenSSL" reveals a majority of the attacks are against ECDSA. Of course, searches aren't the best measure of vulnerability, it's just an indication of popularity.
      - wbl 12y ago
        http://cr.yp.to/antiforgery/cachetiming-20050414.pdf http://cr.yp.to/antiforgery/cachetiming-20050414.pdf ECDSA is nice because you can use partial nonce recovery+lattice reduction. Furthermore recent Intel chips have AES instructions.
riffraff 12y ago
I believe the formally validated alternatives have licenses that don't make them acceptable for a large part of the possible users.
- judk 12y ago
  SSL is important. Proprietary users should buy proprietary licenses
dfc 12y ago
I am not defending OpenSSL but I am not sure your comparisons are very informative and quite frankly some of your sloccounts seem to wander far away from fact. > OpenBSD has had two holes in a heck of a long time Two remote holes in the default install. The default install is configured in such a way as to minimize the attack surface area. Do you know what services are configured to respond to public internet traffic in the default install? OpenSSL on the other hand essentially is always interacting with public internet traffic. Do you use OpenBSD on your desktop and laptop? > 453,000 or so lines, more than five times the size of xv6 Are you referring to Xv6, "the simple Unix-like teaching operating system" a project designed for education and not commercial service offerings? How does this comparison of the size of xv6/OpenSSL inform the debate? The Linux kernel has 12 million lines of code. What should I conclude from this number compared to OpenSSL? Do you think some of PolarSSL's failures with frankencerts can be explained by how small the code base is? Please see the addendum for more sloccount discussion. > Do believe that aspiring to the security...even at the cost of features? I don't mean to be difficult but I have no idea what you are asking here. Which features are you referring to? Without some specificity of "cost of features" this seems to be more about showmanship and rhetorical style than fostering meaningful debate. Do you run OpenBSD on your desktop/laptop? > Why should I use OpenSSL for TLS termination when formally validated alternatives exist? What are these alternatives? I did not realize there were multiple formally validated alternatives to OpenSSL. Maybe we have different opinions about "alternatives" versus "implementations." SLOCCOUNT Addendum: I did not want to inject a pile of sloccount data in the main body of my comment but I do have some questions about your counts. After a little research it seems that your numbers might be somewhat hand wavy which is unfortunate because you gave no indication of this in your comment. What versions of OpenSSL and PolarSSL are you referring to? My count for OpenSSL 1.0.1g puts it at 6.5 times the size of PolarSSL and not the 10 times that you stated in your comment. I could not connect to OpenSSL.org so I do not know if they have released a version since April 7th but I must say I am surprised that that the OpenSSL dev team added 91,344 lines (25% increase) in the last seven days. And even if they did that still puts them 90,000 lines short of "ten times the size of PolarSSL." My sloccount data: # OpenSSL 1.0.1g: Totals grouped by language (dominant language first): ansic: 273820 (75.71%) perl: 69192 (19.13%) asm: 11015 (3.05%) cpp: 4367 (1.21%) sh: 3238 (0.90%) lisp: 24 (0.01%) Total Physical Source Lines of Code (SLOC) = 361,656 # PolarSSL 1.3.6-gpl Totals grouped by language (dominant language first): ansic: 52212 (94.71%) sh: 2165 (3.93%) perl: 748 (1.36%) tcl: 4 (0.01%) Total Physical Source Lines of Code (SLOC) = 55,129 # Linux kernel 3.14 Totals grouped by language (dominant language first): ansic: 11828087 (97.00%) asm: 275047 (2.26%) !!!TRIMMED counts < 0.5%!!! Total Physical Source Lines of Code (SLOC) = 12,193,312 # Xv6 Totals grouped by language (dominant language first): ansic: 7022 (87.81%) sh: 481 (6.01%) asm: 301 (3.76%) perl: 189 (2.36%) lisp: 4 (0.05%) Total Physical Source Lines of Code (SLOC) = 7,997

AaronFriel 12y ago

This is a mistake. The Apache foundation doesn't sell $250/hour consulting gigs for its primary source of revenue. Neither does the Linux Foundation, the SQLite Consortium, or other massive, mission-critical open source products. This is the wrong funding model. It keeps money in OpenSSL developer's pockets, but there is no financial incentive for any OpenSSL developer to work on foundational improvements to OpenSSL. He said himself: there is over $100,000 in open contracts for competent developers to work on non-foundational improvements to the project. If you are an enterprising developer with good C skills and a knack for crypto projects and you apply to work for the OpenSSL foundation, are you going to start servicing that $100,000 pool of contracts or are you going to pretend that money doesn't exist and live on ramen? If nearly all of OpenSSL's revenue comes from clients that want OpenSSL to meet their particular needs, then none of that money is going to developers to strengthen OpenSSL's foundation. This is why OpenSSL looks like a hodgepodge of hacks upon hacks in order to accomplish narrow goals with limited impact testing. It should be no surprise to anyone else: clients are literally paying OpenSSL developers for this, and nothing else. Who is paying OpenSSL for developers to clean up the code base and remove ancient #IFDEFs? Who is paying OpenSSL for developers to analyze code paths and do case analysis? Who is paying OpenSSL for developers to write unit tests or even have a test harness at all? No one will pay an hourly rate to accomplish these tasks. Google is not going to pay by the hour for a developer to stare at a function until they grok it; they want a feature. Joe Company will not pay for developers to write unit tests, they want OpenSSL to handle $QUIRK from a vendor's system, or to know how to make their code handle it. This model needs to go away. Competent OpenSSL developers time is too valuable to waste on client asks. Their project is too important, and as long as the money is flowing only for novel features and not structural improvement, then that money will dictate that only new features are developed.

deleted 12y ago
[deleted]
uuid_to_string 12y ago
This is one of the better comments I have seen on OpenSSL in the past week. Well said. "This is why OpenSSL looks like a hodgepodge of hacks upon hacks in order to accomplish narrow goals with limited impact testing." It doesn't just look like a hodgepodege of accumulated hacks, it is a hodgepodge of accumulated hacks. :) "It should be no surprise to anyone else: clients are literally paying OpenSSL developers for this, and nothing else." One could say this with respect to many popular open source projects, including ones with corporate sponsorship. The complexity just keeps building over time and there is no such thing as "finished, accepting bug fixes only". "Who is paying OpenSSL for developers to clean up the code base and remove ancient #IFDEFs? Who is paying OpenSSL for developers to analyze code paths and do case analysis? Who is paying OpenSSL for developers to write unit tests or even have a test harness at all?" Those are rhetorical questions. We know the answers. Alas, when the people who pay for (open source) software and consulting pay to have "features" removed instead of added, "pigs will fly". Doug McIllroy is quoted as saying, "The hero is the negative coder". (Just in case this need explanation: Prof. McIllroy is the mind behind UNIX pipes and one of computer science's most prominant contributors. "Negative coder" means someone who removes code instead of constantly adding, or "committing", new code.) We could really use some more heros. And as we switch away from OpenSSL there will be a lot of links to libssl to remove. Meanwhile some people have been writing and testing small, auditable and usable open source crypto, more or less for "free". http://tweetnacl.cr.yp.to http://tweetnacl.cr.yp.to My guess (and hope) is that pathological requests for "features" to be added would be met with heavy scrutiny. The authors already have day jobs in academia.

deleted 12y ago

[deleted]

x0x0 12y ago

It's well and fine that Stephen lives very cheaply, but all of this is an attempt to distract from the OpenSSL project's very real issues by wearing a cilice then bitching about it. The fundamental facts are these: openssl contains a large quantity of code that, if I where to check into my company's repo, I would have at best a rough conversation with the cto and at worst I'd get fired. Plus a lack of good tests. These combine to create more than hypothetical problems; we've seen some severe security holes and there's almost certainly more to come. The question that should be discussed is if openssl is, ala sendmail, unsuitable for purpose and, if so, what should it be replaced with.

dfc 12y ago

The tor project is a great example of how open source software (OSS) projects can work with sponsors. Trying to find more information on Qualys and PSW Groups sponsorship of openssl is a nightmare compared to tor project sponsors.[^1][^2] Without the tor project's emphasis on transparency and professionalism I doubt they could post numbers like this: Since meeting the revenue milestones of $1,253,241 in 2009, $1,574,119 in 2010 and $1,681,101 in 2011, Tor has reached new heights in 2012 with over $2 million in revenue (unaudited).[^3] My comment should not be read as a criticism of OpenSSL, it should be interpreted as cause for optimism. The tor project has demonstrated that OSS projects can get sponsored to solve complicated security problems that are difficult to explain to the general public. [^1]: List of sponsors/projects: https://trac.torproject.org/projects/tor/wiki/org/sponsors https://trac.torproject.org/projects/tor/wiki/org/sponsors [^2]: Example monthly report for SponsorF's project: https://lists.torproject.org/pipermail/tor-reports/2014-March/000484.html https://lists.torproject.org/pipermail/tor-reports/2014-Marc... [^3]: Tor Project Annual Report 2012, pg 8, https://www.torproject.org/about/findoc/2012-TorProject-Annual-Report.pdf https://www.torproject.org/about/findoc/2012-TorProject-Annu...

piercebot 12y ago

Reading about Dr. Stephen Henson reminded me of the article written about Tarn Adams, the creator of Dwarf Fortress. http://www.nytimes.com/2011/07/24/magazine/the-brilliance-of-dwarf-fortress.html?pagewanted=all&_r=0 http://www.nytimes.com/2011/07/24/magazine/the-brilliance-of...

conductr 12y ago

Not hard. Just change the license and require commercial use to be paid