7 ms·

If the cloud companies just paid a team of five people 200K each to spend a year rewriting OpenSSL from scratch, they would save multiple millions in scrambling

If the cloud companies just paid a team of five people 200K each to spend a year rewriting OpenSSL from scratch, they would save multiple millions in scrambling to deploy bug fixes.

LeonM 6y ago
Just pay a bunch of talented security researchers to find the bugs. Also donate 200k/year to the OpenSSL team so they have the capacity to fix the bugs whenever found.
- dt3ft 6y ago
  If only the billionaires would listen to you... (Mind you, I fully agree with you here.)
planckscnst 6y ago
Like s2n? https://aws.amazon.com/blogs/security/introducing-s2n-a-new-open-source-tls-implementation/ https://aws.amazon.com/blogs/security/introducing-s2n-a-new-...
- cryptonector 6y ago
  That doesn't replace all the PKIX bits from OpenSSL.
dijit 6y ago
There have been attempts of repairing openssl. Every time it's a big problem due to lack of adoption. LibreSSL would solve a lot of the surface area problems OpenSSL has, but people cling to OpenSSL under the guise that "a lot has improved" and of course that change is hard. The point I'm trying to make is that there _are_ better SSL libraries, but none have the adoption of openssl, and adoption has inertia; nobody is going to go out of their way to avoid OpenSSL because a bug in openssl is not their fault. Just like AWS's availability, people work on the premise that if AWS is down (or OpenSSL has bug!) then everyone is affected and therefore they can't be blamed.
- jamal-kumar 6y ago
  I've been making personal moves towards libressl and openbsd in general. there's just way less friction involved in maintaining those servers vs my linux ones running stuff like this that constantly leaves me hitting up my clients like 'yeah we need to do this at midnight' If libressl is affected by this too I would be incredibly surprised. They really did do a great job of removing all the bad stuff for something that just works as a drop-in replacement compatible with scripts and everything.
  - j16sdiz 6y ago
    > a drop-in replacement It is not. Many cert attribute related function are missing.
    - jamal-kumar 6y ago
      Damn that sucks. Works for me on literally all my client's sites otherwise, I feel happy as a clam... If it doesn't work for you I'm really curious on what's missing though, could you point out specifics?
- dadrian 6y ago
  LibreSSL hasn't improved anything. Deleting all the code that was #ifdef'd out for old platforms might make you feel good, but it doesn't actually help security because none of the code was compiled anyway.
  - dijit 6y ago
    This does a great disservice to the work done by the OpenBSD guys. For one thing they removed the home-grown memory allocator, which prevented a lot of issues and allowed debugging tools to notice memory corruption issues.
    - 77pt77 6y ago
      > allowed debugging tools to notice memory corruption issues You mean like the bug debian introduced here[1]? [1] https://www.debian.org/security/2008/dsa-1571 https://www.debian.org/security/2008/dsa-1571
gregoriol 6y ago
Doing security is hard, it wouldn't be as easy as you describe to replace it! OpenSSL has some issues from time to time, but like any code that exists. However, it also benefits from billions of real-world use-cases and deployments, research work and analysis, that would be very hard to reproduce on an entierly new code base.
rapsey 6y ago
BoringSSL is now very widely used. There is also rustls (fully safe rust code) with OpenSSL API available.
- earthboundkid 6y ago
  Yeah. Thinking about it seriously, the problem is that if you do a “drop in replacement” you’ve just recreated all the existing bugs. If you don’t do a drop in replacement, adoption will be poor, see BoringSSL etc. It’s sort of no-win.
  - jeffbee 6y ago
    Considering that BoringSSL is the library in Chrome, Android, iOS, and Google's GFE, I imagine it's adoption is not "poor". It is probably the most widespread TLS implementation, by traffic.
    - rapsey 6y ago
      Also cloudflare
    - earthboundkid 6y ago
      Okay, but today I upgraded my Ubuntu box which had the OpenSSL bug on it. Why does my Ubuntu box use OpenSSL? I haven't looked into it, but I imagine Python is using it and God knows what else. Can those things move to BoringSSL? Probably, long term, but they haven't yet because it's non-trivial. No easy answer here. It will probably just take a decade of slowly moving everything using OpenSSL onto something else, piece by piece.
justapassenger 6y ago
Complete rewrite of decades old software that’s powering all of world is many orders of magnitude more expensive than what you suggest. And only thing you’re guaranteed is bazillions of incompatibilities, bugs, security issues and other horrible stuff. Especially with such a sensitive thing like crypto. Just off the top - you have to be really careful with implementation to prevent timing attacks. And that’s just one of many hundreds things you need to care about.
- IgorPartola 6y ago
  But what if we rewrite it all in Rust bugs become impossible! /s
  - unilynx 6y ago
    Moving all ASN/X509 handling to Rust might be worth it though. That stuff is a minefield and has had plenty of memory-safety issues in the past.
    - mroche 6y ago
      Genuine question as I'm not a Rust professional, but if one was able to maintain the C API and headers to produce a drop in replacement for the current openssl implementation, would rewriting it in Rust produce a safer variant? This would be based off using the #[no_mangle] extensively, I presume, probably with some amount of "unsafe" usage? At which point has the primary use case of Rust in this situation been lost?
      - bluejekyll 6y ago
        What you're probably looking for is this project, which is a C API (I think ABI compatible) wrapper around Rustls that can be a "drop-in" replacement for OpenSSL: https://mesalink.io/ https://mesalink.io/ (note: not commenting on if one should or should not use it, only mentioning it's existence)
  - bluejekyll 6y ago
indymike 6y ago
Nope, they'd just create new software with different bugs that have not been discovered yet. Then we'd all be scrambling to fix those bugs.
- wizeman 6y ago
  Not if the implementation is formally verified, like miTLS [0] and EverCrypt [1]. Parts of the latter were integrated into Firefox, which even provided a performance boost (10x in one case) [2]. I think what is needed is something like EverCrypt but for TLS. Or in other words, something like miTLS but which extracts to C and/or Assembly, to avoid garbage collection and for easy interoperation with different programming languages (preferably including an OpenSSL-compatible API for backwards compatibility). [0]: https://mitls.org/ https://mitls.org/ [1]: https://hacl-star.github.io/HaclValeEverCrypt.html https://hacl-star.github.io/HaclValeEverCrypt.html [2]: https://blog.mozilla.org/security/2020/07/06/performance-improvements-via-formally-verified-cryptography-in-firefox/ https://blog.mozilla.org/security/2020/07/06/performance-imp...
  - j16sdiz 6y ago
    Formally verified against what? and what assumption were made? miTLS is formally verified against the TLS spec on handshaking, assuming the lower-level crypto routine are good. It is not even free from timing attack. EverCrypt have stronger proof, but it is only safe as in not crashing and correct as in match the spec in all valid input. It is not proved to be free from DoS or invalid input attack. OpenSSL do more then TLS. Lots of interesting thing are in the cert format parsing and management.
    - wizeman 6y ago
      > Formally verified against what? and what assumption were made? Well, tell me again, what is OpenSSL formally verified against? What assumptions were made in OpenSSL? Formal verification does not eliminate very large classes of bugs only when absolutely everything is formally verified, including timing attacks. Instead, it consistently produces more reliable software, many times even when only certain basic properties are proved (such as, memory safety, lack of integer overflows or division by zero, etc). Formal verification can be a continuum, like testing, but proven to work for all inputs (that possibly meet certain conditions, under certain assumptions). The assumptions and properties that are proven can always be strengthened later, as seen in multiple real-world projects (such as seL4 and others). The result is code that is almost always a lot more bug-free than code that is not formally verified. And as I said, more properties can be proven over time, especially as new classes of attacks are discovered (e.g. timing attacks, speculation attacks in CPUs, etc).

If the cloud companies just paid a team of five people 200K each to spend a year rewriting OpenSSL from scratch, they would save multiple millions in scrambling to deploy bug fixes.

LeonM 6y ago

Just pay a bunch of talented security researchers to find the bugs. Also donate 200k/year to the OpenSSL team so they have the capacity to fix the bugs whenever found.

dt3ft 6y ago
If only the billionaires would listen to you... (Mind you, I fully agree with you here.)

planckscnst 6y ago

Like s2n? https://aws.amazon.com/blogs/security/introducing-s2n-a-new-open-source-tls-implementation/ https://aws.amazon.com/blogs/security/introducing-s2n-a-new-...

cryptonector 6y ago
That doesn't replace all the PKIX bits from OpenSSL.

dijit 6y ago

There have been attempts of repairing openssl. Every time it's a big problem due to lack of adoption. LibreSSL would solve a lot of the surface area problems OpenSSL has, but people cling to OpenSSL under the guise that "a lot has improved" and of course that change is hard. The point I'm trying to make is that there _are_ better SSL libraries, but none have the adoption of openssl, and adoption has inertia; nobody is going to go out of their way to avoid OpenSSL because a bug in openssl is not their fault. Just like AWS's availability, people work on the premise that if AWS is down (or OpenSSL has bug!) then everyone is affected and therefore they can't be blamed.

jamal-kumar 6y ago
I've been making personal moves towards libressl and openbsd in general. there's just way less friction involved in maintaining those servers vs my linux ones running stuff like this that constantly leaves me hitting up my clients like 'yeah we need to do this at midnight' If libressl is affected by this too I would be incredibly surprised. They really did do a great job of removing all the bad stuff for something that just works as a drop-in replacement compatible with scripts and everything.
- j16sdiz 6y ago
  > a drop-in replacement It is not. Many cert attribute related function are missing.
  - jamal-kumar 6y ago
    Damn that sucks. Works for me on literally all my client's sites otherwise, I feel happy as a clam... If it doesn't work for you I'm really curious on what's missing though, could you point out specifics?
dadrian 6y ago
LibreSSL hasn't improved anything. Deleting all the code that was #ifdef'd out for old platforms might make you feel good, but it doesn't actually help security because none of the code was compiled anyway.
- dijit 6y ago
  This does a great disservice to the work done by the OpenBSD guys. For one thing they removed the home-grown memory allocator, which prevented a lot of issues and allowed debugging tools to notice memory corruption issues.
  - 77pt77 6y ago
    > allowed debugging tools to notice memory corruption issues You mean like the bug debian introduced here[1]? [1] https://www.debian.org/security/2008/dsa-1571 https://www.debian.org/security/2008/dsa-1571

gregoriol 6y ago

Doing security is hard, it wouldn't be as easy as you describe to replace it! OpenSSL has some issues from time to time, but like any code that exists. However, it also benefits from billions of real-world use-cases and deployments, research work and analysis, that would be very hard to reproduce on an entierly new code base.

rapsey 6y ago

BoringSSL is now very widely used. There is also rustls (fully safe rust code) with OpenSSL API available.

earthboundkid 6y ago
Yeah. Thinking about it seriously, the problem is that if you do a “drop in replacement” you’ve just recreated all the existing bugs. If you don’t do a drop in replacement, adoption will be poor, see BoringSSL etc. It’s sort of no-win.
- jeffbee 6y ago
  Considering that BoringSSL is the library in Chrome, Android, iOS, and Google's GFE, I imagine it's adoption is not "poor". It is probably the most widespread TLS implementation, by traffic.
  - rapsey 6y ago
    Also cloudflare
  - earthboundkid 6y ago
    Okay, but today I upgraded my Ubuntu box which had the OpenSSL bug on it. Why does my Ubuntu box use OpenSSL? I haven't looked into it, but I imagine Python is using it and God knows what else. Can those things move to BoringSSL? Probably, long term, but they haven't yet because it's non-trivial. No easy answer here. It will probably just take a decade of slowly moving everything using OpenSSL onto something else, piece by piece.

justapassenger 6y ago

Complete rewrite of decades old software that’s powering all of world is many orders of magnitude more expensive than what you suggest. And only thing you’re guaranteed is bazillions of incompatibilities, bugs, security issues and other horrible stuff. Especially with such a sensitive thing like crypto. Just off the top - you have to be really careful with implementation to prevent timing attacks. And that’s just one of many hundreds things you need to care about.

IgorPartola 6y ago
But what if we rewrite it all in Rust bugs become impossible! /s
- unilynx 6y ago
  Moving all ASN/X509 handling to Rust might be worth it though. That stuff is a minefield and has had plenty of memory-safety issues in the past.
  - mroche 6y ago
    Genuine question as I'm not a Rust professional, but if one was able to maintain the C API and headers to produce a drop in replacement for the current openssl implementation, would rewriting it in Rust produce a safer variant? This would be based off using the #[no_mangle] extensively, I presume, probably with some amount of "unsafe" usage? At which point has the primary use case of Rust in this situation been lost?
    - bluejekyll 6y ago
      What you're probably looking for is this project, which is a C API (I think ABI compatible) wrapper around Rustls that can be a "drop-in" replacement for OpenSSL: https://mesalink.io/ https://mesalink.io/ (note: not commenting on if one should or should not use it, only mentioning it's existence)
- bluejekyll 6y ago

indymike 6y ago

Nope, they'd just create new software with different bugs that have not been discovered yet. Then we'd all be scrambling to fix those bugs.

wizeman 6y ago
Not if the implementation is formally verified, like miTLS [0] and EverCrypt [1]. Parts of the latter were integrated into Firefox, which even provided a performance boost (10x in one case) [2]. I think what is needed is something like EverCrypt but for TLS. Or in other words, something like miTLS but which extracts to C and/or Assembly, to avoid garbage collection and for easy interoperation with different programming languages (preferably including an OpenSSL-compatible API for backwards compatibility). [0]: https://mitls.org/ https://mitls.org/ [1]: https://hacl-star.github.io/HaclValeEverCrypt.html https://hacl-star.github.io/HaclValeEverCrypt.html [2]: https://blog.mozilla.org/security/2020/07/06/performance-improvements-via-formally-verified-cryptography-in-firefox/ https://blog.mozilla.org/security/2020/07/06/performance-imp...
- j16sdiz 6y ago
  Formally verified against what? and what assumption were made? miTLS is formally verified against the TLS spec on handshaking, assuming the lower-level crypto routine are good. It is not even free from timing attack. EverCrypt have stronger proof, but it is only safe as in not crashing and correct as in match the spec in all valid input. It is not proved to be free from DoS or invalid input attack. OpenSSL do more then TLS. Lots of interesting thing are in the cert format parsing and management.
  - wizeman 6y ago
    > Formally verified against what? and what assumption were made? Well, tell me again, what is OpenSSL formally verified against? What assumptions were made in OpenSSL? Formal verification does not eliminate very large classes of bugs only when absolutely everything is formally verified, including timing attacks. Instead, it consistently produces more reliable software, many times even when only certain basic properties are proved (such as, memory safety, lack of integer overflows or division by zero, etc). Formal verification can be a continuum, like testing, but proven to work for all inputs (that possibly meet certain conditions, under certain assumptions). The assumptions and properties that are proven can always be strengthened later, as seen in multiple real-world projects (such as seL4 and others). The result is code that is almost always a lot more bug-free than code that is not formally verified. And as I said, more properties can be proven over time, especially as new classes of attacks are discovered (e.g. timing attacks, speculation attacks in CPUs, etc).