5 ms·
The State of SSL Stacks
- kccqzy 1y agoI have no idea this happened, but after reading this article my main takeaway was that OpenSSL shot themselves in the foot. Is this a fair assessment? I mean the excessive locking just stands out to be ill-advised for this performance-sensitive piece of code.
- owenthejumper 1y agoYes
- edelbitter 1y ago[flagged]
- mplanchard 1y ago[flagged]
- dugite-code 1y agoIMHO It reads like a corporate communication that's been re-worked 3 or 4 times by several people. AI generated content always sounds like this because It's some of the most readily available content to train them on.
- deleted 1y ago[deleted]
- tptacek 1y agoWilly Tarreau is the original author of HAProxy.
- 2bluesc 1y agoNever heard of aws-lc before this, but now I'm looking for an excuse to use it.
- juangacovas 1y agoIndeed, made some preliminary tests under RHEL 9 (Rocky, etc) for example and if you're used to compile HAProxy from sources to use specific OpenSSL versions, testing "aws-lc" is fairly straightforward. Their BUILD instructions and INSTALL file from HAProxy also help.
- rlnorthcutt 1y agoThere is also a RHEL/CentOS 8/9 + QuicTLS / AWS-LC package available to test out or start with. https://github.com/haproxy/wiki/wiki/Packages https://github.com/haproxy/wiki/wiki/Packages
- jaas 1y ago> We also tested Rustls and its rustls-openssl-compat layer. Rustls could be an interesting library in the future, but the OpenSSL compatibility application binary interface (ABI) was not complete enough to make it work correctly with HAProxy in its current state. Rustls is actively working on improving the OpenSSL compatibility layer. Hopefully we'll have it fully working for HAProxy soon! We've also invested a lot in performance. Next week we'll be publishing a blog post about Rustls server side performance. Since it's relevant to discussions about TLS stack performance, here's a preview: https://docs.google.com/document/d/1xFoRjb7pn4ZtL5BH7_ZwXNgNpSCLPiYAIQ1BcSalY9w/edit?usp=sharing https://docs.google.com/document/d/1xFoRjb7pn4ZtL5BH7_ZwXNgN...
- pixelesque 1y agoI've been working on a project of mine which makes web requests via libCURL, and the number of memory allocations OpenSSL makes for a single TLS connection is astonishing - running it through `heaptrack` was a real eye-opener. Discussion I found about other people mentioning it: https://github.com/openssl/openssl/discussions/26659 https://github.com/openssl/openssl/discussions/26659 In some cases (not all) for my workflows perf record traces show the allocation / deallocation overhead is quite significant, especially in a multi-threaded setup, where contention against the system allocator starts to become a problem in some situations.
- wtarreau 1y agoAbsolutely. Sometimes when using OpenSSL in performance tests, you notice that performances vary significantly just by switching to a different memory allocator, which is totally scary. I hadn't seen the conversation above, thanks for the pointer. It's surrealistic. I don't see how having to support multiple file formats requires to invest so many allocations. In the worst case you open the file (1 malloc and occasionally a few realloc) and you try to parse it into a struct using a few different decoders. I hope they're not allocating one byte at a time when reading a file...
- variadix 1y agoI’ve noticed a similar issue in a different crypto library before (mbedTLS), IIRC their MPI implementation allocated and deallocated _a lot_ of tiny allocations during ECC operations.
- toast0 1y ago> Roman Arutyunyan from NGINX core team were the first to propose a solution with a clever method that abuses the keylog callback to make it possible to extract or inject the required elements, and finally make it possible to have a minimal server-mode QUIC support. This sounds like future pain waiting to happen. My experience with callbacks in Openssl 1.0.x is that letter releases may significantly alter or remove callbacks. In the long term, I guess it worked out, because I figured out a better way to do what I was trying to do and got it accepted into OpenSSL and that fixed things for everyone, but in the short term it was a PITA and I'd have to decide between out of date OpenSSL with my feature working and updating OpenSSL to address whatever security problem of the day but my feature not working while I take a week to figure out how to do it in the new world. In my case, the feature was doing DHE-RSA with Microsoft Schannel and not causing an 'out of memory' error on the client when the server public key has 8 or more bits of zeros at the high end. The client didn't actually run out of memory, but that was the error it reported. Multithreading can be hard; when I had a problem for HAProxy to solve, 1.8 was just out and using multiple processes was much better for my application than threads. Configuration for multiple processes was more difficult, of course, but one time pain in order to get many multiples of throughput was worth it (and I didn't need any shared state between processes, so I didn't lose functionality... not everyone has that liberty) But I gather multithreaded HAProxy has gotten a lot better; and my problem went away and anyway I left the company whose problem I was solving with HAProxy anyway; so I don't have current knowledge. But the description of OpenSSL is describing a multithreading nightmare: > With OpenSSL 3.0, an important goal was apparently to make the library much more dynamic, with a lot of previously constant elements (e.g., algorithm identifiers, etc.) becoming dynamic and having to be looked up in a list instead of being fixed at compile-time. Since the new design allows anyone to update that list at runtime, locks were placed everywhere when accessing the list to ensure consistency. These lists are apparently scanned to find very basic configuration elements, so this operation is performed a lot. I'm not sure this kind of thing needs to be a fixed list at compile time (although it has worked for decades, yeah?); having a way to load a config and fixing the config before sharing it would be a good way to avoid the need for locking. If it does need to be changable at runtime, it can still be done with performance... an instance of configuration should be immutable, then you can have a shared pointer to the current config. At critical points, you might copy the current pointer or something. There's techniques and prior art for this kind of stuff.
- dur-randir 1y agoFor anyone wondering, have fun reading responses to those: - https://github.com/openssl/openssl/issues/20286 https://github.com/openssl/openssl/issues/20286 - https://github.com/openssl/openssl/issues/18814 https://github.com/openssl/openssl/issues/18814 - https://github.com/openssl/openssl/issues/16791 https://github.com/openssl/openssl/issues/16791 - https://github.com/openssl/openssl/issues/17950 https://github.com/openssl/openssl/issues/17950
- DyslexicAtheist 1y ago> Modern implementations must support a range of TLS protocol versions (from legacy TLS 1.0 to current TLS 1.3) So this statement is strange considering "modern" security standards either nudge you (or demand) to deprecate anything that isn't v1.3 or v1.2. If the implementation is "modern" why would I allow 1.0 ? This seems like a HA-Proxy problem. They ought to maintain support for geriatric TLS versions on a dedicated release branch connected to a support-model that nudges their client into updating by increasing their fees for maintaining that system. Not doing so means the vendor is part of the problem why we have slower adoption rates for 1.3 than we could otherwise have. It would have been cool to see AWS's s2n-tls (or s2n-quic https://github.com/aws/s2n-quic https://github.com/aws/s2n-quic) included in their benchmark. One of my all time favorite episode from the SCW podcast goes into the design decisions of s2n: The feeling's mutual: mTLS with Colm MacCárthaigh https://securitycryptographywhatever.com/2021/12/29/the-feeling-s-mutual-mtls-with-colm-maccarthaigh/ https://securitycryptographywhatever.com/2021/12/29/the-feel... From AWS: https://aws.amazon.com/security/opensource/cryptography/ https://aws.amazon.com/security/opensource/cryptography/ > "In 2015, AWS introduced s2n-tls, a fast open source implementation of the TLS protocol. The name "s2n", or "signal to noise," refers to the way encryption masks meaningful signals behind a facade of seemingly random noise. Since then, AWS has launched several other open source cryptographic libraries, including Amazon Corretto Crypto Provider (ACCP) and AWS Libcrypto (AWS-LC). AWS believes that open source benefits everyone, and we are committed to expanding our cryptographic and transport libraries to meet the evolving security needs of our customers." Here is a pdf that provides some performance results for s2n (sadly not s2n-quic): "Performance Analysis of SSL/TLS Crypto Libraries: Based on Operating Platform" https://bhu.ac.in/research_pub/jsr/Volumes/JSR_66_02_2022/12.pdf https://bhu.ac.in/research_pub/jsr/Volumes/JSR_66_02_2022/12...
- owenthejumper 1y agoAFAIK haproxy does not charge their users increased fees for legacy TLS. You would be shocked how much legacy software there is, requiring TLS 1.0. Not saying that is a good thing, just a reality…
- eqvinox 1y ago> If the implementation is "modern" why would I allow 1.0 ? Because there's a distinction between "using" (especially by default) and "implementing". The real world has millions (billions?) of devices that don't receive updates and yet still need to be talked to, in most cases luckily by a small set of communication partners. Would you rather have even the "modern" side of that conversation be forced to use some ancient SSL library? I'd rather have modern software, even if I'm forced to use an older protocol version by the other endpoint. Just disable it by default. And it's not like TLS 1.0 and 1.1 are somehow worse than cleartext communication. They're still encrypted transport protocols that take significant effort to break. That you shouldn't use them if at all possible doesn't mean that you can't use them if anything else is impossible.
- c0l0 1y agoWow... reading this article in full really made me lose hope in OpenSSL, the project and the library. I was well aware of the expected inconveniences any new major OpenSSL release would trigger (esp. older, less actively maintained applications having to adapt their API usage to keep working) going in, but what the linked github issue/PR comments hint at is just... mental. As best illustrated by https://github.com/openssl/openssl/issues/20286#issuecomment-1547053752 https://github.com/openssl/openssl/issues/20286#issuecomment... not only seem the core developers not care about runtime performance at all, they also seem to have a completely absurd perception of their own project, esp. in relation to other, genuinely small FOSS projects. It's just wild. I hope this can still be turned around, and OpenSSL can somehow re-emerge from this clusterfuck as the stable bedrock of web transport security that we learned to rely on from 2014 onwards.
- Hilift 1y ago[flagged]
- wobfan 1y agoTIL that committing a bug disqualifies me from getting a PhD. Lucky that I stopped with my masters degree, that would’ve been a bad surprise
- LtWorf 1y agoI hope there's no bugs in any of your projects or we might have to cancel your master's as well. Maybe even your high school diploma.
- justinrubek 1y agoWell, at least you did a pretty good job of dismissing the point in the comment above you. Maybe that counts for something.
- iforgotpassword 1y agoI don't know how it happens but sometimes very old OSS projects turn into those bikeshedding projects completely disconnected from reality. It only serves the self-fulfillment of the developers. I've recently ranted about libgd here in another comment. Definitely not as bad, definitely not as mission critical as openssl, but same symptoms.
- jeffbee 1y agoIt seems weird to disqualify boringssl due to live-at-head project philosophy. Obviously _your_ project doesn't have to change every time boringssl changes.
- toast0 1y agoThe only way that really works is if HAProxy includes the version of BoringSSL they want to use in their source tree. Otherwise, it's too hard for users to find the right version. But you quickly get to the hard problems of when there are security fixes for BoringSSL. Upstream is only going to fix it on head per their philosophy. So either someone has to backport the fix to whatever version HAProxy uses on their supported versions, or HAProxy has to update to head with whatever gymnastics required on all their supported versions. In the meantime, it's probably more difficult for users to understand if their system has a vulnerable version of boringssl, because it's embedded into HAProxy, and not a separate package. Given that, it makes a lot of sense to not consider it, when AWS-LC is a fork of BoringSSL that's intended to be used by 3rd party projects.
- jeroenhd 1y agoTheir project has LTS versions that need to support the stuff that was supported on the date the release was made. BoringSSL doesn't do versioned/LTS releases, so their code happily moves on, deprecating/altering/removing features as they see fit. You could do fake LTS releases of BoringSSL, manually backporting features and fixes, but that's a lot of work with little pay-off. Once a feature gets removed at HEAD and bugs/vulnerabilities/whatever get found in that feature, you're stuck figuring it out yourself, and altering all future code that the feature depended on to keep it compatible. Combining live-at-HEAD and LTS versions is a pain. There's nothing wrong with either approach, but combining the two is asking for trouble, especially for libraries where you can't just pin a commit and hope for the best like core security libraries.
- yencabulator 1y agoThe State of C SSL Stacks
- drob518 1y agoWell, that was an extremely thorough and well-documented take down of OpenSSL’s issues, both technical and organizational. The OpenSSL team should be embarrassed. What’s your side of the story, OpenSSL?
- jedisct1 1y agoOfficial OpenSSL benchmarks: https://openssl-library.org/performance.html https://openssl-library.org/performance.html OpenSSL 3.0 was catastrophic, but it looks like OpenSSL 3.5 isn't too bad.
- rlnorthcutt 1y agoI agree that it is great to see the improvements. However, in the absence of comparisons to other libraries, it is tough to see the real impact. Even a comparison to the 1.x version would be helpful. The concern is that relative to 3.0, it is improved... but relative to 1.x or a different library, it is not scalable.
- wtarreau 1y agoThe problem is not the performance of the low-level crypto code IMHO, but how it interfaces with the rest, which is where you're crossing a myriad of locks (and atomic ops for newer versions) that cost a lot as soon as you're interested in using more than one CPU core :-/
- Ros23 1y agoLibreSSL is the answer.