3 ms·

Sorry I was not aware that Glibc malloc is also BSD licensed, thanks for providing more information about this. Given that, to me it is a big mystery why they d

by antirez 13y ago

Sorry I was not aware that Glibc malloc is also BSD licensed, thanks for providing more information about this. Given that, to me it is a big mystery why they don't migrate to something more reasonable like jemalloc, which is able to provide less fragmentation and top performances. Jemalloc is also very actively developed.

lomnakkus 13y ago
I think this comes under the heading of squabbling over details. To quote from your blog post: "This is a sign that still performances, even in security critical code, is regarded with too much respect over safety. In this specific instance, it must be admitted that probably when the OpenSSL developers wrapped malloc, they never though of security implications by doing so. However the fact that they cared about a low-level detail like the allocation functions in some system is a sign of deep concerns about performances, while they should be more deeply concerned about the correctness / safety of the system." As you stated in this bit of your post this is a sign that the OpenSSL team have their priorities backward. In other words they are NOT fit to run a security critical project. Period.
- antirez 13y ago
  I believe the stress on performances is largely community-driven in a project like OpenSSL. A lot of environments did not switched to encrypted connections because of cost concerns and unfortunately costs are related to OpenSSL performances. So it's not like they are crazy and mess with priorities IMHO. But it is time to re-evaluate.
  - lomnakkus 13y ago
    Don't get me wrong -- performance can matter, but in the end it's up to the user of your cryptographic solution/process/abstraction to decide what tradeoff is acceptable to them. If you're responsible about it, you try to explain to potential users what the tradeoffs are -- you don't try to optimize before you can be sure that you're doing so sensibly (see next paragraph). Of course you should try to optimize as much as is sensible, but if you don't have any process in place (unit tests, integration tests, fuzz testing, formal methods, timing attacks, whatever you can throw at it) that lend some assurance that Implementation#1 (optimized) actually does exactly the same as Implementation#0 (less or unoptimized) under all circumstances, then you should be as conservative as humanly possible... and choose to NOT optimize unless/before you have those processes in place. (I'll happily admit that this is somewhat armchairy -- I'm glad I don't work in a particularly security sensitive area. I'm not sure I could handle all the obligations/pressures myself.)
- TheCondor 13y ago
  Historically, performance has ALWAYS been a major security issue. To the point that it's been claimed to be easy to build highly secure ciphers and tools, it's difficult to build them that perform well. Performance was a major factor in the AES and SHA selection processes. Performance is a major factor in the interest in ECC. Arguably, performance is why you'd ever use C for anything at all. Seems easy to dismiss them and say their priorities are backwards, fundamentally at the end of the day, a huge majority of the users will simply pick the benchmark winner and use it. It a free project and their priorities are what they are, what are the priorities of the legions that used it instead of writing their own?
TheCondor 13y ago
I don't think the Linux malloc implementation is why OpenSSL has some of it's own memory management stuff built in. I think it's more due to the wide range of other platforms it runs on, including some embedded ones that have very strict allocation policies. jemalloc may very well be a good thing for Linux in general but I don't think it would have changed this.
- antirez 13y ago
  Yes I don't want to imply that glibc malloc was the culprit here, but glibc malloc is one of the reasons why certain systems have their own malloc / slab allocator / whatever. About the embedded systems, usually you don't try to cover everything, it is often a more valid a approach to have specialized libraries for very resource constrained systems. In the case of OpenSSL I guess the problem is that the implementation was hard enough to force OpenSSL to blend instead. This is where to be able to say "no" is a good idea.
xyzzy123 13y ago
At least currently, jemalloc does not have the kind of heap protection you would want for a system allocator :(
- padenot 13y ago
  What would be the heap protection features you need? [1] lists a couple features that I find useful. Of course, in Firefox, we use valgrind and ASAN extensively in automation and during development, so it matters less for us. (edit: to reduce confusion, I just want to say that Firefox uses jemalloc, otherwise my comment makes no sense. It's however disabled if you compile Firefox for use with valgrind). [1]: https://github.com/jemalloc/jemalloc/wiki/Use-Case%3A-Find-a-memory-corruption-bug https://github.com/jemalloc/jemalloc/wiki/Use-Case%3A-Find-a...
  - xyzzy123 13y ago
    Those are the debugging features. Those features (plus the tools you listed) help you during the test phase of your SDLC. In contrast, heap protection / exploit mitigation is intended to protect your software in the field from the bugs you did not know about until after the software shipped. Correct software would not need ASLR, stack canaries, NX or heap protection. But these things are widely accepted as required these days. Desirable features are mainly heap overflow protection and double free protection (see [1] for a list of what ptmalloc does). The situation is not entirely simple because the specific protections you want (say in terms of heap metadata protection) depend on the internal design of the allocator and there are a lot of devils in the details. There is some information around on jemalloc exploitation [2] [3], but I do not know if that information is current. I'm not aware of any jemalloc hardening which has happened since then. If you could provide any information in that area I'd be interested to know about it. [1] https://wiki.ubuntu.com/Security/Features#Heap_Protector https://wiki.ubuntu.com/Security/Features#Heap_Protector [2] http://phrack.org/issues/68/10.html#article http://phrack.org/issues/68/10.html#article [3] https://github.com/argp/unmask_jemalloc https://github.com/argp/unmask_jemalloc
mwcampbell 13y ago
A bit of a tangent, but have you looked at musl libc's malloc? It's way simpler, primarily because it doesn't use per-thread heaps, yet its author claims that it's competitive with jemalloc on performance. I'm inclined to favor the simpler malloc.

by antirez 13y ago

lomnakkus 13y ago

I think this comes under the heading of squabbling over details. To quote from your blog post: "This is a sign that still performances, even in security critical code, is regarded with too much respect over safety. In this specific instance, it must be admitted that probably when the OpenSSL developers wrapped malloc, they never though of security implications by doing so. However the fact that they cared about a low-level detail like the allocation functions in some system is a sign of deep concerns about performances, while they should be more deeply concerned about the correctness / safety of the system." As you stated in this bit of your post this is a sign that the OpenSSL team have their priorities backward. In other words they are NOT fit to run a security critical project. Period.

antirez 13y ago
I believe the stress on performances is largely community-driven in a project like OpenSSL. A lot of environments did not switched to encrypted connections because of cost concerns and unfortunately costs are related to OpenSSL performances. So it's not like they are crazy and mess with priorities IMHO. But it is time to re-evaluate.
- lomnakkus 13y ago
  Don't get me wrong -- performance can matter, but in the end it's up to the user of your cryptographic solution/process/abstraction to decide what tradeoff is acceptable to them. If you're responsible about it, you try to explain to potential users what the tradeoffs are -- you don't try to optimize before you can be sure that you're doing so sensibly (see next paragraph). Of course you should try to optimize as much as is sensible, but if you don't have any process in place (unit tests, integration tests, fuzz testing, formal methods, timing attacks, whatever you can throw at it) that lend some assurance that Implementation#1 (optimized) actually does exactly the same as Implementation#0 (less or unoptimized) under all circumstances, then you should be as conservative as humanly possible... and choose to NOT optimize unless/before you have those processes in place. (I'll happily admit that this is somewhat armchairy -- I'm glad I don't work in a particularly security sensitive area. I'm not sure I could handle all the obligations/pressures myself.)
TheCondor 13y ago
Historically, performance has ALWAYS been a major security issue. To the point that it's been claimed to be easy to build highly secure ciphers and tools, it's difficult to build them that perform well. Performance was a major factor in the AES and SHA selection processes. Performance is a major factor in the interest in ECC. Arguably, performance is why you'd ever use C for anything at all. Seems easy to dismiss them and say their priorities are backwards, fundamentally at the end of the day, a huge majority of the users will simply pick the benchmark winner and use it. It a free project and their priorities are what they are, what are the priorities of the legions that used it instead of writing their own?

TheCondor 13y ago

I don't think the Linux malloc implementation is why OpenSSL has some of it's own memory management stuff built in. I think it's more due to the wide range of other platforms it runs on, including some embedded ones that have very strict allocation policies. jemalloc may very well be a good thing for Linux in general but I don't think it would have changed this.

antirez 13y ago
Yes I don't want to imply that glibc malloc was the culprit here, but glibc malloc is one of the reasons why certain systems have their own malloc / slab allocator / whatever. About the embedded systems, usually you don't try to cover everything, it is often a more valid a approach to have specialized libraries for very resource constrained systems. In the case of OpenSSL I guess the problem is that the implementation was hard enough to force OpenSSL to blend instead. This is where to be able to say "no" is a good idea.

xyzzy123 13y ago

At least currently, jemalloc does not have the kind of heap protection you would want for a system allocator :(

padenot 13y ago
What would be the heap protection features you need? [1] lists a couple features that I find useful. Of course, in Firefox, we use valgrind and ASAN extensively in automation and during development, so it matters less for us. (edit: to reduce confusion, I just want to say that Firefox uses jemalloc, otherwise my comment makes no sense. It's however disabled if you compile Firefox for use with valgrind). [1]: https://github.com/jemalloc/jemalloc/wiki/Use-Case%3A-Find-a-memory-corruption-bug https://github.com/jemalloc/jemalloc/wiki/Use-Case%3A-Find-a...
- xyzzy123 13y ago
  Those are the debugging features. Those features (plus the tools you listed) help you during the test phase of your SDLC. In contrast, heap protection / exploit mitigation is intended to protect your software in the field from the bugs you did not know about until after the software shipped. Correct software would not need ASLR, stack canaries, NX or heap protection. But these things are widely accepted as required these days. Desirable features are mainly heap overflow protection and double free protection (see [1] for a list of what ptmalloc does). The situation is not entirely simple because the specific protections you want (say in terms of heap metadata protection) depend on the internal design of the allocator and there are a lot of devils in the details. There is some information around on jemalloc exploitation [2] [3], but I do not know if that information is current. I'm not aware of any jemalloc hardening which has happened since then. If you could provide any information in that area I'd be interested to know about it. [1] https://wiki.ubuntu.com/Security/Features#Heap_Protector https://wiki.ubuntu.com/Security/Features#Heap_Protector [2] http://phrack.org/issues/68/10.html#article http://phrack.org/issues/68/10.html#article [3] https://github.com/argp/unmask_jemalloc https://github.com/argp/unmask_jemalloc

mwcampbell 13y ago

A bit of a tangent, but have you looked at musl libc's malloc? It's way simpler, primarily because it doesn't use per-thread heaps, yet its author claims that it's competitive with jemalloc on performance. I'm inclined to favor the simpler malloc.