10 ms·

Akamai release source to their custom secure_malloc for OpenSSL

267 points by nikcub 13y ago

nikcub 13y ago
For background, this came up as Akamai mentioned in their Heartbleed announcement that they didn't think they had any sensitive information[0] disclosed by the bug as they had wrapped OpenSSL's malloc to create two heaps: a secure heap for sensitive information like keys and an ordinary heap: https://blogs.akamai.com/2014/04/heartbleed-update.html https://blogs.akamai.com/2014/04/heartbleed-update.html It prompted a lot of interest in their solution, which prompted them to release the patch. edit: [0] by "sensitive information" we mean keys, if I understand the implementation correctly, you could still overrun into web server allocation and get back HTTP headers.
- gphilip 13y ago
  From reading that blog post it is not clear to me if their wrapping code existed prior to the discovery of the heartbleed bug: The post summary says: Akamai patched the announced Heartbleed vulnerability prior to its public announcement. We, like all users of OpenSSL, could have exposed passwords or session cookies transiting our network from August 2012 through 4 April 2014. Our custom memory allocator protected against nearly every circumstance by which Heartbleed could have leaked SSL keys. So: did the custom memory allocator exist already in August 2012? From reading the post this looks to be the case. Could it be that someone at Akamai took a look at the heartbeat (or other OpenSSL) code, decided that it could lead to memory leaks, and wrote their own memory allocator wrapper code to guard against this? Edit: SSL -> OpenSSL
  - computer 13y ago
    How I read it is that this code already protected their private key, but the Heartbleed bug still disclosed other private details (such as submitted user data).
  - alexkus 13y ago
    From the original post: > This patch is a variant of what we've been using to help protect customer keys for a decade. So it protects keys against heartbleed, but not other HTTP related data (urls, cookies, headers, etc).
  - Dylan16807 13y ago
    Keys are not passwords and session cookies.
  - yukichan 13y ago
    Akamai's patch is older than the heartbleed issue in OpenSSL so it wasn't a response to heartbleed, and they were still vulnerable for non-key data.
- brians 13y ago
  We definitely leaked session cookies and passwords, until we patched. This only protected SSL private keys.
  - ccurtsinger 13y ago
    You could do better with a probabilistically secure allocator instead: http://people.cs.umass.edu/~emery/pubs/ccs03-novark.pdf http://people.cs.umass.edu/~emery/pubs/ccs03-novark.pdf Randomized allocation makes it nearly impossible to forge pointers, locate sensitive data in the heap, and makes reuse unpredictable. This is strictly more powerful than ASLR, which does nothing prevent Heartbleed. Moving the base of the heap doesn't change the relative addresses of heap objects with a deterministic allocator. A randomized allocator does change these offsets, which makes it nearly impossible to exploit a heap buffer overrun (and quite a few other heap errors).
    - brians 13y ago
      That paper only seems to mention heap overflows for purposes of writing to a target object that will later be used for indirection or execution. I don't see how it makes Heartbleed any better to extract a shuffled heap instead of a sorted one. What am I missing?
      - ccurtsinger 13y ago
        It's not just a shuffled heap, it's also sparse. Section 4.1 covers heap overflow attacks, with an attacker using overflows from one object to overwrite entries in a nearby object's vtable. Because the objects could be anywhere in the sparse virtual address space, the probability of overwriting the desired object is very low (see section 6.2). The same reasoning applies to reads. If sensitive objects are distributed throughout the sparse heap, the probability of hitting a specific sensitive object is the same as the probability of overwriting the vtable in the above attack. The probability of reading out any sensitive object depends on the number of sensitive objects and the sparsity of the heap. There are also guard pages sprinkled throughout the sparse heap. Section 6.3.1 shows the minimum probability of a one byte overflow (read or write) hitting a guard page. This probability increases with larger objects and larger overflows. You can also increase sparsity to increase this probability, at a performance cost.
bello 13y ago
Disclaimer: I'm not too familiar with the code. "This arena is mmap'd, with guard pages before and after so pointer over- and under-runs won't wander into it." Doesn't that mean that this will only protect against overreads of a certain max length (such that the int16 length in heardbleed)? Seems like that wouldn't help with a length defined as a bigger int. I wonder if there's any better ways of doing this.
- ramidarigaz 13y ago
  The beginning and end of the allocated section of memory (the guard pages they refer to) are marked as PROT_NONE with mprotect, meaning that any access to them will cause a segfault. It's possible that a misbehaving process could jump straight into the unprotected memory, but it would have to not read from the guard pages at all. Buffer overruns don't have that problem (since they access memory sequentially), and would cause the program to crash before any sensitive data could be read (assuming the overrun starts outside the protected area).
  - bello 13y ago
    That makes sense, thank you!
- xyzzy123 13y ago
  The idea is that a read overrun where the base pointer starts in a different arena would have to cross the guard pages to "get to" the secure arena.
okasaki 13y ago
If OpenSSL had been GPL licenced, this would have been public a long time ago.
- zobzu 13y ago
  Actually as long as they don't distribute the code, even under GPL they don't have to share it.
  - dontmakemelaugh 13y ago
    You mean the application? They need to distribute the code if they provide applications based on the code.
  - midas007 13y ago
    Yup. It's pretty common for open source to be used, abused, modified in house and not shared. Just look at Sergey Aleynikov and Goldman Sachs. [0] Further I've seen a shop deploy hundreds of thousands of CentOS nodes to avoid RHEL license fees. Doesn't even begin to scratch the surface of shops not funding FOSS they exploit like OpenSSH, etc. Most of them are stingy, demanding, greedy bastards. [0] http://blog.garrytan.com/goldman-sachs-sent-a-brilliant-computer-scientist-to-jail-over-8mb-of-open-source-code-uploaded-to-an-svn-repo http://blog.garrytan.com/goldman-sachs-sent-a-brilliant-comp...
    - aspensmonster 13y ago
      >Most of them are stingy, demanding, greedy bastards. I'd go so far as to say "nearly all of them."
      - midas007 13y ago
        There's a few counterexamples, which is a good thing: iXSystems does a lot of cool community work via FreeNAS, PC-BSD and FreeBSD. Github (Ruby), thoughtbot (Ruby), Google (Go, Python), Dropbox (Python), AT&T (C++, Ruby), Zurb (Ruby, Foundation), Twitter (Ruby, Scala), ... missing a bunch and some platforms too.
jgrahamc 13y ago
I would be interested to know if this actually protects against heartbleed in the real world. I can see how it would keep the private key protected, but not sure if it protects intermediate values used during calculation. Given that it's likely that the private key can be derived from those parts, it would be good to know if this custom allocator is 'enough' (i.e. are the intermediate values stored in memory allocated using the same allocator)? This week has been full of surprises. Has anyone tested this?
- mrud 13y ago
  I didn't but akamai itself did [1]: To test this belief, our engineers have rebuilt every Akamai configuration since August 2012, when we upgraded to openssl-1.0.1. This included every kernel, hardware, and edge server software combination; and then a careful inspection of the ways in which memory was allocated, to see if any non-long-term memory allocations might border on our secure heap. Most of the configurations were proven safe; but we found one configuration that was not - there was an available memory block in range of the secure key store. This less safe configuration was active on our network for nine days in March 2013 [1] https://blogs.akamai.com/2014/04/heartbleed-update.html https://blogs.akamai.com/2014/04/heartbleed-update.html
  - josephagoss 13y ago
    That is very impressive. I'm amazed they essentially keep a revision history of their system configuration.
    - mrud 13y ago
      Yes, I'm also very impressed by this. It shows that they have their OPs in total control and know what they are doing.
      - brians 13y ago
        Thanks! As someone who grew up in the Akamai ops environment... How else could you get anything done? I've taken this tool for granted since it was built in 2000. I expect every planetary scale computing company to do this. Isn't it what Heroku and similar git-push-to-deploy systems are supposed to get you?
nly 13y ago
Why can't long-term private keys be protected, and used solely by, the kernel? Linux CryptoAPI seems to have an asymmetric key interface[0], RSA signatures are implemented[1], and there's even a x509/ASN1 parser[1]. Shouldn't this be the default on Linux rather than furthering the NIH syndrome with crude library-local allocators? In the very least, shouldn't there be a secure malloc in glibc? If not, why doesn't OpenSSL use an existing secure malloc library[2]? [0] https://www.kernel.org/doc/Documentation/crypto/asymmetric-keys.txt https://www.kernel.org/doc/Documentation/crypto/asymmetric-k... [1] https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/tree/crypto/asymmetric_keys?id=refs/tags/v3.14 https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.... [2] http://www.jabberwocky.com/software/secmalloc/ http://www.jabberwocky.com/software/secmalloc/
- deleted 13y ago
  [deleted]
- thefreeman 13y ago
  There have been a lot of posts about the custom malloc. From what i understand, parts of OpenSSL actually depend on the buggy implementation (specifically, the fact that you can the last thing free'd can be malloc'd again).
- awda 13y ago
  > If not, why doesn't OpenSSL use an existing secure malloc library? Any extra library dependency makes OpenSSL less portable. It doubles the work needed to get it running on your new embedded system. shrug It may also increase the code size and attack surface (although probably a good trade-off). I don't disagree that it sounds like a good idea :).
- eliteraspberrie 13y ago
  OpenSSL aims to be cross-platform, so their options are to use kernel-specific cryptography where possible, and a generic implementation where one isn't available; or always use the generic implementation. They probably reasoned at one point that the latter is less work.
DevX101 13y ago
If Akamai realized there was a major security vulnerability more than a year ago why didn't they share this information with the OpenSSL team?
- matthavener 13y ago
  I don't think they knew about heartbleed. This is stopgap to protect against leaking private keys via memory bugs.
  - ams6110 13y ago
    I think it's a fair question. If they had what they believed to be an improved (more secure) OpenSSL why not contribute the patch back to the community? After all they are standing on the shoulders of giants here, it seems a bit selfish to take an open-source project, improve it, and then not share that back. Yes I know that many open-source licenses do not obligate one to do this, but it still seems like the right thing to do to me.
    - mzs 13y ago
      I can't say what happened in this case but after you submit a patch to openssl and wait 6mo, a year, two, or even close to four, and simply don't hear anything back or if you do that they are doing something their own way instead, you just sort of lose the will and might get to simply be pragmatic and do what you need for your own job and customers after a while.
      - richardwhiuk 13y ago
        Probably this. Submitting patches back to open source is expensive - you have to dedicate engineers to tidying up and submitting patches, for no benefit (other than being closer to upstream, which is of marginal use, especially if there are clear forks) whereas they could be developing new functionality. In the real world, the priority if often to do new development instead.
  - ithkuil 13y ago
brianpgordon 13y ago
A CDN writing a custom malloc sounds kind of crazy, but I guess it paid off.
- judk 13y ago
  An open source SSL library writing a custom malloc sounds kind of crazy, but I guess, oh, it was crazy.
lobster_johnson 13y ago
Here's a good blog article which looks at their patch: http://blog.nullspace.io/akamai-ssl-patch.html http://blog.nullspace.io/akamai-ssl-patch.html.
herf 13y ago
Guard pages address lots of issues with really small overhead, nice. But if you know where to look they're not foolproof. Wondering what the cost of putting each primitive (e.g., RSA private signing + key) in a separate (child) process would be? Mailservers (qmail and postfix) seem to do designs like this. You could also imagine a page type that would only be readable to certain code segments (would take CPU support to do it.)
- richardwhiuk 13y ago
  Expensive, and it's not clear that it would help. Context switching between applications is costly in terms of cache hit and CPU time. The other problem is you then have to IPC the sensitive data to the other process, which will mean putting in temporary memory, which would expose it to stack crash, or heap attacks. You've also got to worry about synchronizing the different processes.
- whoopdedo 13y ago
  > You could also imagine a page type that would only be readable to certain code segments (would take CPU support to do it.) Isn't this what TPM is supposed to provide? It received an unfavorable reputation because of its association with DRM, but it addresses the same problem.
- teacup50 13y ago
  Intel SGX would allow implementing this efficiently in-process: https://software.intel.com/en-us/blogs/2013/09/26/protecting-application-secrets-with-intel-sgx https://software.intel.com/en-us/blogs/2013/09/26/protecting...

267 points by nikcub 13y ago

For background, this came up as Akamai mentioned in their Heartbleed announcement that they didn't think they had any sensitive information[0] disclosed by the bug as they had wrapped OpenSSL's malloc to create two heaps: a secure heap for sensitive information like keys and an ordinary heap: https://blogs.akamai.com/2014/04/heartbleed-update.html https://blogs.akamai.com/2014/04/heartbleed-update.html It prompted a lot of interest in their solution, which prompted them to release the patch. edit: [0] by "sensitive information" we mean keys, if I understand the implementation correctly, you could still overrun into web server allocation and get back HTTP headers.

gphilip 13y ago
From reading that blog post it is not clear to me if their wrapping code existed prior to the discovery of the heartbleed bug: The post summary says: Akamai patched the announced Heartbleed vulnerability prior to its public announcement. We, like all users of OpenSSL, could have exposed passwords or session cookies transiting our network from August 2012 through 4 April 2014. Our custom memory allocator protected against nearly every circumstance by which Heartbleed could have leaked SSL keys. So: did the custom memory allocator exist already in August 2012? From reading the post this looks to be the case. Could it be that someone at Akamai took a look at the heartbeat (or other OpenSSL) code, decided that it could lead to memory leaks, and wrote their own memory allocator wrapper code to guard against this? Edit: SSL -> OpenSSL
- computer 13y ago
  How I read it is that this code already protected their private key, but the Heartbleed bug still disclosed other private details (such as submitted user data).
- alexkus 13y ago
  From the original post: > This patch is a variant of what we've been using to help protect customer keys for a decade. So it protects keys against heartbleed, but not other HTTP related data (urls, cookies, headers, etc).
- Dylan16807 13y ago
  Keys are not passwords and session cookies.
- yukichan 13y ago
  Akamai's patch is older than the heartbleed issue in OpenSSL so it wasn't a response to heartbleed, and they were still vulnerable for non-key data.
brians 13y ago
We definitely leaked session cookies and passwords, until we patched. This only protected SSL private keys.
- ccurtsinger 13y ago
  You could do better with a probabilistically secure allocator instead: http://people.cs.umass.edu/~emery/pubs/ccs03-novark.pdf http://people.cs.umass.edu/~emery/pubs/ccs03-novark.pdf Randomized allocation makes it nearly impossible to forge pointers, locate sensitive data in the heap, and makes reuse unpredictable. This is strictly more powerful than ASLR, which does nothing prevent Heartbleed. Moving the base of the heap doesn't change the relative addresses of heap objects with a deterministic allocator. A randomized allocator does change these offsets, which makes it nearly impossible to exploit a heap buffer overrun (and quite a few other heap errors).
  - brians 13y ago
    That paper only seems to mention heap overflows for purposes of writing to a target object that will later be used for indirection or execution. I don't see how it makes Heartbleed any better to extract a shuffled heap instead of a sorted one. What am I missing?
    - ccurtsinger 13y ago
      It's not just a shuffled heap, it's also sparse. Section 4.1 covers heap overflow attacks, with an attacker using overflows from one object to overwrite entries in a nearby object's vtable. Because the objects could be anywhere in the sparse virtual address space, the probability of overwriting the desired object is very low (see section 6.2). The same reasoning applies to reads. If sensitive objects are distributed throughout the sparse heap, the probability of hitting a specific sensitive object is the same as the probability of overwriting the vtable in the above attack. The probability of reading out any sensitive object depends on the number of sensitive objects and the sparsity of the heap. There are also guard pages sprinkled throughout the sparse heap. Section 6.3.1 shows the minimum probability of a one byte overflow (read or write) hitting a guard page. This probability increases with larger objects and larger overflows. You can also increase sparsity to increase this probability, at a performance cost.

bello 13y ago

Disclaimer: I'm not too familiar with the code. "This arena is mmap'd, with guard pages before and after so pointer over- and under-runs won't wander into it." Doesn't that mean that this will only protect against overreads of a certain max length (such that the int16 length in heardbleed)? Seems like that wouldn't help with a length defined as a bigger int. I wonder if there's any better ways of doing this.

ramidarigaz 13y ago
The beginning and end of the allocated section of memory (the guard pages they refer to) are marked as PROT_NONE with mprotect, meaning that any access to them will cause a segfault. It's possible that a misbehaving process could jump straight into the unprotected memory, but it would have to not read from the guard pages at all. Buffer overruns don't have that problem (since they access memory sequentially), and would cause the program to crash before any sensitive data could be read (assuming the overrun starts outside the protected area).
- bello 13y ago
  That makes sense, thank you!
xyzzy123 13y ago
The idea is that a read overrun where the base pointer starts in a different arena would have to cross the guard pages to "get to" the secure arena.

okasaki 13y ago

If OpenSSL had been GPL licenced, this would have been public a long time ago.

zobzu 13y ago
Actually as long as they don't distribute the code, even under GPL they don't have to share it.
- dontmakemelaugh 13y ago
  You mean the application? They need to distribute the code if they provide applications based on the code.
- midas007 13y ago
  Yup. It's pretty common for open source to be used, abused, modified in house and not shared. Just look at Sergey Aleynikov and Goldman Sachs. [0] Further I've seen a shop deploy hundreds of thousands of CentOS nodes to avoid RHEL license fees. Doesn't even begin to scratch the surface of shops not funding FOSS they exploit like OpenSSH, etc. Most of them are stingy, demanding, greedy bastards. [0] http://blog.garrytan.com/goldman-sachs-sent-a-brilliant-computer-scientist-to-jail-over-8mb-of-open-source-code-uploaded-to-an-svn-repo http://blog.garrytan.com/goldman-sachs-sent-a-brilliant-comp...
  - aspensmonster 13y ago
    >Most of them are stingy, demanding, greedy bastards. I'd go so far as to say "nearly all of them."
    - midas007 13y ago
      There's a few counterexamples, which is a good thing: iXSystems does a lot of cool community work via FreeNAS, PC-BSD and FreeBSD. Github (Ruby), thoughtbot (Ruby), Google (Go, Python), Dropbox (Python), AT&T (C++, Ruby), Zurb (Ruby, Foundation), Twitter (Ruby, Scala), ... missing a bunch and some platforms too.

jgrahamc 13y ago

I would be interested to know if this actually protects against heartbleed in the real world. I can see how it would keep the private key protected, but not sure if it protects intermediate values used during calculation. Given that it's likely that the private key can be derived from those parts, it would be good to know if this custom allocator is 'enough' (i.e. are the intermediate values stored in memory allocated using the same allocator)? This week has been full of surprises. Has anyone tested this?

mrud 13y ago
I didn't but akamai itself did [1]: To test this belief, our engineers have rebuilt every Akamai configuration since August 2012, when we upgraded to openssl-1.0.1. This included every kernel, hardware, and edge server software combination; and then a careful inspection of the ways in which memory was allocated, to see if any non-long-term memory allocations might border on our secure heap. Most of the configurations were proven safe; but we found one configuration that was not - there was an available memory block in range of the secure key store. This less safe configuration was active on our network for nine days in March 2013 [1] https://blogs.akamai.com/2014/04/heartbleed-update.html https://blogs.akamai.com/2014/04/heartbleed-update.html
- josephagoss 13y ago
  That is very impressive. I'm amazed they essentially keep a revision history of their system configuration.
  - mrud 13y ago
    Yes, I'm also very impressed by this. It shows that they have their OPs in total control and know what they are doing.
    - brians 13y ago
      Thanks! As someone who grew up in the Akamai ops environment... How else could you get anything done? I've taken this tool for granted since it was built in 2000. I expect every planetary scale computing company to do this. Isn't it what Heroku and similar git-push-to-deploy systems are supposed to get you?

nly 13y ago

Why can't long-term private keys be protected, and used solely by, the kernel? Linux CryptoAPI seems to have an asymmetric key interface[0], RSA signatures are implemented[1], and there's even a x509/ASN1 parser[1]. Shouldn't this be the default on Linux rather than furthering the NIH syndrome with crude library-local allocators? In the very least, shouldn't there be a secure malloc in glibc? If not, why doesn't OpenSSL use an existing secure malloc library[2]? [0] https://www.kernel.org/doc/Documentation/crypto/asymmetric-keys.txt https://www.kernel.org/doc/Documentation/crypto/asymmetric-k... [1] https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/tree/crypto/asymmetric_keys?id=refs/tags/v3.14 https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.... [2] http://www.jabberwocky.com/software/secmalloc/ http://www.jabberwocky.com/software/secmalloc/

deleted 13y ago
[deleted]
thefreeman 13y ago
There have been a lot of posts about the custom malloc. From what i understand, parts of OpenSSL actually depend on the buggy implementation (specifically, the fact that you can the last thing free'd can be malloc'd again).
awda 13y ago
> If not, why doesn't OpenSSL use an existing secure malloc library? Any extra library dependency makes OpenSSL less portable. It doubles the work needed to get it running on your new embedded system. shrug It may also increase the code size and attack surface (although probably a good trade-off). I don't disagree that it sounds like a good idea :).
eliteraspberrie 13y ago
OpenSSL aims to be cross-platform, so their options are to use kernel-specific cryptography where possible, and a generic implementation where one isn't available; or always use the generic implementation. They probably reasoned at one point that the latter is less work.

DevX101 13y ago

If Akamai realized there was a major security vulnerability more than a year ago why didn't they share this information with the OpenSSL team?

matthavener 13y ago
I don't think they knew about heartbleed. This is stopgap to protect against leaking private keys via memory bugs.
- ams6110 13y ago
  I think it's a fair question. If they had what they believed to be an improved (more secure) OpenSSL why not contribute the patch back to the community? After all they are standing on the shoulders of giants here, it seems a bit selfish to take an open-source project, improve it, and then not share that back. Yes I know that many open-source licenses do not obligate one to do this, but it still seems like the right thing to do to me.
  - mzs 13y ago
    I can't say what happened in this case but after you submit a patch to openssl and wait 6mo, a year, two, or even close to four, and simply don't hear anything back or if you do that they are doing something their own way instead, you just sort of lose the will and might get to simply be pragmatic and do what you need for your own job and customers after a while.
    - richardwhiuk 13y ago
      Probably this. Submitting patches back to open source is expensive - you have to dedicate engineers to tidying up and submitting patches, for no benefit (other than being closer to upstream, which is of marginal use, especially if there are clear forks) whereas they could be developing new functionality. In the real world, the priority if often to do new development instead.
- ithkuil 13y ago

brianpgordon 13y ago

A CDN writing a custom malloc sounds kind of crazy, but I guess it paid off.

judk 13y ago
An open source SSL library writing a custom malloc sounds kind of crazy, but I guess, oh, it was crazy.

lobster_johnson 13y ago

Here's a good blog article which looks at their patch: http://blog.nullspace.io/akamai-ssl-patch.html http://blog.nullspace.io/akamai-ssl-patch.html.

herf 13y ago

Guard pages address lots of issues with really small overhead, nice. But if you know where to look they're not foolproof. Wondering what the cost of putting each primitive (e.g., RSA private signing + key) in a separate (child) process would be? Mailservers (qmail and postfix) seem to do designs like this. You could also imagine a page type that would only be readable to certain code segments (would take CPU support to do it.)

richardwhiuk 13y ago
Expensive, and it's not clear that it would help. Context switching between applications is costly in terms of cache hit and CPU time. The other problem is you then have to IPC the sensitive data to the other process, which will mean putting in temporary memory, which would expose it to stack crash, or heap attacks. You've also got to worry about synchronizing the different processes.
whoopdedo 13y ago
> You could also imagine a page type that would only be readable to certain code segments (would take CPU support to do it.) Isn't this what TPM is supposed to provide? It received an unfavorable reputation because of its association with DRM, but it addresses the same problem.
teacup50 13y ago
Intel SGX would allow implementing this efficiently in-process: https://software.intel.com/en-us/blogs/2013/09/26/protecting-application-secrets-with-intel-sgx https://software.intel.com/en-us/blogs/2013/09/26/protecting...