7 ms·

"CloudFlare later boasts on its blog about how they were able to protect their clients before many others." I wonder if some companies will boast to potential n

by jdubs 12y ago

"CloudFlare later boasts on its blog about how they were able to protect their clients before many others." I wonder if some companies will boast to potential new customers regarding their relationships with vendors that will offer them advanced patches on critical issues such as heartbleed. Kudos to their opportunistic marketing team, but I hope this trend does not continue.

tptacek 12y ago
While I don't care so much who got notified first (any given embargo timeline is going to frustrate large numbers of HN people; if that's a problem for you, start finding bugs or post a bounty), I find several things not to like in Cloudflare's marketing. Near as I can tell, Cloudflare is the origin of the notion that TLS private keys weren't going to be in the heap near packet data, a supposition they jumped to for no reason I can discern. They didn't find the bug. They benefited from a heads-up on the bug. Then they promoted a mistaken assumption about the bug, along with a gamified challenge site that was in fact a poor vehicle for investigating nginx+OpenSSL (how about, for instance, any decent debugger instead?). Meanwhile, ops teams at big companies were in heated debates with security about whether keys needed to roll. There are good people working at Cloudflare and I'm not part of any outrage battalion. I'm just not a fan of how they handled this particular incident.
- sinak 12y ago
  Neel Mehta tweeted that private keys wouldn't be in the heap near packet data on April 8th: https://twitter.com/neelmehta/status/453625474879471616 https://twitter.com/neelmehta/status/453625474879471616
  - tptacek 12y ago
    I think I'm fine giving the guy who found the bug a pass for his 140 character suggestion. I'm not looking to collect pelts. I just think there was a better way to address the question of private key exposure than "The Cloudflare Challenge". In the future, maybe we can address serious questions with engineering instead of marketing stunts; how about the "let's all work to instrument OpenSSL" challenge?
    - daeken 12y ago
      My issue with the Cloudflare Challenge can be summed up in: no matter what the results are, it will give people a diminished impression of the bug's actual impact. I can't fathom any way in which the Cloudflare Challenge was beneficial to the security of their customers (or anyone else, for that matter), which should be the goal of bounties; not to simply be a PR move.
      - tptacek 12y ago
        My problem is that if you're trying to figure out how key material is distributed throughout heap memory, asking people to answer that question about an unknown private key through heartbleed "peeks" is about the most obtuse possible way to find out.
    - throwaway5752 12y ago
      Yeah, the whole thing left a vaguely unpleasant taste. It worked in this case, but now that the marketing genie is out of the bottle it's going to make security vulnerability intelligence harder to evaluate. IMHO.
  - daeken 12y ago
    No, he didn't. He said that it's unlikely, and he's right. It's unlikely that I'm going to get heads 100 times in a row if I'm flipping a coin, but it suddenly becomes likely if I try it a couple million times. Being unlikely doesn't mean you shouldn't assume that it's possible for a motivated attacker. Especially when the barrier to entry is so extraordinarily low, as we've seen repeatedly (Cloudflare and Akamai, I'm looking at you).
    - makomk 12y ago
      Except that it isn't actually at all unlikely with some common configurations, which actually seem to give up the private key on every single attempt.
      - daeken 12y ago
        Fair point -- you're right that it's extremely configuration dependent. For many (especially in the Apache realm) it's trivial and very likely; others (especially Nginx) it's quite a bit more work. But even if he was 100% correct in it being unlikely, it still doesn't justify ignoring the threat, IMO. I think we're in violent agreement here.
- makomk 12y ago
  To be fair, Cloudflare didn't just jump to this conclusion for no reason, they apparently did test this by looking at the location of request buffers compared to the private key, trying to extract it themselves, and investigating the heap layout as a whole: http://blog.cloudflare.com/answering-the-critical-question-can-you-get-private-ssl-keys-using-heartbleed http://blog.cloudflare.com/answering-the-critical-question-c... They probably just got some or all of their assumptions wrong, as many have.
  - tptacek 12y ago
    I'm not sure how much of this I buy. I did what Willem did: I instrumented a small OpenSSL driver program and snapshotted memory. I did not go through the effort Jeremi Gosney and Willem and Thai and Ben Murphey went through to trace things through the code, but it was immediately apparent that there was more going on than the blog post Cloudflare wrote suggested. More importantly, the whole thesis of that blog post is that RSA private keys are loaded into memory once and never moved again. But that's obviously not true: intermediates based on private key components are created during Montgomery multiplication, for instance. The bigger problem is not that Cloudflare got things wrong. It's that they (a) marketed the wrong conclusion, and (b) put the conclusion to trial in a way that spent smart people's time unnecessarily.
    - eastdakota 12y ago
      Here was our initial conclusion from the CloudFlare post you referenced: ============ We think the stealing private keys on most NGINX servers is at least extremely hard and, likely, impossible. Even with Apache, which we think may be slightly more vulnerable, and we do not use at CloudFlare, we believe the likelihood of private SSL keys being revealed with the Heartbleed vulnerability is very low. That’s about the only good news of the last week. We want others to test our results so we created the Heartbleed Challenge. Aristotle struggled with the problem of disproving the existence of something that doesn’t exist. You can’t prove the negative, so through experimental results we will never be absolutely sure there’s not a condition we haven’t tested. However, the more eyes we get on the problem, the more confident we will be that, in spite of a number of other ways the Heartbleed vulnerability was extremely bad, we may have gotten lucky and been spared the worst of the potential consequences. That said, we’re proceeding assuming the worst. With respect to private keys held by CloudFlare, we patched the vulnerability before the public had knowledge of the vulnerability, making it unlikely that attackers were able to obtain private keys. Still, to be safe, as outlined at the beginning of this post, we are executing on a plan to reissue and revoke potentially affected certificates, including the cloudflare.com certificate. Vulnerabilities like this one are challenging because people have imperfect information about the risks they pose. It is important that the community works together to identify the real risks and work towards a safer Internet. We’ll monitor the results on the Heartbleed Challenge and immediately publicize results that challenge any of the above. ============ Which is exactly what we did. To be clear, we were wrong. Our mistaken assumption was focusing on the private key itself and not focusing enough on the exponents that are used to generate the key -- which is what the researchers who solved the challenge were able to obtain. As the wishy-washy conclusion above hopefully makes clear, even when we said it was hard to get the private keys, we were very uncertain and uncomfortable with that conclusion. That's why we posted the challenge. What the challenge did was answer the question definitively: you can get private SSL keys. Knowing that has been valuable for us in deciding to accelerate reissue/revocation process for all the certs we manage on behalf of our customers. Remember: at our scale, revoking hundreds of thousands of certs risked breaking our CA partners' infrastructure, so it wasn't without a cost. Knowing the risk is higher than we originally thought accelerated our efforts which will be complete in the next 48 hours. And, beyond CloudFlare, our hope is knowing the risk is real and proven has benefited other organizations as well.

by jdubs 12y ago

tptacek 12y ago

While I don't care so much who got notified first (any given embargo timeline is going to frustrate large numbers of HN people; if that's a problem for you, start finding bugs or post a bounty), I find several things not to like in Cloudflare's marketing. Near as I can tell, Cloudflare is the origin of the notion that TLS private keys weren't going to be in the heap near packet data, a supposition they jumped to for no reason I can discern. They didn't find the bug. They benefited from a heads-up on the bug. Then they promoted a mistaken assumption about the bug, along with a gamified challenge site that was in fact a poor vehicle for investigating nginx+OpenSSL (how about, for instance, any decent debugger instead?). Meanwhile, ops teams at big companies were in heated debates with security about whether keys needed to roll. There are good people working at Cloudflare and I'm not part of any outrage battalion. I'm just not a fan of how they handled this particular incident.

sinak 12y ago
Neel Mehta tweeted that private keys wouldn't be in the heap near packet data on April 8th: https://twitter.com/neelmehta/status/453625474879471616 https://twitter.com/neelmehta/status/453625474879471616
- tptacek 12y ago
  I think I'm fine giving the guy who found the bug a pass for his 140 character suggestion. I'm not looking to collect pelts. I just think there was a better way to address the question of private key exposure than "The Cloudflare Challenge". In the future, maybe we can address serious questions with engineering instead of marketing stunts; how about the "let's all work to instrument OpenSSL" challenge?
  - daeken 12y ago
    My issue with the Cloudflare Challenge can be summed up in: no matter what the results are, it will give people a diminished impression of the bug's actual impact. I can't fathom any way in which the Cloudflare Challenge was beneficial to the security of their customers (or anyone else, for that matter), which should be the goal of bounties; not to simply be a PR move.
    - tptacek 12y ago
      My problem is that if you're trying to figure out how key material is distributed throughout heap memory, asking people to answer that question about an unknown private key through heartbleed "peeks" is about the most obtuse possible way to find out.
  - throwaway5752 12y ago
    Yeah, the whole thing left a vaguely unpleasant taste. It worked in this case, but now that the marketing genie is out of the bottle it's going to make security vulnerability intelligence harder to evaluate. IMHO.
- daeken 12y ago
  No, he didn't. He said that it's unlikely, and he's right. It's unlikely that I'm going to get heads 100 times in a row if I'm flipping a coin, but it suddenly becomes likely if I try it a couple million times. Being unlikely doesn't mean you shouldn't assume that it's possible for a motivated attacker. Especially when the barrier to entry is so extraordinarily low, as we've seen repeatedly (Cloudflare and Akamai, I'm looking at you).
  - makomk 12y ago
    Except that it isn't actually at all unlikely with some common configurations, which actually seem to give up the private key on every single attempt.
    - daeken 12y ago
      Fair point -- you're right that it's extremely configuration dependent. For many (especially in the Apache realm) it's trivial and very likely; others (especially Nginx) it's quite a bit more work. But even if he was 100% correct in it being unlikely, it still doesn't justify ignoring the threat, IMO. I think we're in violent agreement here.
makomk 12y ago
To be fair, Cloudflare didn't just jump to this conclusion for no reason, they apparently did test this by looking at the location of request buffers compared to the private key, trying to extract it themselves, and investigating the heap layout as a whole: http://blog.cloudflare.com/answering-the-critical-question-can-you-get-private-ssl-keys-using-heartbleed http://blog.cloudflare.com/answering-the-critical-question-c... They probably just got some or all of their assumptions wrong, as many have.
- tptacek 12y ago
  I'm not sure how much of this I buy. I did what Willem did: I instrumented a small OpenSSL driver program and snapshotted memory. I did not go through the effort Jeremi Gosney and Willem and Thai and Ben Murphey went through to trace things through the code, but it was immediately apparent that there was more going on than the blog post Cloudflare wrote suggested. More importantly, the whole thesis of that blog post is that RSA private keys are loaded into memory once and never moved again. But that's obviously not true: intermediates based on private key components are created during Montgomery multiplication, for instance. The bigger problem is not that Cloudflare got things wrong. It's that they (a) marketed the wrong conclusion, and (b) put the conclusion to trial in a way that spent smart people's time unnecessarily.
  - eastdakota 12y ago
    Here was our initial conclusion from the CloudFlare post you referenced: ============ We think the stealing private keys on most NGINX servers is at least extremely hard and, likely, impossible. Even with Apache, which we think may be slightly more vulnerable, and we do not use at CloudFlare, we believe the likelihood of private SSL keys being revealed with the Heartbleed vulnerability is very low. That’s about the only good news of the last week. We want others to test our results so we created the Heartbleed Challenge. Aristotle struggled with the problem of disproving the existence of something that doesn’t exist. You can’t prove the negative, so through experimental results we will never be absolutely sure there’s not a condition we haven’t tested. However, the more eyes we get on the problem, the more confident we will be that, in spite of a number of other ways the Heartbleed vulnerability was extremely bad, we may have gotten lucky and been spared the worst of the potential consequences. That said, we’re proceeding assuming the worst. With respect to private keys held by CloudFlare, we patched the vulnerability before the public had knowledge of the vulnerability, making it unlikely that attackers were able to obtain private keys. Still, to be safe, as outlined at the beginning of this post, we are executing on a plan to reissue and revoke potentially affected certificates, including the cloudflare.com certificate. Vulnerabilities like this one are challenging because people have imperfect information about the risks they pose. It is important that the community works together to identify the real risks and work towards a safer Internet. We’ll monitor the results on the Heartbleed Challenge and immediately publicize results that challenge any of the above. ============ Which is exactly what we did. To be clear, we were wrong. Our mistaken assumption was focusing on the private key itself and not focusing enough on the exponents that are used to generate the key -- which is what the researchers who solved the challenge were able to obtain. As the wishy-washy conclusion above hopefully makes clear, even when we said it was hard to get the private keys, we were very uncertain and uncomfortable with that conclusion. That's why we posted the challenge. What the challenge did was answer the question definitively: you can get private SSL keys. Knowing that has been valuable for us in deciding to accelerate reissue/revocation process for all the certs we manage on behalf of our customers. Remember: at our scale, revoking hundreds of thousands of certs risked breaking our CA partners' infrastructure, so it wasn't without a cost. Knowing the risk is higher than we originally thought accelerated our efforts which will be complete in the next 48 hours. And, beyond CloudFlare, our hope is knowing the risk is real and proven has benefited other organizations as well.