5 ms·

As much shit as CloudFlare took for the way they humble-bragged about finding out about Heartbleed early, I have to give kudos to the team for doing the right t

As much shit as CloudFlare took for the way they humble-bragged about finding out about Heartbleed early, I have to give kudos to the team for doing the right thing and reissuing and revoking all the certs, as well as detailing the exploit and offering up the test site. I'll admit, when the CloudFlare Challenge launched, the cynical part of me saw it as merely a way for the company to convince itself that it didn't need to reissue and revoke all of its certs (and I maintain that is exactly what it was), but the way the company handled things after the challenge was won is what really impressed me. They didn't have to be that transparent. Moreover, the test server has provided a way for Filippo Valsorda to improve his excellent Heartbleed checker, so well played.

gatehouse 13y ago
Yeah, it sounds like the reason that they came off as glib and triumphant is because they didn't anticipate how serious the vulnerability would end up being.
- eastdakota 13y ago
  Yes that and the fact that we didn't appreciate how few people had been notified. It was a big surprise to learn we were one of only a small handful of companies.
  - j_s 13y ago
    This explains a lot; hopefully you can add "confirm extent of pre-disclosure distribution" to the list of things to do when this type of info arrives in the future!
tptacek 13y ago
I'm a little irritated by the challenge not because of any delay in rotating certs, but because its whole premise was invalid; the challenge was accompanied by a long blog post that presumed a model for how RSA keys hit heap memory that appears to have been self-evidently false (again: I might be wrong here). Or, to put it differently: Akamai, which attempted to write a secure heap to contain memory corruption flaws in OpenSSL, conceded immediately Willem Pinckaers observation that they weren't protecting the CRT intermediates. Akamai seemed to believe ahead of time that keys weren't simply loaded once and held. There are lots of smart people working at Cloudflare, but I think the "challenge site" approach was a waste of time. The stuff John Graham-Cumming posted today, showing bitmap diagrams of the memory space of OpenSSL processes, would have been much more productive as a starting point. Diagrams like that are a standard part of reverse-engineering practice. They should have started there, and not with the challenge site. We all make mistakes (some of mine are easy to find). I'd just say that a lesson to learn from this is, "err on the side of greater severity, not lesser." (Is what I think.)
- jgrahamc 13y ago
  There's one significant difference I think between what Akamai did and what we (CloudFlare) did. Akamai thought their memory allocator meant they were safe and so didn't change their SSL keys; we weren't convinced by our own analysis (which I was involved in) launched the challenge, and simultaneously started the process of revoking keys. In the challenge blog post we said: "We’ve begun the process of reissuing and revoking the keys CloudFlare manages on behalf of our customers. In order to ensure that we don’t overburden the certificate authority resources, we are staging this process. We expect that it will be complete by early next week." That wasn't BS, we were doing that and we had a _ton_ of keys to deal with. Today's blog post is a result of further analysis and the announcement that we revoked everything. BTW I'm not done with analysis of OpenSSL memory. There's still something weird going on (I think in the bignum library) where data isn't getting OPENSSL_cleanse'd from memory.
  - filmgirlcw 13y ago
    That's a really good point. I had read the original post as "we're reissuing and revoking some keys" but didn't understand you would be doing all keys, regardless of the challenge results, therefore I thought the process was expanded to everyone AFTER the challenge was won. Glad to hear you were already in the process of doing the staggered reissue/revoke cycle. And also props for reissuing and revoking everything. That's hugely important.
  - tptacek 13y ago
    I am "a little bit" not a fan of waiting to rekey. A lot of people did that. I am "a lot" not a fan of setting up a challenge game to have people try to pull the key with one hand figuratively tied behind their back, when a debugger or memory dump would have decisively answered the question --- and, presumably, did exactly that for at least some of the winners. I am "severely" not a fan of marketing skepticism about the severity of a flaw. This is almost vulnerability response 101: it's the reason Cisco and Microsoft tag memory corruption flaws "possible remote code execution" even when they're obviously not exploitable. Err on the side of severity. As evidence for my concern, I'll again observe that Bruce Schneier's read of the Crowdflare Challenge blog post was that your team believed it to be "next to impossible" to extract RSA keys. As you know, very few people in your audience have the technical background to evaluate that claim, especially when it's accompanied by heap metadata diagrams. I don't think any of this is earth-shattering or even all that controversial. I would just hate to see a standard response to complicated vulnerabilities become "set up a challenge site". Challenges are great! You guys should do more of them! Just don't do them to answer critical security questions. :) Edited to sound less petulant.

gatehouse 13y ago

Yeah, it sounds like the reason that they came off as glib and triumphant is because they didn't anticipate how serious the vulnerability would end up being.

eastdakota 13y ago
Yes that and the fact that we didn't appreciate how few people had been notified. It was a big surprise to learn we were one of only a small handful of companies.
- j_s 13y ago
  This explains a lot; hopefully you can add "confirm extent of pre-disclosure distribution" to the list of things to do when this type of info arrives in the future!

tptacek 13y ago

I'm a little irritated by the challenge not because of any delay in rotating certs, but because its whole premise was invalid; the challenge was accompanied by a long blog post that presumed a model for how RSA keys hit heap memory that appears to have been self-evidently false (again: I might be wrong here). Or, to put it differently: Akamai, which attempted to write a secure heap to contain memory corruption flaws in OpenSSL, conceded immediately Willem Pinckaers observation that they weren't protecting the CRT intermediates. Akamai seemed to believe ahead of time that keys weren't simply loaded once and held. There are lots of smart people working at Cloudflare, but I think the "challenge site" approach was a waste of time. The stuff John Graham-Cumming posted today, showing bitmap diagrams of the memory space of OpenSSL processes, would have been much more productive as a starting point. Diagrams like that are a standard part of reverse-engineering practice. They should have started there, and not with the challenge site. We all make mistakes (some of mine are easy to find). I'd just say that a lesson to learn from this is, "err on the side of greater severity, not lesser." (Is what I think.)

jgrahamc 13y ago
There's one significant difference I think between what Akamai did and what we (CloudFlare) did. Akamai thought their memory allocator meant they were safe and so didn't change their SSL keys; we weren't convinced by our own analysis (which I was involved in) launched the challenge, and simultaneously started the process of revoking keys. In the challenge blog post we said: "We’ve begun the process of reissuing and revoking the keys CloudFlare manages on behalf of our customers. In order to ensure that we don’t overburden the certificate authority resources, we are staging this process. We expect that it will be complete by early next week." That wasn't BS, we were doing that and we had a _ton_ of keys to deal with. Today's blog post is a result of further analysis and the announcement that we revoked everything. BTW I'm not done with analysis of OpenSSL memory. There's still something weird going on (I think in the bignum library) where data isn't getting OPENSSL_cleanse'd from memory.
- filmgirlcw 13y ago
  That's a really good point. I had read the original post as "we're reissuing and revoking some keys" but didn't understand you would be doing all keys, regardless of the challenge results, therefore I thought the process was expanded to everyone AFTER the challenge was won. Glad to hear you were already in the process of doing the staggered reissue/revoke cycle. And also props for reissuing and revoking everything. That's hugely important.
- tptacek 13y ago
  I am "a little bit" not a fan of waiting to rekey. A lot of people did that. I am "a lot" not a fan of setting up a challenge game to have people try to pull the key with one hand figuratively tied behind their back, when a debugger or memory dump would have decisively answered the question --- and, presumably, did exactly that for at least some of the winners. I am "severely" not a fan of marketing skepticism about the severity of a flaw. This is almost vulnerability response 101: it's the reason Cisco and Microsoft tag memory corruption flaws "possible remote code execution" even when they're obviously not exploitable. Err on the side of severity. As evidence for my concern, I'll again observe that Bruce Schneier's read of the Crowdflare Challenge blog post was that your team believed it to be "next to impossible" to extract RSA keys. As you know, very few people in your audience have the technical background to evaluate that claim, especially when it's accompanied by heap metadata diagrams. I don't think any of this is earth-shattering or even all that controversial. I would just hate to see a standard response to complicated vulnerabilities become "set up a challenge site". Challenges are great! You guys should do more of them! Just don't do them to answer critical security questions. :) Edited to sound less petulant.