4 ms·

This whole thing was totally mismanaged. Cloudflare had 1 week's notice from these codenomicon guys, and they patched it and stayed silent (NDA?). Then the co

by chomp 13y ago

This whole thing was totally mismanaged. Cloudflare had 1 week's notice from these codenomicon guys, and they patched it and stayed silent (NDA?). Then the codenomicon came up with a fancy name, bought a domain that I'm guessing they paid a fair sum for (for publicity?) and publicized it without notifying anyone (Amazon? Yahoo?, any Linux distro?) I mean, openssl.org's own website was vulnerable until not too long ago. I don't appreciate having to build custom RPMs in the middle of the night because my distribution had no prior warning. I really don't appreciate this spectacle to advertise a security services firm.

tptacek 13y ago
Everybody wants prior warning, but very early in the process you hit a point where secret notices create more harm than help. The worst situation you can be in is where everyone knows something is going on, but only a small group of people know what it is. At that point, the incentives strongly favor the attackers, who will work quickly to join the in-crowd and exploit the window of time where they know how to exploit the flaw, and site operators don't know enough about it to take it seriously. It's especially hard to stage-manage these updates with open source software. Look at the patch; to someone who takes exploit code seriously, the vulnerability is instantly obvious. How was anyone supposed to give you a heads up without tipping off attackers?
- parennoob 13y ago
  > At that point, the incentives strongly favor the attackers, who will work quickly to join the in-crowd and exploit the window of time where they know how to exploit the flaw, and site operators don't know enough about it to take it seriously. Has this sort of behaviour been observed for other vulnerabilities in the wild that can be cited? I would think that the patch is small enough that a co-ordinated deployment to all the distros could have been managed so that they had packages ready before attackers could exploit the vulnerability. Not sure that the correct solution is to announce it with a corporate blog post and a vanity web site instead.
  - frandroid 13y ago
    There's another name for the phenomenon described by tptacek above: insider trading. Plenty of examples in the wild. The more insiders know what's happening before the markets do, the more likely someone will act on it, or sell the information.
    - deleted 13y ago
      [deleted]
      - rdl 13y ago
        Someone still being on 0.9.8 and this not being vulnerable isn't too meaningful as an indicator. If they mysteriously went from 1.0.1f back to 0.9.8 without changing any other part of their configuration, that would be suspicious. The lulzy thing about this bug is that people who are still on 0.9.8 aren't doing the right thing; they just got lucky on this one bug.
  - drewcrawford 13y ago
    > Has this sort of behaviour been observed for other vulnerabilities in the wild that can be cited? Yes. The mailing list that was used as recently as 2011 to coordinate exactly this kind of issue was backdoored. Further reading: http://www.openwall.com/lists/oss-security/2011/03/03/3 http://www.openwall.com/lists/oss-security/2011/03/03/3
- chomp 13y ago
  I think we can agree that there is a happy medium between the slow and risky process of notifying every single organization, and only one SSL terminator "in the know". The logistics of disclosure for something as ground breaking as this are probably very difficult. I suppose the details are still sketchy at this point, but I have no idea how CloudFlare received a week's notice, and no one more important[1] knew of it until it was publicized. CloudFlare being the special group makes me suspicious of how it all went down. [1] I'm not picking on CloudFlare, they have a huge footprint on the internet.
  - 0x0 13y ago
    Apparently Akamai was also given early notice, from the OpenSSL team. https://blogs.akamai.com/2014/04/heartbleed-faq-akamai-systems-patched.html https://blogs.akamai.com/2014/04/heartbleed-faq-akamai-syste... I wonder why the various linux distros like debian could not have been notified simultaneously.
    - jessaustin 13y ago
      Has distro notification caused problems in the past? E.g. patches posted early, emails forwarded, public discussion, etc.?
    - danielweber 13y ago
      No advance channel set up? This stuff is always tricky, and I admittedly have a strong adverse reaction to the "just blow everything open, it will be lulzy when the web breaks" attitude. To be on such a list you also need to have a positive prior record demonstrating your ability to keep your mouth shut, which is hard to prove.
- IgorPartola 13y ago
  What about the Postgres bug from I think last year where instead of putting up a fancy looking site the people involved coordinated in secret with every major distro to release a patched package? That seemed to me to be much more professional. Or is this bug significantly different?
rdl 13y ago
The only people you can really give advance notice to are those who will keep it secret until the embargo is lifted. That is basically attackers (TAO), service providers (like Cloudflare and Akamai), and very small software development teams who don't release code to public repositories. I'm sure glad they gave CF advance notice vs TAO (presumably). I'd prefer if it just dropped with zero notice for anyone, although if a security company wants to provide priority notice about vulnerabilities it discovers to people who pay the company in advance, good for them.
- dfc 13y ago
  TAO = NSA Tailored Access Operations https://en.wikipedia.org/wiki/Tailored_Access_Operations https://en.wikipedia.org/wiki/Tailored_Access_Operations
- mentat 13y ago
  I fail to grasp on Cloudflare and Akamai are "service providers" and Amazon Web Services is not.
  - indolering 13y ago
    This is just speculation, but Amazon isn't known for playing very nice in the open-source community. Maybe they lacked the personal relationships required for someone in the know to trust them. OpenSSL, CloudFlare, Google … those are all open-source shops.

by chomp 13y ago

tptacek 13y ago

Everybody wants prior warning, but very early in the process you hit a point where secret notices create more harm than help. The worst situation you can be in is where everyone knows something is going on, but only a small group of people know what it is. At that point, the incentives strongly favor the attackers, who will work quickly to join the in-crowd and exploit the window of time where they know how to exploit the flaw, and site operators don't know enough about it to take it seriously. It's especially hard to stage-manage these updates with open source software. Look at the patch; to someone who takes exploit code seriously, the vulnerability is instantly obvious. How was anyone supposed to give you a heads up without tipping off attackers?

parennoob 13y ago
> At that point, the incentives strongly favor the attackers, who will work quickly to join the in-crowd and exploit the window of time where they know how to exploit the flaw, and site operators don't know enough about it to take it seriously. Has this sort of behaviour been observed for other vulnerabilities in the wild that can be cited? I would think that the patch is small enough that a co-ordinated deployment to all the distros could have been managed so that they had packages ready before attackers could exploit the vulnerability. Not sure that the correct solution is to announce it with a corporate blog post and a vanity web site instead.
- frandroid 13y ago
  There's another name for the phenomenon described by tptacek above: insider trading. Plenty of examples in the wild. The more insiders know what's happening before the markets do, the more likely someone will act on it, or sell the information.
  - deleted 13y ago
    [deleted]
    - rdl 13y ago
      Someone still being on 0.9.8 and this not being vulnerable isn't too meaningful as an indicator. If they mysteriously went from 1.0.1f back to 0.9.8 without changing any other part of their configuration, that would be suspicious. The lulzy thing about this bug is that people who are still on 0.9.8 aren't doing the right thing; they just got lucky on this one bug.
- drewcrawford 13y ago
  > Has this sort of behaviour been observed for other vulnerabilities in the wild that can be cited? Yes. The mailing list that was used as recently as 2011 to coordinate exactly this kind of issue was backdoored. Further reading: http://www.openwall.com/lists/oss-security/2011/03/03/3 http://www.openwall.com/lists/oss-security/2011/03/03/3
chomp 13y ago
I think we can agree that there is a happy medium between the slow and risky process of notifying every single organization, and only one SSL terminator "in the know". The logistics of disclosure for something as ground breaking as this are probably very difficult. I suppose the details are still sketchy at this point, but I have no idea how CloudFlare received a week's notice, and no one more important[1] knew of it until it was publicized. CloudFlare being the special group makes me suspicious of how it all went down. [1] I'm not picking on CloudFlare, they have a huge footprint on the internet.
- 0x0 13y ago
  Apparently Akamai was also given early notice, from the OpenSSL team. https://blogs.akamai.com/2014/04/heartbleed-faq-akamai-systems-patched.html https://blogs.akamai.com/2014/04/heartbleed-faq-akamai-syste... I wonder why the various linux distros like debian could not have been notified simultaneously.
  - jessaustin 13y ago
    Has distro notification caused problems in the past? E.g. patches posted early, emails forwarded, public discussion, etc.?
  - danielweber 13y ago
    No advance channel set up? This stuff is always tricky, and I admittedly have a strong adverse reaction to the "just blow everything open, it will be lulzy when the web breaks" attitude. To be on such a list you also need to have a positive prior record demonstrating your ability to keep your mouth shut, which is hard to prove.
IgorPartola 13y ago
What about the Postgres bug from I think last year where instead of putting up a fancy looking site the people involved coordinated in secret with every major distro to release a patched package? That seemed to me to be much more professional. Or is this bug significantly different?

rdl 13y ago

The only people you can really give advance notice to are those who will keep it secret until the embargo is lifted. That is basically attackers (TAO), service providers (like Cloudflare and Akamai), and very small software development teams who don't release code to public repositories. I'm sure glad they gave CF advance notice vs TAO (presumably). I'd prefer if it just dropped with zero notice for anyone, although if a security company wants to provide priority notice about vulnerabilities it discovers to people who pay the company in advance, good for them.

dfc 13y ago
TAO = NSA Tailored Access Operations https://en.wikipedia.org/wiki/Tailored_Access_Operations https://en.wikipedia.org/wiki/Tailored_Access_Operations
mentat 13y ago
I fail to grasp on Cloudflare and Akamai are "service providers" and Amazon Web Services is not.
- indolering 13y ago
  This is just speculation, but Amazon isn't known for playing very nice in the open-source community. Maybe they lacked the personal relationships required for someone in the know to trust them. OpenSSL, CloudFlare, Google … those are all open-source shops.