5 ms·

Heartbleed is about to get worse, and it will slow the Internet to a crawl

26 points by aburan28 13y ago

falcolas 13y ago
Nothing really new here - just the acknowledgement that the ability to steal private ssl keys requires a massive effort to replace those keys. "If anything is compromised, assume everything is compromised." Our keys have been updated (and the old certs revoked), our servers patched, our sessions expired, and passwords expired. Mitigation more or less complete. Helps we're running a very small site, for sure, but the steps were clear from the beginning.
- gvb 13y ago
  But have you revoked your old primary keys? Everybody that was vulnerable to Heartbleed should revoke their old primary keys so that web browsers no longer trust those potentially stolen keys. If they are not revoked and a Bad Guy stole your primary key(s), they can create a fake web site that identifies itself as your web site and all web browsers would trust it as legitimate, complete with the little green "you can trust me" padlock. The CloudFlare challenge proved that the primary keys could be stolen. There were millions of vulnerable sites (the internet is approaching a billion total web sites, and a significant number of them were vulnerable). The revocation list method used by browsers does not scale well with those kinds of numbers. [1] http://news.netcraft.com/archives/category/web-server-survey/ http://news.netcraft.com/archives/category/web-server-survey...
  - schrodinger 13y ago
    Doesn't rekeying result in the old certificate based on the compromised private key being revoked?
    - mpyne 13y ago
      No, presenting a certificate that has been rekeyed does not mean that the old certificate stopped being cryptographically valid. In fact the old cert will always be cryptographically valid, which is why a separate revocation list has to be created as a positive acknowledgement that the old cert, though valid, should not be trusted.
      - lmz 13y ago
        Do the browsers actually use revocation checking?
        sneak 13y ago
        Some do, but most of them fail unsafe when the check endpoint is unavailable IIRC.
        falcolas 13y ago
        Firefox does. Found that out in my development box which I hadn't updated with the new certs, the old certs showed up as revoked and Firefox wouldn't let me proceed.
      - schrodinger 13y ago
        I understand that, but I thought in practice most CAs will automatically revoke the former certificate when rekeying. It sounds like godaddy does, at least.
        mpyne 13y ago
        That's up to the CA. It seems that StartSSL doesn't automatically revoke an old one even with rekeying.
  - falcolas 13y ago
    Yup, as part of the cert re-issue, we revoked the previous cert. Could have been more explicit there, sorry.
001sky 13y ago
According to Paul Mutton, a security consultant at the Web services company Netcraft. Checking a site’s identity will take vastly longer. “If a certificate authority has to revoke 10,000 certificates, that entry will have 10,000 certificates on it,” Mutton said. “And if browsers have to download that . . . we’re talking hundreds of megabytes.” It’s roughly the equivalent of having to download 30 minutes’ worth of standard-definition video just to view a single Web page. > is this accurate?
- nitrogen 13y ago
  Would it be possible to use a probabilistic data structure (e.g. Bloom filter) to create a privacy-enabled CRL as a service?
  - daeken 13y ago
    I don't see why not, especially because you already have the old key signature/fingerprint to use when pushing it into the Bloom filter. Seems like a perfect case for it, actually.
  - gvb 13y ago
    You could distrust any certificate that was signed before April 7 (or some slightly later date). It might be easier to do that and whitelist the good "pre-apocalypse" certificates. The nice thing about doing this is that your whitelist will inherently go to zero as certificates time out and get renewed (typically 1-3 years?). The hard thing would be to generate an accurate whitelist.
    - mygrant 13y ago
      We had certs that were only used and present on machines that were not vulnerable, and therefore did not replace or revoke them. Please don't do this.
      - nathanvanfleet 13y ago
        Furthermore servers that weren't even using OpenSSL or the particular version(s) that were susceptible. What is the actual percentage of servers that needed to react to this?
matt__rose 13y ago
This sentence: "No examples have surfaced of anyone actually exploiting the vulnerability" is incorrect. The Canada Revenue Agency revealed that 900 SINs (canadian equivalent of SSNs, but nowhere near as totemically identifying) were stolen due to the heartbleed bug. http://www.theglobeandmail.com/technology/mounties-chasing-viable-lead-in-cras-heartbleed-breach/article18002731/ http://www.theglobeandmail.com/technology/mounties-chasing-v...
- justinhj 13y ago
  In order to know this they must have stored and subsequently analysed the ssl traffic. Something that I would think is quite unusual. So the true scope of data theft is probably much larger.
  - simcop2387 13y ago
    They might have turned on that logging when the bug was announced so that they'd have an idea of the threat they were under while updating.
- thisisdave 13y ago
  I thought that sentence was referring specifically to vulnerable certificates.
sitkack 13y ago
1) Lets just call a do-over, EVERYONE generate new private keys and request new certs. Any cert created before April 10 2014 will be considered suspect. 2) Create a bloom filter for revoked certs. Have a remote cert revocation server/cache so I don't need to have a 50MB file of revocations.
- the_ancient 13y ago
  while the problem is wide spread, it is no where near enough to say "all certs after 4/10 are suspect" There are an ever larger number of certs not impacted by this event.
- jerf 13y ago
  False positives are a problem for a Bloom filter. You could conceivably create one for a given registrar, though, which could in theory create a Bloom filter, iterate through all the certs it has issued, and then list out the certs that the Bloom filter issues false positives for, but are still valid. However, it's too late to put that into play. Nobody ever expected to have to revoke this quantity of certs. (In hindsight, that was obviously wrong. Something like this was going to happen someday. But this is exactly the sort of thing that's very easy to argue away, because nobody wants to be convinced that this amount of work will be necessary.)
justinhj 13y ago
Question: with a stolen private key don't you still have the problem that is has the host name in it? So to truly spoof a browser you need to compromise the domain and serve a site from within it?
- mpyne 13y ago
  It depends on what you're going after, but if you're just targeting a local subnet, that's what a "man-in-the-middle" attack is for. What is probably easier is somehow redirecting the DNS entry for the target site to point to your malware-serving site, which has happened before as well.
kijin 13y ago
The situation would be much more manageable if browser vendors provided an online service that quickly allows individual certificates to be checked against the latest revocation list from all trusted CAs. There should be no need for each user to download the full revocation list, since most people will never visit 99.9% of the websites in that list. Until now, revocation lists were small enough that this was not a problem. But it's probably time to fix that. There are already third-party add-ons like Perspectives that check the certificates you see against those seen by other people. Chrome already knows which Google certs are valid or not. There's no reason why Mozilla, for example, shouldn't make something like this an official security feature of Firefox, too.
- astrange 13y ago
  This is OCSP. Chrome removed their support for it, but I think Firefox might check it.
mballantyne 13y ago
Do CRLs even work? My understanding is that the only browser that hard-fails to load a page if OCSP is blocked is Chrome for certs in CRLSets. Everyone else is vulnerable to MITM if access to the CRL / OCSP servers is blocked. I would love to be wrong. Does anyone know if anything has changed for the better since 2011? http://blog.spiderlabs.com/2011/04/certificate-revocation-behavior-in-modern-browsers.html http://blog.spiderlabs.com/2011/04/certificate-revocation-be... http://dev.chromium.org/Home/chromium-security/crlsets http://dev.chromium.org/Home/chromium-security/crlsets https://www.imperialviolet.org/2011/03/18/revocation.html https://www.imperialviolet.org/2011/03/18/revocation.html

26 points by aburan28 13y ago

Nothing really new here - just the acknowledgement that the ability to steal private ssl keys requires a massive effort to replace those keys. "If anything is compromised, assume everything is compromised." Our keys have been updated (and the old certs revoked), our servers patched, our sessions expired, and passwords expired. Mitigation more or less complete. Helps we're running a very small site, for sure, but the steps were clear from the beginning.

gvb 13y ago
But have you revoked your old primary keys? Everybody that was vulnerable to Heartbleed should revoke their old primary keys so that web browsers no longer trust those potentially stolen keys. If they are not revoked and a Bad Guy stole your primary key(s), they can create a fake web site that identifies itself as your web site and all web browsers would trust it as legitimate, complete with the little green "you can trust me" padlock. The CloudFlare challenge proved that the primary keys could be stolen. There were millions of vulnerable sites (the internet is approaching a billion total web sites, and a significant number of them were vulnerable). The revocation list method used by browsers does not scale well with those kinds of numbers. [1] http://news.netcraft.com/archives/category/web-server-survey/ http://news.netcraft.com/archives/category/web-server-survey...
- schrodinger 13y ago
  Doesn't rekeying result in the old certificate based on the compromised private key being revoked?
  - mpyne 13y ago
    No, presenting a certificate that has been rekeyed does not mean that the old certificate stopped being cryptographically valid. In fact the old cert will always be cryptographically valid, which is why a separate revocation list has to be created as a positive acknowledgement that the old cert, though valid, should not be trusted.
    - lmz 13y ago
      Do the browsers actually use revocation checking?
      - sneak 13y ago
        Some do, but most of them fail unsafe when the check endpoint is unavailable IIRC.
      - falcolas 13y ago
        Firefox does. Found that out in my development box which I hadn't updated with the new certs, the old certs showed up as revoked and Firefox wouldn't let me proceed.
    - schrodinger 13y ago
      I understand that, but I thought in practice most CAs will automatically revoke the former certificate when rekeying. It sounds like godaddy does, at least.
      - mpyne 13y ago
        That's up to the CA. It seems that StartSSL doesn't automatically revoke an old one even with rekeying.
- falcolas 13y ago
  Yup, as part of the cert re-issue, we revoked the previous cert. Could have been more explicit there, sorry.

001sky 13y ago

According to Paul Mutton, a security consultant at the Web services company Netcraft. Checking a site’s identity will take vastly longer. “If a certificate authority has to revoke 10,000 certificates, that entry will have 10,000 certificates on it,” Mutton said. “And if browsers have to download that . . . we’re talking hundreds of megabytes.” It’s roughly the equivalent of having to download 30 minutes’ worth of standard-definition video just to view a single Web page. > is this accurate?

nitrogen 13y ago
Would it be possible to use a probabilistic data structure (e.g. Bloom filter) to create a privacy-enabled CRL as a service?
- daeken 13y ago
  I don't see why not, especially because you already have the old key signature/fingerprint to use when pushing it into the Bloom filter. Seems like a perfect case for it, actually.
- gvb 13y ago
  You could distrust any certificate that was signed before April 7 (or some slightly later date). It might be easier to do that and whitelist the good "pre-apocalypse" certificates. The nice thing about doing this is that your whitelist will inherently go to zero as certificates time out and get renewed (typically 1-3 years?). The hard thing would be to generate an accurate whitelist.
  - mygrant 13y ago
    We had certs that were only used and present on machines that were not vulnerable, and therefore did not replace or revoke them. Please don't do this.
    - nathanvanfleet 13y ago
      Furthermore servers that weren't even using OpenSSL or the particular version(s) that were susceptible. What is the actual percentage of servers that needed to react to this?

matt__rose 13y ago

This sentence: "No examples have surfaced of anyone actually exploiting the vulnerability" is incorrect. The Canada Revenue Agency revealed that 900 SINs (canadian equivalent of SSNs, but nowhere near as totemically identifying) were stolen due to the heartbleed bug. http://www.theglobeandmail.com/technology/mounties-chasing-viable-lead-in-cras-heartbleed-breach/article18002731/ http://www.theglobeandmail.com/technology/mounties-chasing-v...

justinhj 13y ago
In order to know this they must have stored and subsequently analysed the ssl traffic. Something that I would think is quite unusual. So the true scope of data theft is probably much larger.
- simcop2387 13y ago
  They might have turned on that logging when the bug was announced so that they'd have an idea of the threat they were under while updating.
thisisdave 13y ago
I thought that sentence was referring specifically to vulnerable certificates.

sitkack 13y ago

1) Lets just call a do-over, EVERYONE generate new private keys and request new certs. Any cert created before April 10 2014 will be considered suspect. 2) Create a bloom filter for revoked certs. Have a remote cert revocation server/cache so I don't need to have a 50MB file of revocations.

the_ancient 13y ago
while the problem is wide spread, it is no where near enough to say "all certs after 4/10 are suspect" There are an ever larger number of certs not impacted by this event.
jerf 13y ago
False positives are a problem for a Bloom filter. You could conceivably create one for a given registrar, though, which could in theory create a Bloom filter, iterate through all the certs it has issued, and then list out the certs that the Bloom filter issues false positives for, but are still valid. However, it's too late to put that into play. Nobody ever expected to have to revoke this quantity of certs. (In hindsight, that was obviously wrong. Something like this was going to happen someday. But this is exactly the sort of thing that's very easy to argue away, because nobody wants to be convinced that this amount of work will be necessary.)

justinhj 13y ago

Question: with a stolen private key don't you still have the problem that is has the host name in it? So to truly spoof a browser you need to compromise the domain and serve a site from within it?

mpyne 13y ago
It depends on what you're going after, but if you're just targeting a local subnet, that's what a "man-in-the-middle" attack is for. What is probably easier is somehow redirecting the DNS entry for the target site to point to your malware-serving site, which has happened before as well.

kijin 13y ago

The situation would be much more manageable if browser vendors provided an online service that quickly allows individual certificates to be checked against the latest revocation list from all trusted CAs. There should be no need for each user to download the full revocation list, since most people will never visit 99.9% of the websites in that list. Until now, revocation lists were small enough that this was not a problem. But it's probably time to fix that. There are already third-party add-ons like Perspectives that check the certificates you see against those seen by other people. Chrome already knows which Google certs are valid or not. There's no reason why Mozilla, for example, shouldn't make something like this an official security feature of Firefox, too.

astrange 13y ago
This is OCSP. Chrome removed their support for it, but I think Firefox might check it.

mballantyne 13y ago

Do CRLs even work? My understanding is that the only browser that hard-fails to load a page if OCSP is blocked is Chrome for certs in CRLSets. Everyone else is vulnerable to MITM if access to the CRL / OCSP servers is blocked. I would love to be wrong. Does anyone know if anything has changed for the better since 2011? http://blog.spiderlabs.com/2011/04/certificate-revocation-behavior-in-modern-browsers.html http://blog.spiderlabs.com/2011/04/certificate-revocation-be... http://dev.chromium.org/Home/chromium-security/crlsets http://dev.chromium.org/Home/chromium-security/crlsets https://www.imperialviolet.org/2011/03/18/revocation.html https://www.imperialviolet.org/2011/03/18/revocation.html