14 ms·

NSA Said to Have Used Heartbleed Bug for at Least Two Years (2014)

285 points by ColanR 7y ago

java-man 7y ago
To this day I think it was a deliberately placed vulnerability. I don't have any data to back it up, of course.
- hn_throwaway_99 7y ago
  I sincerely, sincerely doubt it. Just look at the history of the actual bug (not hard at all to believe it could have happened by accident), and the fact of how undermanned OpenSSL was, and I'm just surprised it didn't happen sooner.
- robocat 7y ago
  Your problem is that absolutely anything can be explained by a conspiracy. This case was a few guys writing software in their free time - 100% likely it was an honest mistake. See https://www.buzzfeed.com/chrisstokelwalker/the-internet-is-being-protected-by-two-guys-named-st https://www.buzzfeed.com/chrisstokelwalker/the-internet-is-b...
  - nickpsecurity 7y ago
    What's funny about what you linked is a government demand led to the vulnerability at a point when owners or main people were thinking of walking away. A conspiracy around that would be more believable than most. Let's ignore that, though. The thing is, they add vulnerabilities in a number of ways. They can do it directly with code. They can do it indirectly with standards hard to code correctly w/out vulnerabilities or side channels. There's lots of options. Whatever they do will usually look like a helpful contribution or useful requirement that went wrong in a way that leads to an attack. The better ones are those that look like common or inevitable errors. That's because obvious backdoors make folks run away from a project or supplier maybe forever on top of question who put it there. So, it's usually these flaws that look like obvious errors that still get the job done with everyone around defending the person that put them there. And I'm not saying it was an NSA job. I have no idea. They've been doing too good of a job on most things for me to know. Could've been an accident. Even probability supports it's an accident just like it did all the times it was a subversion. At $200+ mil a year budget for backdoors/hacking, you bet there were a lot of accidents that, in non-TS version, had nothing to do with the NSA. ;) Edit: There's lots of questionable things in this article. My favorite is this: "And this group are the best of the best of the best." The OpenBSD team doing LibreSSL had all kinds of summaries, live updates, and even presentations of what they found. It was about as far from that quote as you could imagine. Although my memory sucks, I think at one point they said there was even code that checked to see if endianness changed while it was operating. They at least had that covered. There were so many oddities about that codebase.
    - java-man 7y ago
      Exactly. I don't have any evidence to back up my suspicion. If the evidence exists, it exists in a locked safe somewhere at Ft Meade (or other place) and in the brains of a very few people. I can feel what is known as code smell. So, let's develop a new feature in the most widely used security library. The very first thing that must be done is to sanitize network input. This is the first thing I would expect to be done by a seasoned developer. The lack of this check is suspicious. It could be an honest mistake, of course - we all make mistakes, and I am sure I've made my share of idiotic changes. But this isn't something I would expect about OpenSSL. I agree with @nickpsecurity, "many oddities". Commit that introduced the vulnerability: https://git.openssl.org/gitweb/?a=commit&h=4817504d069b4c5082161b02a22116ad75f822b1&p=openssl.git https://git.openssl.org/gitweb/?a=commit&h=4817504d069b4c508... Fix: https://github.com/openssl/openssl/commit/96db9023b881d7cd9f379b0c154650d6c108e9a3 https://github.com/openssl/openssl/commit/96db9023b881d7cd9f...
    - peter_d_sherman 7y ago
      LibreSSL: More Than 30 Days Later https://www.openbsd.org/papers/eurobsdcon2014-libressl.html https://www.openbsd.org/papers/eurobsdcon2014-libressl.html Not pretty to read from a security perspective... But, on the other hand, quite eye opening, if you want to have your eyes opened...
      - nickpsecurity 7y ago
        Thanks! I'll add it to book.arks for next time this comes up.
  - OrgNet 7y ago
    And your problem is that absolutely anything can be explained by malicious intent.
- jMyles 7y ago
  That's a serious and damning accusation to level against a volunteer contributor to an open source project, and in very bad taste if indeed you have no evidence (or even a reasoned narrative of the hows and whys).
  - java-man 7y ago
    in a sibling comment. tl,dr: the first thing one does in this field is to sanitize the input.
    - thephyber 7y ago
      I would also argue that processes and tools should supplement developer mistakes, negligence, or maliciousness. Unit tests, static analysis, fuzzing, integration tests, security audits, code reviews, principle of least privilege, etc. all have a part to play and yet this lapse in validation still managed to make it into production and infect all of the downstream libraries and applications. I would argue that even if you could pin an accusation of negligence on the developer (I've not seen any evidence that could substantiate this accusation), it doesn't rest only with that one developer. The project itself lacked redundant checks. The downstream applications that import OpenSSL similarly failed to audit it. I think in the whole scheme of things, the Open Source movement had a lot of momentum by the time that code was written, but the corporations that relied on the benefits of open source largely didn't contribute to paying to maintain highly secure coding practices. HeartBleed was one of the incidents that made the internet infrastructure/platform companies (among others) start paying for humans, tools, and reviews to help make these common libraries more secure. Google's Project Zero was started in July 2014, soon after HeartBleed was announced.
      - java-man 7y ago
        thank you.
- cjbprime 7y ago
  It sounds like you probably haven't worked on a lot of C code. OpenSSL is a giant security vulnerability pile. It is not at all hard to believe that someone has added a vulnerability by mistake: it would be more surprising to me if they hadn't.
hannob 7y ago
This was never confirmed with any evidence and bloomberg only cites anonymous sources. Given Bloomberg's bad track record on infosec stories I say I have my doubts. Of course I don't know whether or not the NSA knew about Heartbleed. But nothing that would even remotely qualify as evidence has ever been presented.
- ClumsyPilot 7y ago
  on the balance of probabilities, it is probably true. I would expect NSA to make use of any vulnerabilities they find, because their job is to hack others, not to keep us safe. Unfortunately.
  - wnevets 7y ago
    Well the real question isn't if NSA used it but if they knew about it long before everyone else and never reported it.
    - grumio 7y ago
      It's their SOP. See: eternal blue. https://en.m.wikipedia.org/wiki/EternalBlue https://en.m.wikipedia.org/wiki/EternalBlue
    - tptacek 7y ago
      That's how CNO exploitation works. They generally can't report; their adversaries are recording their own networks, and will retrospectively detect intrusions.
  - jjtheblunt 7y ago
    NSA is resposible for so-called SIGINT and SIGSEC, acronyms for signals intelligence, to which you refer, and signals security, which IS to keep our communications safe. It seems of course that SIGINT is what's "popular" in news.
    - deleted 7y ago
      [deleted]
ENOTTY 7y ago
> The NSA has issued a statement denying the report. In an email to Ars, NSA spokesperson Vanee VInes provided this official statement: “NSA was not aware of the recently identified vulnerability in OpenSSL, the so-called Heartbleed vulnerability, until it was made public in a private-sector cybersecurity report. Reports that say otherwise are wrong.” https://arstechnica.com/information-technology/2014/04/nsa-used-heartbleed-nearly-from-the-start-report-claims/ https://arstechnica.com/information-technology/2014/04/nsa-u...
- luxuryballs 7y ago
  Riiight, Snowden even said in Citizen Four that they claim their access is going dark but in reality it just keeps getting better and better. Oh and don’t forget James Clapper, who TOLD CONGRESS that they weren’t collecting American’s phone records, or at least “not wittingly”. Don’t mind if I tinfoil hat a little bit but I wouldn’t be surprised if “Intel” put that in there just for them.
- thaumaturgy 7y ago
  Oh. Are we still taking NSA's statements at face value post-Snowden?
  - LeoPanthera 7y ago
    They're at least as valuable as Bloomberg's anonymous sources. You can't choose which sources you trust more just because they agree with your pre-existing beliefs.
  - thephyber 7y ago
    I'd rather the PR statement be added to the discussion, even if not all readers trust it.
- burtonator 7y ago
  What's the legality of using a vulnerability to monitor someone if you have a warrant? Seems like it would still be illegal.
OrgNet 7y ago
I thought that we knew that they don't report bugs that they know about (because they don't care about the greater good).
williesleg 7y ago
Bloomberg is fake news. They are so compromised by advertising telling them what to write it's sickening. On top of that, they hate the USA and love China.
Railsify 7y ago
Excellent, US government putting my money to work!
ngneer 7y ago
This is 2019 and the security community has yet to deliver a proper solution to prevent the existence of such bugs. The mismatch between programmer intent and code behavior is appalling. Sure, super smart coders can avoid the bugs, much like super safe drivers can avoid the shoulder, but rumble strips are there for a reason. The bug would not have arisen if the language supported dependent typing. See Agda, for example. One day... https://en.wikipedia.org/wiki/Dependent_type https://en.wikipedia.org/wiki/Dependent_type
- cjbprime 7y ago
  > security community has yet to deliver a proper solution What do you want them to do? I think their solution is "use memory-safe languages like Golang or Rust (or even JavaScript/TypeScript) for new projects, not C or C++" and "use extensive fuzzing on legacy C code that hasn't been replaced yet". Fuzzing was capable of finding Heartbleed, and it's advanced massively (and been set up at scale to continuously test open source projects) since then.
  - deleted 7y ago
    [deleted]
  - guerrilla 7y ago
    > What do you want them to do? If you read to the end of the comment, it suggests dependent typing as the solution.
  - z3phyr 7y ago
    Rust is pretty useful against memory based bugs, and it is prudent that new projects take advantage of it. But this in no way fixes the headache of security, because (1) almost all of the existing infrastructure code depends on C (2) New classes of bugs will eventually emmerge, some partly unfixable by software solution.
    - carlmr 7y ago
      >New classes of bugs will eventually emmerge, some partly unfixable by software solution. Sure Rust, ADA and such don't remove all classes of bugs, but they can reduce the attack surface considerably, giving you more time to focus on the remaining security bugs. And maybe people will invent software solutions that reduce the attack surface even more. Assembly is a fast car with no security features. C is a sportscar with a seatbelt. Rust is a sportscar with a seatbelt, airbags, ABS, ESC and emergency breaking. Of course you can still crash and die, it's just that you're less likely to do so.
emmelaich 7y ago
The weird thing to me is that heartbleed depended on a feature in TLS that required the server to receive a request from the client specifying a size. Is there any other similar TLS feature? Even if this is not the only one, wouldn't you audit the shit out of that?
jokoon 7y ago
So far the state of the computer industry is pretty simple, if you're using american products, you're under american surveillance. Governments will always seize the monopoly of security, that's how civilization works. And even if you're using open source, I'm sure the NSA has written tools that can scan source code to find vulnerabilities, and maybe generate exploit if they sprinkled some ML on it. To be honest I'd rather have a government body have the monopoly of security than witness a cybersecurity chaos, which would quickly destroy the internet. The problem is that only the US does it well.
- mk89 7y ago
  > The problem is that only the US does it well. The main difference with other countries is that we all know about NSA thanks to what happened to people like Snowden, otherwise it would not be any different from other countries -> pure speculation. And why would the US do it better than countries like China, Russia or France?
  - jokoon 7y ago
    Because the US has the silicon valley, historically it invented the internet and modern computers, the US holds most of the core tech companies (intel, AMD, microsoft, google). The US just have much more expertise and engineers, which is essential if you want the NSA to recruit and be the best at what it does. It has many aspects, I guess cerebral and technical capital are important notions. Even if other countries can compete with the US on cybersecurity, the US is holding most of the data but also is writing most of the software and designing everything around computers, so it makes it trivial for them to turn those products against other countries who buy them. Except linux, I really don't see any computer product that doesn't have critical parts or system made in the US. And as I said I'm certain the NSA can exploit open source very easily since it's a problem with solutions: Torvalds said "given enough eyeballs, all bugs are shallow". That is true, but if NSA is supplying eyeballs to find vulnerabilities and use them at their advantage, linux will be an asset for the US.
    - kelnos 7y ago
harshvladha 7y ago
If someone is not able to read article because Bloomberg asks for premium fees, then don't miss out to check out their HTML ;)
paulcarroty 7y ago
I don't fully trust Bloomberg, but there's funny thing: nobody hit them hard in court. So, logically, it means a part of their materials are true and evidences exists.

285 points by ColanR 7y ago

java-man 7y ago

To this day I think it was a deliberately placed vulnerability. I don't have any data to back it up, of course.

hn_throwaway_99 7y ago
I sincerely, sincerely doubt it. Just look at the history of the actual bug (not hard at all to believe it could have happened by accident), and the fact of how undermanned OpenSSL was, and I'm just surprised it didn't happen sooner.
robocat 7y ago
Your problem is that absolutely anything can be explained by a conspiracy. This case was a few guys writing software in their free time - 100% likely it was an honest mistake. See https://www.buzzfeed.com/chrisstokelwalker/the-internet-is-being-protected-by-two-guys-named-st https://www.buzzfeed.com/chrisstokelwalker/the-internet-is-b...
- nickpsecurity 7y ago
  What's funny about what you linked is a government demand led to the vulnerability at a point when owners or main people were thinking of walking away. A conspiracy around that would be more believable than most. Let's ignore that, though. The thing is, they add vulnerabilities in a number of ways. They can do it directly with code. They can do it indirectly with standards hard to code correctly w/out vulnerabilities or side channels. There's lots of options. Whatever they do will usually look like a helpful contribution or useful requirement that went wrong in a way that leads to an attack. The better ones are those that look like common or inevitable errors. That's because obvious backdoors make folks run away from a project or supplier maybe forever on top of question who put it there. So, it's usually these flaws that look like obvious errors that still get the job done with everyone around defending the person that put them there. And I'm not saying it was an NSA job. I have no idea. They've been doing too good of a job on most things for me to know. Could've been an accident. Even probability supports it's an accident just like it did all the times it was a subversion. At $200+ mil a year budget for backdoors/hacking, you bet there were a lot of accidents that, in non-TS version, had nothing to do with the NSA. ;) Edit: There's lots of questionable things in this article. My favorite is this: "And this group are the best of the best of the best." The OpenBSD team doing LibreSSL had all kinds of summaries, live updates, and even presentations of what they found. It was about as far from that quote as you could imagine. Although my memory sucks, I think at one point they said there was even code that checked to see if endianness changed while it was operating. They at least had that covered. There were so many oddities about that codebase.
  - java-man 7y ago
    Exactly. I don't have any evidence to back up my suspicion. If the evidence exists, it exists in a locked safe somewhere at Ft Meade (or other place) and in the brains of a very few people. I can feel what is known as code smell. So, let's develop a new feature in the most widely used security library. The very first thing that must be done is to sanitize network input. This is the first thing I would expect to be done by a seasoned developer. The lack of this check is suspicious. It could be an honest mistake, of course - we all make mistakes, and I am sure I've made my share of idiotic changes. But this isn't something I would expect about OpenSSL. I agree with @nickpsecurity, "many oddities". Commit that introduced the vulnerability: https://git.openssl.org/gitweb/?a=commit&h=4817504d069b4c5082161b02a22116ad75f822b1&p=openssl.git https://git.openssl.org/gitweb/?a=commit&h=4817504d069b4c508... Fix: https://github.com/openssl/openssl/commit/96db9023b881d7cd9f379b0c154650d6c108e9a3 https://github.com/openssl/openssl/commit/96db9023b881d7cd9f...
  - peter_d_sherman 7y ago
    LibreSSL: More Than 30 Days Later https://www.openbsd.org/papers/eurobsdcon2014-libressl.html https://www.openbsd.org/papers/eurobsdcon2014-libressl.html Not pretty to read from a security perspective... But, on the other hand, quite eye opening, if you want to have your eyes opened...
    - nickpsecurity 7y ago
      Thanks! I'll add it to book.arks for next time this comes up.
- OrgNet 7y ago
  And your problem is that absolutely anything can be explained by malicious intent.
jMyles 7y ago
That's a serious and damning accusation to level against a volunteer contributor to an open source project, and in very bad taste if indeed you have no evidence (or even a reasoned narrative of the hows and whys).
- java-man 7y ago
  in a sibling comment. tl,dr: the first thing one does in this field is to sanitize the input.
  - thephyber 7y ago
    I would also argue that processes and tools should supplement developer mistakes, negligence, or maliciousness. Unit tests, static analysis, fuzzing, integration tests, security audits, code reviews, principle of least privilege, etc. all have a part to play and yet this lapse in validation still managed to make it into production and infect all of the downstream libraries and applications. I would argue that even if you could pin an accusation of negligence on the developer (I've not seen any evidence that could substantiate this accusation), it doesn't rest only with that one developer. The project itself lacked redundant checks. The downstream applications that import OpenSSL similarly failed to audit it. I think in the whole scheme of things, the Open Source movement had a lot of momentum by the time that code was written, but the corporations that relied on the benefits of open source largely didn't contribute to paying to maintain highly secure coding practices. HeartBleed was one of the incidents that made the internet infrastructure/platform companies (among others) start paying for humans, tools, and reviews to help make these common libraries more secure. Google's Project Zero was started in July 2014, soon after HeartBleed was announced.
    - java-man 7y ago
      thank you.
cjbprime 7y ago
It sounds like you probably haven't worked on a lot of C code. OpenSSL is a giant security vulnerability pile. It is not at all hard to believe that someone has added a vulnerability by mistake: it would be more surprising to me if they hadn't.

hannob 7y ago

This was never confirmed with any evidence and bloomberg only cites anonymous sources. Given Bloomberg's bad track record on infosec stories I say I have my doubts. Of course I don't know whether or not the NSA knew about Heartbleed. But nothing that would even remotely qualify as evidence has ever been presented.

ClumsyPilot 7y ago
on the balance of probabilities, it is probably true. I would expect NSA to make use of any vulnerabilities they find, because their job is to hack others, not to keep us safe. Unfortunately.
- wnevets 7y ago
  Well the real question isn't if NSA used it but if they knew about it long before everyone else and never reported it.
  - grumio 7y ago
    It's their SOP. See: eternal blue. https://en.m.wikipedia.org/wiki/EternalBlue https://en.m.wikipedia.org/wiki/EternalBlue
  - tptacek 7y ago
    That's how CNO exploitation works. They generally can't report; their adversaries are recording their own networks, and will retrospectively detect intrusions.
- jjtheblunt 7y ago
  NSA is resposible for so-called SIGINT and SIGSEC, acronyms for signals intelligence, to which you refer, and signals security, which IS to keep our communications safe. It seems of course that SIGINT is what's "popular" in news.
  - deleted 7y ago
    [deleted]

ENOTTY 7y ago

> The NSA has issued a statement denying the report. In an email to Ars, NSA spokesperson Vanee VInes provided this official statement: “NSA was not aware of the recently identified vulnerability in OpenSSL, the so-called Heartbleed vulnerability, until it was made public in a private-sector cybersecurity report. Reports that say otherwise are wrong.” https://arstechnica.com/information-technology/2014/04/nsa-used-heartbleed-nearly-from-the-start-report-claims/ https://arstechnica.com/information-technology/2014/04/nsa-u...

luxuryballs 7y ago
Riiight, Snowden even said in Citizen Four that they claim their access is going dark but in reality it just keeps getting better and better. Oh and don’t forget James Clapper, who TOLD CONGRESS that they weren’t collecting American’s phone records, or at least “not wittingly”. Don’t mind if I tinfoil hat a little bit but I wouldn’t be surprised if “Intel” put that in there just for them.
thaumaturgy 7y ago
Oh. Are we still taking NSA's statements at face value post-Snowden?
- LeoPanthera 7y ago
  They're at least as valuable as Bloomberg's anonymous sources. You can't choose which sources you trust more just because they agree with your pre-existing beliefs.
- thephyber 7y ago
  I'd rather the PR statement be added to the discussion, even if not all readers trust it.
burtonator 7y ago
What's the legality of using a vulnerability to monitor someone if you have a warrant? Seems like it would still be illegal.

OrgNet 7y ago

I thought that we knew that they don't report bugs that they know about (because they don't care about the greater good).

williesleg 7y ago

Bloomberg is fake news. They are so compromised by advertising telling them what to write it's sickening. On top of that, they hate the USA and love China.

Railsify 7y ago

Excellent, US government putting my money to work!

ngneer 7y ago

This is 2019 and the security community has yet to deliver a proper solution to prevent the existence of such bugs. The mismatch between programmer intent and code behavior is appalling. Sure, super smart coders can avoid the bugs, much like super safe drivers can avoid the shoulder, but rumble strips are there for a reason. The bug would not have arisen if the language supported dependent typing. See Agda, for example. One day... https://en.wikipedia.org/wiki/Dependent_type https://en.wikipedia.org/wiki/Dependent_type

cjbprime 7y ago
> security community has yet to deliver a proper solution What do you want them to do? I think their solution is "use memory-safe languages like Golang or Rust (or even JavaScript/TypeScript) for new projects, not C or C++" and "use extensive fuzzing on legacy C code that hasn't been replaced yet". Fuzzing was capable of finding Heartbleed, and it's advanced massively (and been set up at scale to continuously test open source projects) since then.
- deleted 7y ago
  [deleted]
- guerrilla 7y ago
  > What do you want them to do? If you read to the end of the comment, it suggests dependent typing as the solution.
- z3phyr 7y ago
  Rust is pretty useful against memory based bugs, and it is prudent that new projects take advantage of it. But this in no way fixes the headache of security, because (1) almost all of the existing infrastructure code depends on C (2) New classes of bugs will eventually emmerge, some partly unfixable by software solution.
  - carlmr 7y ago
    >New classes of bugs will eventually emmerge, some partly unfixable by software solution. Sure Rust, ADA and such don't remove all classes of bugs, but they can reduce the attack surface considerably, giving you more time to focus on the remaining security bugs. And maybe people will invent software solutions that reduce the attack surface even more. Assembly is a fast car with no security features. C is a sportscar with a seatbelt. Rust is a sportscar with a seatbelt, airbags, ABS, ESC and emergency breaking. Of course you can still crash and die, it's just that you're less likely to do so.

emmelaich 7y ago

The weird thing to me is that heartbleed depended on a feature in TLS that required the server to receive a request from the client specifying a size. Is there any other similar TLS feature? Even if this is not the only one, wouldn't you audit the shit out of that?

jokoon 7y ago

So far the state of the computer industry is pretty simple, if you're using american products, you're under american surveillance. Governments will always seize the monopoly of security, that's how civilization works. And even if you're using open source, I'm sure the NSA has written tools that can scan source code to find vulnerabilities, and maybe generate exploit if they sprinkled some ML on it. To be honest I'd rather have a government body have the monopoly of security than witness a cybersecurity chaos, which would quickly destroy the internet. The problem is that only the US does it well.

mk89 7y ago
> The problem is that only the US does it well. The main difference with other countries is that we all know about NSA thanks to what happened to people like Snowden, otherwise it would not be any different from other countries -> pure speculation. And why would the US do it better than countries like China, Russia or France?
- jokoon 7y ago
  Because the US has the silicon valley, historically it invented the internet and modern computers, the US holds most of the core tech companies (intel, AMD, microsoft, google). The US just have much more expertise and engineers, which is essential if you want the NSA to recruit and be the best at what it does. It has many aspects, I guess cerebral and technical capital are important notions. Even if other countries can compete with the US on cybersecurity, the US is holding most of the data but also is writing most of the software and designing everything around computers, so it makes it trivial for them to turn those products against other countries who buy them. Except linux, I really don't see any computer product that doesn't have critical parts or system made in the US. And as I said I'm certain the NSA can exploit open source very easily since it's a problem with solutions: Torvalds said "given enough eyeballs, all bugs are shallow". That is true, but if NSA is supplying eyeballs to find vulnerabilities and use them at their advantage, linux will be an asset for the US.
  - kelnos 7y ago

harshvladha 7y ago

If someone is not able to read article because Bloomberg asks for premium fees, then don't miss out to check out their HTML ;)

paulcarroty 7y ago

I don't fully trust Bloomberg, but there's funny thing: nobody hit them hard in court. So, logically, it means a part of their materials are true and evidences exists.