4 ms·

Has anyone asked how two security researches supposedly found the same exact bug (that's been in the code for 2+ years) within days of each other? How likely is

Has anyone asked how two security researches supposedly found the same exact bug (that's been in the code for 2+ years) within days of each other? How likely is that?

mahmud 13y ago
Security researchers are social animals. There might have been a hint on IRC or mailinglist somewhere, and the other team got on it.
tptacek 13y ago
It happens routinely. Allow me to illustrate: on a pentest a couple years ago, Vitaly McLain from our Chicago team found that if you accidentally slipped a NUL into a header that nginx reverse-proxied, nginx would short-copy (it used strncpy) and reveal server memory. We reported to our client, and in the time it took for them to OK our upstream report --- a matter of hours --- someone else reported the same bug. What's funny about that is that the nginx bug is functionally identical to Heartbleed (modulo that it only worked on nginx).
- rpicard 13y ago
  I've been reading So Good They Can't Ignore You and there's an interesting section on this phenomenon: “The Baffling Popularity of Randomized Linear Network Coding As I write this chapter, I’m attending a computer science conference in San Jose, California. Earlier today, something interesting happened. I attended a session in which four different professors from four different universities presented their latest research. Surprisingly, all four presentations tackled the same narrow problem—information dissemination in networks—using the same narrow technique—randomized linear network coding. It was as if my research community woke up one morning and collectively and spontaneously decided to tackle the same esoteric problem. This example of joint discovery surprised me, but it would not have surprised the science writer Steven Johnson. In his engaging 2010 book, Where Good Ideas Come From, Johnson explains that such “multiples” are frequent in the history of science. Consider the discovery of sunspots in 1611: As Johnson notes, four scientists, from four different countries, all identified the phenomenon during that same year. The first electrical battery? Invented twice in the mid-eighteenth century. Oxygen? Isolated independently in 1772 and 1774. In one study cited by Johnson, researchers from Columbia University found just shy of 150 different examples of prominent scientific breakthroughs made by multiple researchers at near the same time.” Excerpt From: Newport, Cal. “So Good They Can't Ignore You: Why Skills Trump Passion in the Quest for Work You Love.” His thesis on this is basically that these discoveries depend on a lot of other things occurring first and that once those things have occurred, anyone looking in the right place will see it. With all of that said, I remain skeptical, at least in the Heartbleed case (they're just SOO close together). tptacek has more experience in these things than me of course, so I'll defer to his thoughts.
- azakai 13y ago
  > It happens routinely. I think that still leaves open the question of how likely it is, or perhaps the question is better phrased as whether it is due to chance or not. It is extremely unlikely to be due to chance in this case. It was out for years, but discovered independently within days of each other. One possibility is that it was discovered independently many times, but we only find out about it after someone actually goes public. At that point, those finding it earlier either never wanted to mention publicly that they found it (or they already would have), or even if they want to, they would be admitting to finding it earlier but not disclosing, so they'd look bad. That leaves others finding it slightly after the group that goes public - soon enough that it wasn't public yet, so they have evidence that they actually did find it independently. That would lead to exactly the result we see here, in fact. But this does imply that there were, very likely, multiple other groups that found this earlier but never went public. The only other option is that it wasn't random, but some event led to both discoveries. Perhaps a hint was around, a new method of finding vulnerabilities, or anything at all that could nudge people's minds in that direction. That option is much less worrying, but very hard to prove.
  - tptacek 13y ago
    I think if you reread my comment closely, you'll see that it describes a scenario that is spookily similar to Heartbleed: same impact, same latent bug (presumably in the code for many years, maybe even longer than TLS heartbeats), and co-discovery not in a matter of days but hours. Sorry, there's probably nothing interesting happening here, except for coincidence.
    - azakai 13y ago
      Yes, it is spookily similar, I agree 100% (I did read it carefully ;) What I disagree with is that one event happening means that another event very similar to it is likely in a statistical sense. Of course, it is an argument that supports that to some extent. But it is fairly weak support, when on the other hand statistics strongly imply the opposite.
      - jballanc 13y ago
- syncsynchalt 13y ago
  An aside - was it really strncpy? That function has such baffling behavior (it creates unterminated strings on overflow — contrary to its str* name) that I'm surprised it isn't banned by every static checker in existence.
gojomo 13y ago
The most innocent explanation would be pure coincidence, or a coincidence slightly nudged by the fact the recent Apple/GnuTLS bugs are causing an industry-wide look-back at similar old under-reviewed open-source code. A more sinister explanation would be that evidence of exploitation helped focus attention by each team once it touched them. However, a colleague of Neel Mehta implies that this was an audit-driven discovery without regard to active exploitation (https://news.ycombinator.com/item?id=7558015 https://news.ycombinator.com/item?id=7558015). That could still leave the possibility that news of Mehta's discovery leaked, as either a vague hint or as enough info to create larger scans, which then helped tip off the Codenomicon group. Despite all the reasons for secrecy, non-disclosure, and protection of proprietary methods, I hope each discoverer eventually says more about the steps leading up to discovery.
- azakai 13y ago
  Coincidence seems very unlikely, but the other explanations you suggest do seem plausible. Hopefully it's one of those, because otherwise given it's unlikely, it means it was most likely discovered earlier multiple times but not disclosed.
- tiredoldfart 13y ago
  http://www.openssl.org/news/secadv_20040317.txt http://www.openssl.org/news/secadv_20040317.txt -> http://www.openssl.org/news/secadv_20080528.txt http://www.openssl.org/news/secadv_20080528.txt -> http://www.openssl.org/news/secadv_20120510.txt http://www.openssl.org/news/secadv_20120510.txt -> heartbleed ... a pattern emerges? At least for another one of the discoverers. :) Furthermore "Goto Fail;" is actually cited twice on the http://heartbleed.com http://heartbleed.com and GnuTLS bug once.
robert_tweed 13y ago
Perhaps not as unlikely as you might think: http://en.wikipedia.org/wiki/Multiple_discovery http://en.wikipedia.org/wiki/Multiple_discovery I remember years ago reading about how some researchers were investigating a specific variation of this phenomenon, in which a number of independent and geographically dispersed labs would be, for example, trying to grow a particular crystal. For a number of years none were successful. Then within a matter of weeks of each other, all of the labs would independently discover the same technique that made it possible. This happened so close together that pure coincidence was incredibly unlikely. Apparently this same phenomenon was occurring far more often than it should in theory (based on the probability of pure coincidences), which is what prompted the research into it. Unfortunately, I don't know what the final outcome was. It would be amazing if anyone else remembers this and can point out the original source. Personally I am inclined to believe that it is simply a matter of prior events causing multiple individuals to be thinking about the same things. If you have lots of people searching in the same locality, there's a much higher chance that they will all find the same things around the same time than if they are all operating in a truly random search space.
rodgerd 13y ago
You might as well ask how two people (that we know of) discovered calculus independently in much the same timeframe.
deleted 13y ago
[deleted]

Has anyone asked how two security researches supposedly found the same exact bug (that's been in the code for 2+ years) within days of each other? How likely is that?

mahmud 13y ago

Security researchers are social animals. There might have been a hint on IRC or mailinglist somewhere, and the other team got on it.

tptacek 13y ago

It happens routinely. Allow me to illustrate: on a pentest a couple years ago, Vitaly McLain from our Chicago team found that if you accidentally slipped a NUL into a header that nginx reverse-proxied, nginx would short-copy (it used strncpy) and reveal server memory. We reported to our client, and in the time it took for them to OK our upstream report --- a matter of hours --- someone else reported the same bug. What's funny about that is that the nginx bug is functionally identical to Heartbleed (modulo that it only worked on nginx).

rpicard 13y ago
I've been reading So Good They Can't Ignore You and there's an interesting section on this phenomenon: “The Baffling Popularity of Randomized Linear Network Coding As I write this chapter, I’m attending a computer science conference in San Jose, California. Earlier today, something interesting happened. I attended a session in which four different professors from four different universities presented their latest research. Surprisingly, all four presentations tackled the same narrow problem—information dissemination in networks—using the same narrow technique—randomized linear network coding. It was as if my research community woke up one morning and collectively and spontaneously decided to tackle the same esoteric problem. This example of joint discovery surprised me, but it would not have surprised the science writer Steven Johnson. In his engaging 2010 book, Where Good Ideas Come From, Johnson explains that such “multiples” are frequent in the history of science. Consider the discovery of sunspots in 1611: As Johnson notes, four scientists, from four different countries, all identified the phenomenon during that same year. The first electrical battery? Invented twice in the mid-eighteenth century. Oxygen? Isolated independently in 1772 and 1774. In one study cited by Johnson, researchers from Columbia University found just shy of 150 different examples of prominent scientific breakthroughs made by multiple researchers at near the same time.” Excerpt From: Newport, Cal. “So Good They Can't Ignore You: Why Skills Trump Passion in the Quest for Work You Love.” His thesis on this is basically that these discoveries depend on a lot of other things occurring first and that once those things have occurred, anyone looking in the right place will see it. With all of that said, I remain skeptical, at least in the Heartbleed case (they're just SOO close together). tptacek has more experience in these things than me of course, so I'll defer to his thoughts.
azakai 13y ago
> It happens routinely. I think that still leaves open the question of how likely it is, or perhaps the question is better phrased as whether it is due to chance or not. It is extremely unlikely to be due to chance in this case. It was out for years, but discovered independently within days of each other. One possibility is that it was discovered independently many times, but we only find out about it after someone actually goes public. At that point, those finding it earlier either never wanted to mention publicly that they found it (or they already would have), or even if they want to, they would be admitting to finding it earlier but not disclosing, so they'd look bad. That leaves others finding it slightly after the group that goes public - soon enough that it wasn't public yet, so they have evidence that they actually did find it independently. That would lead to exactly the result we see here, in fact. But this does imply that there were, very likely, multiple other groups that found this earlier but never went public. The only other option is that it wasn't random, but some event led to both discoveries. Perhaps a hint was around, a new method of finding vulnerabilities, or anything at all that could nudge people's minds in that direction. That option is much less worrying, but very hard to prove.
- tptacek 13y ago
  I think if you reread my comment closely, you'll see that it describes a scenario that is spookily similar to Heartbleed: same impact, same latent bug (presumably in the code for many years, maybe even longer than TLS heartbeats), and co-discovery not in a matter of days but hours. Sorry, there's probably nothing interesting happening here, except for coincidence.
  - azakai 13y ago
    Yes, it is spookily similar, I agree 100% (I did read it carefully ;) What I disagree with is that one event happening means that another event very similar to it is likely in a statistical sense. Of course, it is an argument that supports that to some extent. But it is fairly weak support, when on the other hand statistics strongly imply the opposite.
    - jballanc 13y ago
syncsynchalt 13y ago
An aside - was it really strncpy? That function has such baffling behavior (it creates unterminated strings on overflow — contrary to its str* name) that I'm surprised it isn't banned by every static checker in existence.

gojomo 13y ago

The most innocent explanation would be pure coincidence, or a coincidence slightly nudged by the fact the recent Apple/GnuTLS bugs are causing an industry-wide look-back at similar old under-reviewed open-source code. A more sinister explanation would be that evidence of exploitation helped focus attention by each team once it touched them. However, a colleague of Neel Mehta implies that this was an audit-driven discovery without regard to active exploitation (https://news.ycombinator.com/item?id=7558015 https://news.ycombinator.com/item?id=7558015). That could still leave the possibility that news of Mehta's discovery leaked, as either a vague hint or as enough info to create larger scans, which then helped tip off the Codenomicon group. Despite all the reasons for secrecy, non-disclosure, and protection of proprietary methods, I hope each discoverer eventually says more about the steps leading up to discovery.

azakai 13y ago
Coincidence seems very unlikely, but the other explanations you suggest do seem plausible. Hopefully it's one of those, because otherwise given it's unlikely, it means it was most likely discovered earlier multiple times but not disclosed.
tiredoldfart 13y ago
http://www.openssl.org/news/secadv_20040317.txt http://www.openssl.org/news/secadv_20040317.txt -> http://www.openssl.org/news/secadv_20080528.txt http://www.openssl.org/news/secadv_20080528.txt -> http://www.openssl.org/news/secadv_20120510.txt http://www.openssl.org/news/secadv_20120510.txt -> heartbleed ... a pattern emerges? At least for another one of the discoverers. :) Furthermore "Goto Fail;" is actually cited twice on the http://heartbleed.com http://heartbleed.com and GnuTLS bug once.

robert_tweed 13y ago

Perhaps not as unlikely as you might think: http://en.wikipedia.org/wiki/Multiple_discovery http://en.wikipedia.org/wiki/Multiple_discovery I remember years ago reading about how some researchers were investigating a specific variation of this phenomenon, in which a number of independent and geographically dispersed labs would be, for example, trying to grow a particular crystal. For a number of years none were successful. Then within a matter of weeks of each other, all of the labs would independently discover the same technique that made it possible. This happened so close together that pure coincidence was incredibly unlikely. Apparently this same phenomenon was occurring far more often than it should in theory (based on the probability of pure coincidences), which is what prompted the research into it. Unfortunately, I don't know what the final outcome was. It would be amazing if anyone else remembers this and can point out the original source. Personally I am inclined to believe that it is simply a matter of prior events causing multiple individuals to be thinking about the same things. If you have lots of people searching in the same locality, there's a much higher chance that they will all find the same things around the same time than if they are all operating in a truly random search space.

rodgerd 13y ago

You might as well ask how two people (that we know of) discovered calculus independently in much the same timeframe.

deleted 13y ago

[deleted]