4 ms·

Given enough eyeballs, all bugs are shallow. - Linus Torvalds

systems 9y ago
what is that supposed to mean "all bugs are shallow" .. does he mean easy to find, or obvious , or look small .. what does he mean?
- Zelphyr 9y ago
  I think he means that given enough people looking at a particular code base, even the most obscure (deep) bugs will be found (made shallow).
  - DonHopkins 9y ago
    Please note that the quote is mis-attributed to Linus Torvalds, but it's actually from ESR. Reality has proven him quite wrong. http://heartbleed.com/ http://heartbleed.com/ Not only is ESR wrong, but also his mis-attributed slogan overpromises a false sense of security, which is dangerous. In response, he tried to construct a straw man argument that "proprietary software is worse than open source software", which does not in any way support his claim about "all bugs being shallow". http://esr.ibiblio.org/?p=5665 http://esr.ibiblio.org/?p=5665 Anyone who thinks all bugs are shallow under any circumstances just hasn't seen many interesting real world bugs with their own eyes. Their experience is limited and their confidence in their software, security and mastery of programming and debugging is pure Dunning-Kruger effect. I'm with Ken on this one.
DonHopkins 9y ago
Actually, that quote was "formulated" by Eric S Raymond (to whom Theo was referring as "the originator of the statement"), and is only deceptively named "Linux's Law" [1] in "honor" of Linus Torvalds, which is ironic because it actually dishonors him by being invalid. The point that Theo was making is that ESR talks and talks and types and types about many eyeballs looking at code, but when it comes down to actually auditing code, he never actually bothers, and neither do most other of his minions who are so quick to parrot his ill-conceived "Linux's Law". Neither "enough eyeballs" nor "the right eyeballs" are a GIVEN, even for open source software. Google "Heartbleed". "Not enough eyeballs" (or "ZERO eyeballs" as he loves to claim) are NOT a GIVEN for proprietary software, because you can license much proprietary source code, and some proprietary source code is available for you to read and audit for free, under licenses like Microsoft's "Shared Source" license. https://en.wikipedia.org/wiki/Shared_source https://en.wikipedia.org/wiki/Shared_source And qualified eyeballs are NOT FREE, and usually very busy being well paid to look at much more interesting things than poorly written buggy code like OpenSSL. I doubt that Eric Raymond has contributed any of the profits from his books or VA Linux stocks to Theo De Raadt or anyone else who actually takes the long time and tedious effort to actually audit code. The one time ESR actually did try to audit some code didn't go so well: The little experience Raymond DOES have auditing code has been a total fiasco and embarrassing failure, since his understanding of the code was incompetent and deeply tainted by his preconceived political ideology and conspiracy theories about global warming, which was his only motivation for auditing the code in the first place. His sole quest was to discredit the scientists who warned about global warming. The code he found and highlighted was actually COMMENTED OUT, and he never addressed the fact that the scientists were vindicated. http://rationalwiki.org/wiki/Eric_S._Raymond http://rationalwiki.org/wiki/Eric_S._Raymond >During the Climategate fiasco, Raymond's ability to read other peoples' source code (or at least his honesty about it) was called into question when he was caught quote-mining analysis software written by the CRU researchers, presenting a commented-out section of source code used for analyzing counterfactuals as evidence of deliberate data manipulation. When confronted with the fact that scientists as a general rule are scrupulously honest, Raymond claimed it was a case of an "error cascade," a concept that makes sense in computer science and other places where all data goes through a single potential failure point, but in areas where outside data and multiple lines of evidence are used for verification, doesn't entirely make sense. (He was curiously silent when all the researchers involved were exonerated of scientific misconduct.) [1] https://en.wikipedia.org/wiki/Linus%27s_Law https://en.wikipedia.org/wiki/Linus%27s_Law Linus's Law is a claim about software development, named in honor of Linus Torvalds and formulated by Eric S. Raymond in his essay and book [redacted]. [...] Validity In Facts and Fallacies about Software Engineering, Robert Glass refers to the law as a "mantra" of the open source movement, but calls it a fallacy due to the lack of supporting evidence and because research has indicated that the rate at which additional bugs are uncovered does not scale linearly with the number of reviewers; rather, there is a small maximum number of useful reviewers, between two and four, and additional reviewers above this number uncover bugs at a much lower rate. While closed-source practitioners also promote stringent, independent code analysis during a software project's development, they focus on in-depth review by a few and not primarily the number of "eyeballs". Although detection of even deliberately inserted flaws can be attributed to Raymond's claim, the persistence of the Heartbleed security bug in a critical piece of code for two years has been considered as a refutation of Raymond's dictum. Larry Seltzer suspects that the availability of source code may cause some developers and researchers to perform less extensive tests than they would with closed source software, making it easier for bugs to remain. In 2015, the Linux Foundation's executive director Jim Zemlin argued that the complexity of modern software has increased to such levels that specific resource allocation is desirable to improve its security. Regarding some of 2014's largest global open source software vulnerabilities, he says, "In these cases, the eyeballs weren't really looking". Large scale experiments or peer-reviewed surveys to test how well the mantra holds in practice have not been performed. https://www.datamation.com/open-source/does-heartbleed-disprove-open-source-is-safer-1.html https://www.datamation.com/open-source/does-heartbleed-dispr... https://www.esecurityplanet.com/open-source-security/why-all-linux-security-bugs-arent-shallow.html https://www.esecurityplanet.com/open-source-security/why-all...
- countryslicker 9y ago
  That is a lot of words to say that it doesn't help to have access to the source code if qualified people don't perform adequate audits. In this regard, open source software is still better. Google "Heartbleed". When a neglected piece of critical infrastructure code was finally examined, it was shown to have serious flaws. Top talent immediately shifted their focus and began sorely needed maintenance. Nobody asked permission. Nobody called their lawyer. They just got to work to get-er-done. I think Theo was among those to help resolve issues. I am grateful for those eyeballs and those fingers.
  - deleted 9y ago
    [deleted]
- nickpsecurity 9y ago
  It would probably be better to use a survey of vulnerabilities plus Heartbleed instead of just repeating Heartbleed. It can be dismissed as an outlier: "That's just one project everyone was freeloading on." I know, it's a security-critical project that should've gotten many eyeballs. I just prefer to show how pervasive the problem is when debunking this stuff. The other thing I noted on a Heartbleed-related thread was that FOSS never produced high-assurance security despite its labor advantage. Proprietary sector, either industry or CompSci teams, beat them about every time. https://www.schneier.com/blog/archives/2014/04/reverse_heartbl.html#c5581737 https://www.schneier.com/blog/archives/2014/04/reverse_heart... I elaborate more in replies to DB. It comes down to them not caring enough to apply the level of QA necessary. You have to pay people to do that. You also have to find the right people that can do it since even the knowledge of it isn't widespread. They'll know 100 frameworks and such but not the basic activities for assurance past unit/acceptance testing and review. They'll probably have never heard of Ada/SPARK or SPIN when they tell you about Rust. So on and so forth. Raymond's claim is utter bullshit. If you want best security, you're better off buying a 3rd-party-evaluated product from high-assurance proprietary. If you want good security that's cheap, it's a small number of proprietary and FOSS projects whose stakeholders put time in for thorough review and analysis. The rest is shit waiting to happen or (looks at CVE list) happening all the time.
- 9y ago

what is that supposed to mean "all bugs are shallow" .. does he mean easy to find, or obvious , or look small .. what does he mean?

Zelphyr 9y ago
I think he means that given enough people looking at a particular code base, even the most obscure (deep) bugs will be found (made shallow).
- DonHopkins 9y ago
  Please note that the quote is mis-attributed to Linus Torvalds, but it's actually from ESR. Reality has proven him quite wrong. http://heartbleed.com/ http://heartbleed.com/ Not only is ESR wrong, but also his mis-attributed slogan overpromises a false sense of security, which is dangerous. In response, he tried to construct a straw man argument that "proprietary software is worse than open source software", which does not in any way support his claim about "all bugs being shallow". http://esr.ibiblio.org/?p=5665 http://esr.ibiblio.org/?p=5665 Anyone who thinks all bugs are shallow under any circumstances just hasn't seen many interesting real world bugs with their own eyes. Their experience is limited and their confidence in their software, security and mastery of programming and debugging is pure Dunning-Kruger effect. I'm with Ken on this one.

DonHopkins 9y ago

Actually, that quote was "formulated" by Eric S Raymond (to whom Theo was referring as "the originator of the statement"), and is only deceptively named "Linux's Law" [1] in "honor" of Linus Torvalds, which is ironic because it actually dishonors him by being invalid. The point that Theo was making is that ESR talks and talks and types and types about many eyeballs looking at code, but when it comes down to actually auditing code, he never actually bothers, and neither do most other of his minions who are so quick to parrot his ill-conceived "Linux's Law". Neither "enough eyeballs" nor "the right eyeballs" are a GIVEN, even for open source software. Google "Heartbleed". "Not enough eyeballs" (or "ZERO eyeballs" as he loves to claim) are NOT a GIVEN for proprietary software, because you can license much proprietary source code, and some proprietary source code is available for you to read and audit for free, under licenses like Microsoft's "Shared Source" license. https://en.wikipedia.org/wiki/Shared_source https://en.wikipedia.org/wiki/Shared_source And qualified eyeballs are NOT FREE, and usually very busy being well paid to look at much more interesting things than poorly written buggy code like OpenSSL. I doubt that Eric Raymond has contributed any of the profits from his books or VA Linux stocks to Theo De Raadt or anyone else who actually takes the long time and tedious effort to actually audit code. The one time ESR actually did try to audit some code didn't go so well: The little experience Raymond DOES have auditing code has been a total fiasco and embarrassing failure, since his understanding of the code was incompetent and deeply tainted by his preconceived political ideology and conspiracy theories about global warming, which was his only motivation for auditing the code in the first place. His sole quest was to discredit the scientists who warned about global warming. The code he found and highlighted was actually COMMENTED OUT, and he never addressed the fact that the scientists were vindicated. http://rationalwiki.org/wiki/Eric_S._Raymond http://rationalwiki.org/wiki/Eric_S._Raymond >During the Climategate fiasco, Raymond's ability to read other peoples' source code (or at least his honesty about it) was called into question when he was caught quote-mining analysis software written by the CRU researchers, presenting a commented-out section of source code used for analyzing counterfactuals as evidence of deliberate data manipulation. When confronted with the fact that scientists as a general rule are scrupulously honest, Raymond claimed it was a case of an "error cascade," a concept that makes sense in computer science and other places where all data goes through a single potential failure point, but in areas where outside data and multiple lines of evidence are used for verification, doesn't entirely make sense. (He was curiously silent when all the researchers involved were exonerated of scientific misconduct.) [1] https://en.wikipedia.org/wiki/Linus%27s_Law https://en.wikipedia.org/wiki/Linus%27s_Law Linus's Law is a claim about software development, named in honor of Linus Torvalds and formulated by Eric S. Raymond in his essay and book [redacted]. [...] Validity In Facts and Fallacies about Software Engineering, Robert Glass refers to the law as a "mantra" of the open source movement, but calls it a fallacy due to the lack of supporting evidence and because research has indicated that the rate at which additional bugs are uncovered does not scale linearly with the number of reviewers; rather, there is a small maximum number of useful reviewers, between two and four, and additional reviewers above this number uncover bugs at a much lower rate. While closed-source practitioners also promote stringent, independent code analysis during a software project's development, they focus on in-depth review by a few and not primarily the number of "eyeballs". Although detection of even deliberately inserted flaws can be attributed to Raymond's claim, the persistence of the Heartbleed security bug in a critical piece of code for two years has been considered as a refutation of Raymond's dictum. Larry Seltzer suspects that the availability of source code may cause some developers and researchers to perform less extensive tests than they would with closed source software, making it easier for bugs to remain. In 2015, the Linux Foundation's executive director Jim Zemlin argued that the complexity of modern software has increased to such levels that specific resource allocation is desirable to improve its security. Regarding some of 2014's largest global open source software vulnerabilities, he says, "In these cases, the eyeballs weren't really looking". Large scale experiments or peer-reviewed surveys to test how well the mantra holds in practice have not been performed. https://www.datamation.com/open-source/does-heartbleed-disprove-open-source-is-safer-1.html https://www.datamation.com/open-source/does-heartbleed-dispr... https://www.esecurityplanet.com/open-source-security/why-all-linux-security-bugs-arent-shallow.html https://www.esecurityplanet.com/open-source-security/why-all...

countryslicker 9y ago
That is a lot of words to say that it doesn't help to have access to the source code if qualified people don't perform adequate audits. In this regard, open source software is still better. Google "Heartbleed". When a neglected piece of critical infrastructure code was finally examined, it was shown to have serious flaws. Top talent immediately shifted their focus and began sorely needed maintenance. Nobody asked permission. Nobody called their lawyer. They just got to work to get-er-done. I think Theo was among those to help resolve issues. I am grateful for those eyeballs and those fingers.
- deleted 9y ago
  [deleted]
nickpsecurity 9y ago
It would probably be better to use a survey of vulnerabilities plus Heartbleed instead of just repeating Heartbleed. It can be dismissed as an outlier: "That's just one project everyone was freeloading on." I know, it's a security-critical project that should've gotten many eyeballs. I just prefer to show how pervasive the problem is when debunking this stuff. The other thing I noted on a Heartbleed-related thread was that FOSS never produced high-assurance security despite its labor advantage. Proprietary sector, either industry or CompSci teams, beat them about every time. https://www.schneier.com/blog/archives/2014/04/reverse_heartbl.html#c5581737 https://www.schneier.com/blog/archives/2014/04/reverse_heart... I elaborate more in replies to DB. It comes down to them not caring enough to apply the level of QA necessary. You have to pay people to do that. You also have to find the right people that can do it since even the knowledge of it isn't widespread. They'll know 100 frameworks and such but not the basic activities for assurance past unit/acceptance testing and review. They'll probably have never heard of Ada/SPARK or SPIN when they tell you about Rust. So on and so forth. Raymond's claim is utter bullshit. If you want best security, you're better off buying a 3rd-party-evaluated product from high-assurance proprietary. If you want good security that's cheap, it's a small number of proprietary and FOSS projects whose stakeholders put time in for thorough review and analysis. The rest is shit waiting to happen or (looks at CVE list) happening all the time.
9y ago