9 ms·

This is yet another nail in the coffin of Eric Raymond's irresponsible and fallacious "many eyes make all bugs shallow" theory. https://en.wikipedia.org/wiki/L

This is yet another nail in the coffin of Eric Raymond's irresponsible and fallacious "many eyes make all bugs shallow" theory. https://en.wikipedia.org/wiki/Linus's_Law https://en.wikipedia.org/wiki/Linus's_Law Linus's Law as described by Raymond is a claim about software development, named in honor of Linus Torvalds and formulated by Raymond in his essay and book "The Cathedral and the Bazaar" (1999). The law states that "given enough eyeballs, all bugs are shallow"; or more formally: "Given a large enough beta-tester and co-developer base, almost every problem will be characterized quickly and the fix will be obvious to someone." Presenting the code to multiple developers with the purpose of reaching consensus about its acceptance is a simple form of software reviewing. Researchers and practitioners have repeatedly shown the effectiveness of various types of reviewing process in finding bugs and security issues, and also that reviews may be more efficient than testing. In Facts and Fallacies about Software Engineering, Robert Glass refers to the law as a "mantra" of the open source movement, but calls it a fallacy due to the lack of supporting evidence and because research has indicated that the rate at which additional bugs are uncovered does not scale linearly with the number of reviewers; rather, there is a small maximum number of useful reviewers, between two and four, and additional reviewers above this number uncover bugs at a much lower rate. While closed-source practitioners also promote stringent, independent code analysis during a software project's development, they focus on in-depth review by a few and not primarily the number of "eyeballs".

fluidcruft 13y ago
What evidence is there that there actually were "many eyes" on this code? If anything this underscores the importance of license compatibility in order to maximize the utility of the "many eyes" resource. Honestly, GnuTLS seems to exist purely for ideological shim reasons, and it's not surprising that ideological shim reasons don't motivate "many eyes" to bother show up.
- tptacek 13y ago
  That's a No True Scotsman argument. The problem with the "many eyes bugs shallow" theory is that all eyes aren't created equally, and the valuable eyes aren't distributed uniformly across all software.
  - fluidcruft 13y ago
    So are you saying that GnuTLS had "valuable eyes" and missed this or that they had "many eyes" and missed this? What exactly is your alternate hypothesis?
    - tptacek 13y ago
      I'm sure lots of people have tried to find bugs in GnuTLS.
      - tripzilch 13y ago
        So do you mean that (until now) they just failed to find them, or that (until now) they were only found by people who did not have the incentive to report these bugs? Because in the top post you call the bug "simple and basic". But then in that same top post you imply (in your last line) that GnuTLS gets so very little attention, but now you say that lots of people have tried to find bugs in it. I'm not trying to criticize you here, but trying to figure out what you're trying to say?
  - raldi 13y ago
    It would only be a No-True-Scotsman argument if the original statement of Linus' Law were, "In an open-source project, all bugs are shallow" and someone were now trying to claim that GnuTLS wasn't open-sourcey enough. In reality, the law is about code that has many eyeballs on it, and it's a fair argument to point out that evidence suggests GnuTLS didn't have that many eyeballs on it.
    - tptacek 13y ago
      Can you present some of that evidence about the lack of eyeballs on GnuTLS? Because my point is that the right kind of eyeballs were not on GnuTLS; my point isn't compatible with this supposed "law".
- InclinedPlane 13y ago
  That's part of the fallacy. In the OSS world it's assumed that when code passes through many hands, is depended upon by many projects, and is critical to the functioning of many systems used by many users then naturally the code in use will be seen by many eyes. But this is anything but true. The reality is that code review is far less common than code use, with many defects impacting many users as the logical consequence.
  - einhverfr 13y ago
    > That's part of the fallacy. In the OSS world it's assumed that when code passes through many hands, is depended upon by many projects, and is critical to the functioning of many systems used by many users then naturally the code in use will be seen by many eyes. Right and as you point out that's not true. What gets less attention is why. I know in my own code there are dependencies I know very, very well. Some of them I have helped to author. Some of them I have helped with regarding maintenance programming later. But there are many which I don't. There are a bunch of reasons for this: 1. Code has varying degrees of readability and is easily comprehensible to various extents. Code which reads like a book gets more time than that which I have to figure out its structure first. 2. Code that is well maintained provides less necessity to add someone else. I tend to be more likely to read poorly maintained code than well maintained code, for example because bugs don't get fixed is a good reason to try to fix them myself....
javert 13y ago
I think your analysis is irresponsible and fallacious. ESR is making a completely valid point, and the underlying premise of his theory---that having software open to review can help---is only confirmed by this incident, not rejected. Specifically: If GnuTLS were closed source, this problem would likely never be publicly discovered and disclosed. So overall, ESR's theory is accurate and useful. Note the word "almost" in the theory, which serves as a (completely valid) escape hatch (that you are mistakenly neglecting) for incidents like this---which fit the underlying premise, but are "corner cases" rather than "common cases."
- luckydude 13y ago
  I think the counter to Eric's claim is this: If it is open source, as in a Debian release, an end user's recourse is to fix it themselves. If it is commercial, be it closed source or not, an end user's recourse is to sue the supplier (or something similar). The commercial supplier has a financial incentive to get it right, the open source developer has an intellectual and street cred incentive to get it right. I'm not sure which one actually works better, I know that the popular opinion is the ESR eyeballs claim but it's not clear to me which gets it more correct. Seems like they both fail at times.
  - dasil003 13y ago
    > If it is commercial, be it closed source or not, an end user's recourse is to sue the supplier (or something similar). That is assuming the supplier is still in business, which is probably a dubious proposition for a majority of commercial software that has ever shipped.
  - emiliobumachar 13y ago
    The terms for almost any software redundantly state "no warranty".
  - mjn 13y ago
    > If it is commercial, be it closed source or not, an end user's recourse is to sue the supplier (or something similar). Are there examples of doing this successfully? As far as I can tell, software manufacturers have largely been successful at avoiding traditional product liability for damages caused by malfunctioning software, through a mixture of EULAs and courts buying the "but software is different" argument. Here's an article series on that: http://www.newrepublic.com/article/115402/sad-state-software-liability-law-bad-code-part-4 http://www.newrepublic.com/article/115402/sad-state-software...
cookiecaper 13y ago
Raymond's quip may be somewhat hyperbolic for effect, but I think it's harder to argue with the assertion that "with many eyes, many bugs are shallow". True, there are some things that are esoteric and which only a few very specially trained persons would notice, but in many cases, it's not that way. The fact that a random person will show up on the mailing lists of open-source projects, even minor ones, and have a bugfix patch validates the theory that an open codebase contributes significantly to software quality, and particularly a greater degree of software buglessness. I think that people who argue over Raymond's quip are just being pedants. It's true that not all bugs will be shallow, no matter how many eyes are on software, but I don't think the expression was meant to be taken literally.
- guscost 13y ago
  great_success = sum([i.debug_skill * log(i.audit_time) for i in code_reviewers])
  - SimHacker 13y ago
    And you can't just go: struct code_reviewer *qualified_code_reviewer = (struct code_reviewer *)malloc( sizeof(struct code_reviewer)); assert(qualified_code_reviewer != NULL);
ori_b 13y ago
To quote Theo De Raadt: My favorite part of the "many eyes" argument is how few bugs were found by the two eyes of Eric (the originator of the statement). All the many eyes are apparently attached to a lot of hands that type lots of words about many eyes, and never actually audit code.
- javert 13y ago
  This is just a nasty and unjust way to bring down ESR, whose intellectual contributions are valuable. I don't personally know Theo De Raadt but if he spends his time bringing down other people, he's probably not a very happy person himself.
  - tptacek 13y ago
    The comments you perceive to be nastily and unjustly bringing down ESR are presumably being made by people who do not agree with you about the value of his intellectual contributions. In fact, those comments are directly rebutting his best-known intellectual contribution. So your objection begs the question.
    - bittercynic 13y ago
      Thank you for using "begs the question" correctly. It is far too rare to see that on the web. </tangent>
      - SimHacker 13y ago
        A+++++ Speedy delivery. Well packed. Top notch. As listed. Great value. Good communication. Would do business again.
    - enneff 13y ago
      Indeed. A lot of people just see the criticisms and think that he is unfairly targeted, but you don't have to look far to find the obnoxious and arrogant behavior that earned him the derision he attracts today.
carbocation 13y ago
In other literatures, there is a notion of diffusion of responsibility; I haven't read the works that you mentioned, but is such a risk risk mentioned in them?
7952 13y ago
Cryptography is uniquely difficult to audit precisely because it is encrypted. It requires more effort to tell what an application is doing if all its input/output are completely incompressible. Maybe cryptography should be handled exclusively by the OS. Software would communicate through a protocol that can be easily audited (like normal http with special headers). Firewalls could be used to verify what is going on and normal tools could be used to inspect the data. That kind of checking is essential and doesn't require the talent needed for code review.
wiredfool 13y ago
Unfortunately, it may take 2^1024 eyes on crypto code before the right set looks at the code.
digikata 13y ago
I think all that means is that "many users" != "many eyes". I think ESR's ultimate point is that Open Source, if the security or bugs deeply matter to you, you can independently add your own eyes (or provide resources for the same). It's a very different matter with proprietary software.
- npsimons 13y ago
  Exactly. How can you trust something you don't have source code to? You can't, full stop. Any thinking otherwise is at best childishly naive ignorance. Given the recent revelations of the past few years, it should be glaringly obvious to anyone with half a clue that companies have been compromised, either willingly or otherwise, so that trusting closed source is a bad idea (as ESR, RMS and Bruce Schneier have been saying for decades!). The finding of this bug in GNUTLS is a good thing! Claiming that this bug would have been found at all, much less fixed, in proprietary software is galling, to put it lightly.
  - MichaelGG 13y ago
    You can audit binaries. So definitely not "You can't full stop". Most people are in no position to audit all the source code they use, just like most technical people are in no position to audit binaries they use. (And I'm putting aside the whole issue of backdoors in hardware, compilers, etc.)
falconfunction 13y ago
It's kind of like saying free libraries lead to a utopia without any extra social programs.
arca_vorago 13y ago
ESR is doing an "AMA" on /. today, and I posted asking about what he thought about this, specifically my theory is that everyone is ignoring the amount of code in proportion to the eyes that see it.
brlewis 13y ago
Certain bugs are more likely to be found if you have a large number of users doing diverse things with your software. Security vulnerabilities have never worked like that. This "nail" is not new.
brudgers 13y ago
Raymond wrote Cathedral and Bazaar almost 20 years ago. In the context of the time and the state of open software "all bugs are shallow" was a pretty damn accurate response to its critics. It still isn't a bad description in many cases. There are always edge cases where beta testing and multiple eyes fall short of mathematical possibility [never mind sophisticated attacks]. That such a bug as this matters 17 years after Raymond made his remarks is a testimony to the robustness of the mechanism he described.
- easytiger 13y ago
  It was 15 years ago. 1999. But i am sitting here with a scotch.
  - gonzo 13y ago
    1997, but do enjoy your scotch. http://www.catb.org/esr/writings/cathedral-bazaar/cathedral-bazaar/ http://www.catb.org/esr/writings/cathedral-bazaar/cathedral-...
smithzvk 13y ago
I have never thought of that as the meaning. In fact, how does this wording apply at all to bug discovery when it seems to me discussing bug resolution, which feels like a very different thing? However, I have never read The Cathedral and the Bazaar, so I just took the meaning that made the most sense.
sdegutis 13y ago
THIS
jsoffer 13y ago
It's clear that there aren't "enough (properly skilled, willing and available) eyeballs" for every project out there. Raymond's argument could be correct; still, the irresponsible thing would be to expect, just because of the openness of the code, that someone will ever care for no reason.
easytiger 13y ago
> This is yet another nail in the coffin of Eric Raymond's irresponsible and fallacious "many eyes make all bugs shallow" theory. What a facetious and unsupported assertion you have made. You are seemingly purposefully disingenuous. The simple fact of the matter is, ESR is still right. Perhaps because few people use a piece of s/w these things slip through. The seriousness of this flaw is limited and thus not subject to the fierce post humous questioning you give it. Please define how many eyes saw or used or benefited from this code. I certainly live in the world of this code and don't depend on it. There is a chance you are talking shit.
- mpyne 13y ago
  > The simple fact of the matter is, ESR is still right. ESR is not completely mistaken here, but the problem is that he's not exhaustively right with "Linus's Law". It's more a description of why beta testing with access to source code is good, than a description of why open source is inherently good (and before you flip out, I've contributed to open source longer than I've done my day job). Some types of code flaws will simply never show up to the kind of beta testing that introducing software to a wide population provides. E.g. proving that a X.509 certificate which is faulty in a certain way is actually caught by a software library; few people run into that in practice. What these types of bugs require is code auditing (after the fact) or before-the-fact code review that prevents their entry into the source in the first place. But neither auditing nor pre-commit review are inherent to open-source, and in fact it could be argued that closed-source software companies are better able to ensure sure things happen. The saving grace for open source is that these companies optimize for market success and not code quality (except to the bare extent needed for market success). Additionally you can pay to audit open source code much more easily than you can closed source (e.g. Google's audit teams that do exactly this). But "bugs are shallow in the presence of sufficient eyeballs" is not unique to open source, either in theory or in practice.
- SimHacker 13y ago
  What supports my assertion are the two recent gigantic but shallow security holes that many eyes (except for the NSA's) didn't see for many years.
zeckalpha 13y ago
Not sure about this, because someone found the bug.
Spooky23 13y ago
Im not an ESR fan, but let's be real here, the bug got found. How many bugs of similar impact are hiding in crufty old Windows code that nobody is looking at. Unless you're talking about an IBM mainframe, validated at EAL 5, there are security bugs all over the place. With open source, you don't get a platoon of elves scanning the code, but you have a much better chance of someone happening across a defect or identifying the responsible party.
- 72deluxe 13y ago
  But how would you do know that those bugs in closed source systems haven't been found? The bug reporting systems are closed too, and the hotfixes and patches that get issued don't link to a list of bug reports. So you are none the wiser. For all we know, bugs are found in closed sourced systems all of the time and are fixed frequently; the only difference is that they're not publicised.
  - Spooky23 13y ago
    I agree. What I'm saying is, without the benefit of open source, you're relying on third-party certification to evaluate the security of products.
- SimHacker 13y ago
  No, actually, there IS a platoon of elves with millions of eyes, carefully scanning all of the open source the code for bugs, meticulously going over it line by line, building rigorous test harnesses, feeding it every input imaginable to probe its weaknesses, and writing up reports describing every quirk flaw they detect in it. And then not fixing it. And your tax dollars are paying for all of that work. And you can be sure those elves have known about those bugs for years. And those bugs have caused many unfortunate consequences. So it's just not wise to go around giving people a false sense of security in order to promote your brand.
gonzo 13y ago
It's worth reading what Eric actually wrote: Linus was directly aiming to maximize the number of person-hours thrown at debugging and development, even at the possible cost of instability in the code and user-base burnout if any serious bug proved intractable. Linus was behaving as though he believed something like this: 8. Given a large enough beta-tester and co-developer base, almost every problem will be characterized quickly and the fix obvious to someone. Or, less formally, ``Given enough eyeballs, all bugs are shallow.'' I dub this: ``Linus's Law''. My original formulation was that every problem ``will be transparent to somebody''. Linus demurred that the person who understands and fixes the problem is not necessarily or even usually the person who first characterizes it. ``Somebody finds the problem,'' he says, ``and somebody else understands it. And I'll go on record as saying that finding it is the bigger challenge.'' That correction is important; we'll see how in the next section, when we examine the practice of debugging in more detail. But the key point is that both parts of the process (finding and fixing) tend to happen rapidly. In Linus's Law, I think, lies the core difference underlying the cathedral-builder and bazaar styles. In the cathedral-builder view of programming, bugs and development problems are tricky, insidious, deep phenomena. It takes months of scrutiny by a dedicated few to develop confidence that you've winkled them all out. Thus the long release intervals, and the inevitable disappointment when long-awaited releases are not perfect. In the bazaar view, on the other hand, you assume that bugs are generally shallow phenomena—or, at least, that they turn shallow pretty quickly when exposed to a thousand eager co-developers pounding on every single new release. Accordingly you release often in order to get more corrections, and as a beneficial side effect you have less to lose if an occasional botch gets out the door. And that's it. That's enough. If ``Linus's Law'' is false, then any system as complex as the Linux kernel, being hacked over by as many hands as the that kernel was, should at some point have collapsed under the weight of unforseen bad interactions and undiscovered ``deep'' bugs. If it's true, on the other hand, it is sufficient to explain Linux's relative lack of bugginess and its continuous uptimes spanning months or even years. One can only conclude that either ESR is wrong (it wouldn't be the first time), or that ESR's beloved open source "bazaar" has become a cathedral.
- astrange 13y ago
  I would be surprised to see a security library developed with the bazaar model. It'd be difficult to assemble enough interest and capable reviewers. In this case it looks like the GnuTLS bug was introduced and fixed by the same person, but I didn't go look to see how many others there were.
cwp 13y ago
Straw man. The phrase "all bugs are shallow" doesn't mean all bugs will be found. It means that, once a bug is found, the correct fix will be obvious to someone.
hyc_symas 13y ago
This not a case of GnuTLS not being audited, or audited by the right/wrong eyes. Audits were done; the Debian maintainers chose to ignore the conclusions.

fluidcruft 13y ago

What evidence is there that there actually were "many eyes" on this code? If anything this underscores the importance of license compatibility in order to maximize the utility of the "many eyes" resource. Honestly, GnuTLS seems to exist purely for ideological shim reasons, and it's not surprising that ideological shim reasons don't motivate "many eyes" to bother show up.

tptacek 13y ago
That's a No True Scotsman argument. The problem with the "many eyes bugs shallow" theory is that all eyes aren't created equally, and the valuable eyes aren't distributed uniformly across all software.
- fluidcruft 13y ago
  So are you saying that GnuTLS had "valuable eyes" and missed this or that they had "many eyes" and missed this? What exactly is your alternate hypothesis?
  - tptacek 13y ago
    I'm sure lots of people have tried to find bugs in GnuTLS.
    - tripzilch 13y ago
      So do you mean that (until now) they just failed to find them, or that (until now) they were only found by people who did not have the incentive to report these bugs? Because in the top post you call the bug "simple and basic". But then in that same top post you imply (in your last line) that GnuTLS gets so very little attention, but now you say that lots of people have tried to find bugs in it. I'm not trying to criticize you here, but trying to figure out what you're trying to say?
- raldi 13y ago
  It would only be a No-True-Scotsman argument if the original statement of Linus' Law were, "In an open-source project, all bugs are shallow" and someone were now trying to claim that GnuTLS wasn't open-sourcey enough. In reality, the law is about code that has many eyeballs on it, and it's a fair argument to point out that evidence suggests GnuTLS didn't have that many eyeballs on it.
  - tptacek 13y ago
    Can you present some of that evidence about the lack of eyeballs on GnuTLS? Because my point is that the right kind of eyeballs were not on GnuTLS; my point isn't compatible with this supposed "law".
InclinedPlane 13y ago
That's part of the fallacy. In the OSS world it's assumed that when code passes through many hands, is depended upon by many projects, and is critical to the functioning of many systems used by many users then naturally the code in use will be seen by many eyes. But this is anything but true. The reality is that code review is far less common than code use, with many defects impacting many users as the logical consequence.
- einhverfr 13y ago
  > That's part of the fallacy. In the OSS world it's assumed that when code passes through many hands, is depended upon by many projects, and is critical to the functioning of many systems used by many users then naturally the code in use will be seen by many eyes. Right and as you point out that's not true. What gets less attention is why. I know in my own code there are dependencies I know very, very well. Some of them I have helped to author. Some of them I have helped with regarding maintenance programming later. But there are many which I don't. There are a bunch of reasons for this: 1. Code has varying degrees of readability and is easily comprehensible to various extents. Code which reads like a book gets more time than that which I have to figure out its structure first. 2. Code that is well maintained provides less necessity to add someone else. I tend to be more likely to read poorly maintained code than well maintained code, for example because bugs don't get fixed is a good reason to try to fix them myself....

javert 13y ago

I think your analysis is irresponsible and fallacious. ESR is making a completely valid point, and the underlying premise of his theory---that having software open to review can help---is only confirmed by this incident, not rejected. Specifically: If GnuTLS were closed source, this problem would likely never be publicly discovered and disclosed. So overall, ESR's theory is accurate and useful. Note the word "almost" in the theory, which serves as a (completely valid) escape hatch (that you are mistakenly neglecting) for incidents like this---which fit the underlying premise, but are "corner cases" rather than "common cases."

luckydude 13y ago
I think the counter to Eric's claim is this: If it is open source, as in a Debian release, an end user's recourse is to fix it themselves. If it is commercial, be it closed source or not, an end user's recourse is to sue the supplier (or something similar). The commercial supplier has a financial incentive to get it right, the open source developer has an intellectual and street cred incentive to get it right. I'm not sure which one actually works better, I know that the popular opinion is the ESR eyeballs claim but it's not clear to me which gets it more correct. Seems like they both fail at times.
- dasil003 13y ago
  > If it is commercial, be it closed source or not, an end user's recourse is to sue the supplier (or something similar). That is assuming the supplier is still in business, which is probably a dubious proposition for a majority of commercial software that has ever shipped.
- emiliobumachar 13y ago
  The terms for almost any software redundantly state "no warranty".
- mjn 13y ago
  > If it is commercial, be it closed source or not, an end user's recourse is to sue the supplier (or something similar). Are there examples of doing this successfully? As far as I can tell, software manufacturers have largely been successful at avoiding traditional product liability for damages caused by malfunctioning software, through a mixture of EULAs and courts buying the "but software is different" argument. Here's an article series on that: http://www.newrepublic.com/article/115402/sad-state-software-liability-law-bad-code-part-4 http://www.newrepublic.com/article/115402/sad-state-software...

cookiecaper 13y ago

Raymond's quip may be somewhat hyperbolic for effect, but I think it's harder to argue with the assertion that "with many eyes, many bugs are shallow". True, there are some things that are esoteric and which only a few very specially trained persons would notice, but in many cases, it's not that way. The fact that a random person will show up on the mailing lists of open-source projects, even minor ones, and have a bugfix patch validates the theory that an open codebase contributes significantly to software quality, and particularly a greater degree of software buglessness. I think that people who argue over Raymond's quip are just being pedants. It's true that not all bugs will be shallow, no matter how many eyes are on software, but I don't think the expression was meant to be taken literally.

guscost 13y ago
great_success = sum([i.debug_skill * log(i.audit_time) for i in code_reviewers])
- SimHacker 13y ago
  And you can't just go: struct code_reviewer *qualified_code_reviewer = (struct code_reviewer *)malloc( sizeof(struct code_reviewer)); assert(qualified_code_reviewer != NULL);

ori_b 13y ago

To quote Theo De Raadt: My favorite part of the "many eyes" argument is how few bugs were found by the two eyes of Eric (the originator of the statement). All the many eyes are apparently attached to a lot of hands that type lots of words about many eyes, and never actually audit code.

javert 13y ago
This is just a nasty and unjust way to bring down ESR, whose intellectual contributions are valuable. I don't personally know Theo De Raadt but if he spends his time bringing down other people, he's probably not a very happy person himself.
- tptacek 13y ago
  The comments you perceive to be nastily and unjustly bringing down ESR are presumably being made by people who do not agree with you about the value of his intellectual contributions. In fact, those comments are directly rebutting his best-known intellectual contribution. So your objection begs the question.
  - bittercynic 13y ago
    Thank you for using "begs the question" correctly. It is far too rare to see that on the web. </tangent>
    - SimHacker 13y ago
      A+++++ Speedy delivery. Well packed. Top notch. As listed. Great value. Good communication. Would do business again.
  - enneff 13y ago
    Indeed. A lot of people just see the criticisms and think that he is unfairly targeted, but you don't have to look far to find the obnoxious and arrogant behavior that earned him the derision he attracts today.

carbocation 13y ago

In other literatures, there is a notion of diffusion of responsibility; I haven't read the works that you mentioned, but is such a risk risk mentioned in them?

7952 13y ago

Cryptography is uniquely difficult to audit precisely because it is encrypted. It requires more effort to tell what an application is doing if all its input/output are completely incompressible. Maybe cryptography should be handled exclusively by the OS. Software would communicate through a protocol that can be easily audited (like normal http with special headers). Firewalls could be used to verify what is going on and normal tools could be used to inspect the data. That kind of checking is essential and doesn't require the talent needed for code review.

wiredfool 13y ago

Unfortunately, it may take 2^1024 eyes on crypto code before the right set looks at the code.

digikata 13y ago

I think all that means is that "many users" != "many eyes". I think ESR's ultimate point is that Open Source, if the security or bugs deeply matter to you, you can independently add your own eyes (or provide resources for the same). It's a very different matter with proprietary software.

npsimons 13y ago
Exactly. How can you trust something you don't have source code to? You can't, full stop. Any thinking otherwise is at best childishly naive ignorance. Given the recent revelations of the past few years, it should be glaringly obvious to anyone with half a clue that companies have been compromised, either willingly or otherwise, so that trusting closed source is a bad idea (as ESR, RMS and Bruce Schneier have been saying for decades!). The finding of this bug in GNUTLS is a good thing! Claiming that this bug would have been found at all, much less fixed, in proprietary software is galling, to put it lightly.
- MichaelGG 13y ago
  You can audit binaries. So definitely not "You can't full stop". Most people are in no position to audit all the source code they use, just like most technical people are in no position to audit binaries they use. (And I'm putting aside the whole issue of backdoors in hardware, compilers, etc.)

falconfunction 13y ago

It's kind of like saying free libraries lead to a utopia without any extra social programs.

arca_vorago 13y ago

ESR is doing an "AMA" on /. today, and I posted asking about what he thought about this, specifically my theory is that everyone is ignoring the amount of code in proportion to the eyes that see it.

brlewis 13y ago

Certain bugs are more likely to be found if you have a large number of users doing diverse things with your software. Security vulnerabilities have never worked like that. This "nail" is not new.

brudgers 13y ago

Raymond wrote Cathedral and Bazaar almost 20 years ago. In the context of the time and the state of open software "all bugs are shallow" was a pretty damn accurate response to its critics. It still isn't a bad description in many cases. There are always edge cases where beta testing and multiple eyes fall short of mathematical possibility [never mind sophisticated attacks]. That such a bug as this matters 17 years after Raymond made his remarks is a testimony to the robustness of the mechanism he described.

easytiger 13y ago
It was 15 years ago. 1999. But i am sitting here with a scotch.
- gonzo 13y ago
  1997, but do enjoy your scotch. http://www.catb.org/esr/writings/cathedral-bazaar/cathedral-bazaar/ http://www.catb.org/esr/writings/cathedral-bazaar/cathedral-...

smithzvk 13y ago

I have never thought of that as the meaning. In fact, how does this wording apply at all to bug discovery when it seems to me discussing bug resolution, which feels like a very different thing? However, I have never read The Cathedral and the Bazaar, so I just took the meaning that made the most sense.

sdegutis 13y ago

THIS

jsoffer 13y ago

It's clear that there aren't "enough (properly skilled, willing and available) eyeballs" for every project out there. Raymond's argument could be correct; still, the irresponsible thing would be to expect, just because of the openness of the code, that someone will ever care for no reason.

easytiger 13y ago

> This is yet another nail in the coffin of Eric Raymond's irresponsible and fallacious "many eyes make all bugs shallow" theory. What a facetious and unsupported assertion you have made. You are seemingly purposefully disingenuous. The simple fact of the matter is, ESR is still right. Perhaps because few people use a piece of s/w these things slip through. The seriousness of this flaw is limited and thus not subject to the fierce post humous questioning you give it. Please define how many eyes saw or used or benefited from this code. I certainly live in the world of this code and don't depend on it. There is a chance you are talking shit.

mpyne 13y ago
> The simple fact of the matter is, ESR is still right. ESR is not completely mistaken here, but the problem is that he's not exhaustively right with "Linus's Law". It's more a description of why beta testing with access to source code is good, than a description of why open source is inherently good (and before you flip out, I've contributed to open source longer than I've done my day job). Some types of code flaws will simply never show up to the kind of beta testing that introducing software to a wide population provides. E.g. proving that a X.509 certificate which is faulty in a certain way is actually caught by a software library; few people run into that in practice. What these types of bugs require is code auditing (after the fact) or before-the-fact code review that prevents their entry into the source in the first place. But neither auditing nor pre-commit review are inherent to open-source, and in fact it could be argued that closed-source software companies are better able to ensure sure things happen. The saving grace for open source is that these companies optimize for market success and not code quality (except to the bare extent needed for market success). Additionally you can pay to audit open source code much more easily than you can closed source (e.g. Google's audit teams that do exactly this). But "bugs are shallow in the presence of sufficient eyeballs" is not unique to open source, either in theory or in practice.
SimHacker 13y ago
What supports my assertion are the two recent gigantic but shallow security holes that many eyes (except for the NSA's) didn't see for many years.

zeckalpha 13y ago

Not sure about this, because someone found the bug.

Spooky23 13y ago

Im not an ESR fan, but let's be real here, the bug got found. How many bugs of similar impact are hiding in crufty old Windows code that nobody is looking at. Unless you're talking about an IBM mainframe, validated at EAL 5, there are security bugs all over the place. With open source, you don't get a platoon of elves scanning the code, but you have a much better chance of someone happening across a defect or identifying the responsible party.

72deluxe 13y ago
But how would you do know that those bugs in closed source systems haven't been found? The bug reporting systems are closed too, and the hotfixes and patches that get issued don't link to a list of bug reports. So you are none the wiser. For all we know, bugs are found in closed sourced systems all of the time and are fixed frequently; the only difference is that they're not publicised.
- Spooky23 13y ago
  I agree. What I'm saying is, without the benefit of open source, you're relying on third-party certification to evaluate the security of products.
SimHacker 13y ago
No, actually, there IS a platoon of elves with millions of eyes, carefully scanning all of the open source the code for bugs, meticulously going over it line by line, building rigorous test harnesses, feeding it every input imaginable to probe its weaknesses, and writing up reports describing every quirk flaw they detect in it. And then not fixing it. And your tax dollars are paying for all of that work. And you can be sure those elves have known about those bugs for years. And those bugs have caused many unfortunate consequences. So it's just not wise to go around giving people a false sense of security in order to promote your brand.

gonzo 13y ago

It's worth reading what Eric actually wrote: Linus was directly aiming to maximize the number of person-hours thrown at debugging and development, even at the possible cost of instability in the code and user-base burnout if any serious bug proved intractable. Linus was behaving as though he believed something like this: 8. Given a large enough beta-tester and co-developer base, almost every problem will be characterized quickly and the fix obvious to someone. Or, less formally, ``Given enough eyeballs, all bugs are shallow.'' I dub this: ``Linus's Law''. My original formulation was that every problem ``will be transparent to somebody''. Linus demurred that the person who understands and fixes the problem is not necessarily or even usually the person who first characterizes it. ``Somebody finds the problem,'' he says, ``and somebody else understands it. And I'll go on record as saying that finding it is the bigger challenge.'' That correction is important; we'll see how in the next section, when we examine the practice of debugging in more detail. But the key point is that both parts of the process (finding and fixing) tend to happen rapidly. In Linus's Law, I think, lies the core difference underlying the cathedral-builder and bazaar styles. In the cathedral-builder view of programming, bugs and development problems are tricky, insidious, deep phenomena. It takes months of scrutiny by a dedicated few to develop confidence that you've winkled them all out. Thus the long release intervals, and the inevitable disappointment when long-awaited releases are not perfect. In the bazaar view, on the other hand, you assume that bugs are generally shallow phenomena—or, at least, that they turn shallow pretty quickly when exposed to a thousand eager co-developers pounding on every single new release. Accordingly you release often in order to get more corrections, and as a beneficial side effect you have less to lose if an occasional botch gets out the door. And that's it. That's enough. If ``Linus's Law'' is false, then any system as complex as the Linux kernel, being hacked over by as many hands as the that kernel was, should at some point have collapsed under the weight of unforseen bad interactions and undiscovered ``deep'' bugs. If it's true, on the other hand, it is sufficient to explain Linux's relative lack of bugginess and its continuous uptimes spanning months or even years. One can only conclude that either ESR is wrong (it wouldn't be the first time), or that ESR's beloved open source "bazaar" has become a cathedral.

astrange 13y ago
I would be surprised to see a security library developed with the bazaar model. It'd be difficult to assemble enough interest and capable reviewers. In this case it looks like the GnuTLS bug was introduced and fixed by the same person, but I didn't go look to see how many others there were.

cwp 13y ago

Straw man. The phrase "all bugs are shallow" doesn't mean all bugs will be found. It means that, once a bug is found, the correct fix will be obvious to someone.

hyc_symas 13y ago

This not a case of GnuTLS not being audited, or audited by the right/wrong eyes. Audits were done; the Debian maintainers chose to ignore the conclusions.