3 ms·

I would say that the law about enough eyeballs and shallow bugs, call it what you will, works remarkably well. It doesn't imply that bugs get fixed, only that t

I would say that the law about enough eyeballs and shallow bugs, call it what you will, works remarkably well. It doesn't imply that bugs get fixed, only that they are shallow. Notice the vagueness of this statement. When some eyeballs find excuses to not fix the bugs or find a more entertaining problem to bikeshed, other enterprising eyeballs are busy creating 0day exploits for bugs they shallow.

user-the-name 5y ago
Does it work remarkably well? Are bugs significantly less common in open source software? I do not really see any indication of this in practice.
- WesolyKubeczek 5y ago
  It works well. It doesn't imply the bugs will be fixed. Sometimes it implies an exploit will be made. Sometimes the list of open bugs grows embarrassingly big, and that's it. Shallow. But, nevertheless, transparent it is.
DonHopkins 5y ago
>It doesn't imply that bugs get fixed, only that they are shallow. No, actually, it intentionally does imply that the bugs will get fixed: that's the whole point. Where did you get that idea -- have you even bothered to cast your own eyes on the original source of the quote, or any of the discussion around Heartbleed and other bugs, or is that just your "feeling"? If you go back and read the original source of the quote in context, you will see that it explicitly states that the key point is that bugs will get rapidly FIXED, in no uncertain terms: http://www.catb.org/~esr/writings/cathedral-bazaar/cathedral-bazaar/ar01s04.html http://www.catb.org/~esr/writings/cathedral-bazaar/cathedral... >[...] Linus was directly aiming to maximize the number of person-hours thrown at debugging and development, even at the possible cost of instability in the code and user-base burnout if any serious bug proved intractable. Linus was behaving as though he believed something like this: >8. Given a large enough beta-tester and co-developer base, almost every problem will be characterized quickly and the fix obvious to someone. [...] >But the key point is that both parts of the process (finding and fixing) tend to happen rapidly. [...] >In practice, the theoretical loss of efficiency due to duplication of work by debuggers almost never seems to be an issue in the Linux world. One effect of a ``release early and often'' policy is to minimize such duplication by propagating fed-back fixes quickly [JH]. [...] Notwithstanding the fact that ESR himself incompetently and dishonestly claimed to have found bugs in climate analysis software and didn't bother to fix them, because his actual point was to falsely discredit the scientists (whose exoneration he still has not acknowledged), not to altruistically advance science, but to mendaciously deny science. Is that what you mean by "bikeshedding"? Did the finding and fixing of the Heartbleed bug happen rapidly because it was open source? No, definitely not! Open source is not magical pixie dust. https://www.datamation.com/open-source/does-heartbleed-disprove-open-source-is-safer/ https://www.datamation.com/open-source/does-heartbleed-dispr... >Does Heartbleed Disprove ‘Open Source is Safer’? >[...] Or, as Eric Raymond famously said, “given enough eyeballs, all bugs are shallow.” Yet, somehow, Heartbleed appears to have existed for over two years before being discovered. It may even have been used by American security agencies in their surveillance of the public. [...] >Implicit in the description is not only the idea that peer review can substitute for software testing, but also that no special effort is needed to detect bugs. Simply by going about their business as developers, FOSS project members are likely to notice bugs so that they can be repaired. >This claim has not gone unchallenged. It is a statement of belief, not the conclusion of a scientific study, a rationalization of the fact that peer review in FOSS has always been easier than software testing. Moreover, in Facts and Fallacies about Software Engineering, Robert L. Glass claims that no correlation exists between the number of bugs reported and the number of reviewers. >Yet despite the claim’s weaknesses, it remains one of FOSS’s major assertions of superiority. Heartbleed seems an exception that at least challenges the widely believed rule, or maybe even overturns it completely. [...] >A more useful analysis has been offered by Theo de Raadt, the founder of OpenBSD and OpenSSH. De Raadt notes that malloc, a memory allocation library, was long ago patched to prevent Heartbleed-type exploitations. However, at the same time, OpenSSL added “a wrapper around malloc & free so that the library will cache memory on its own, and not free it to the protective malloc” — all in the name of improving performance on some systems. >In other words, the potential for a bug was detected and patched, but was by-passed by an engineering decision that favored efficiency over security. Perhaps, too, the wrapper was never examined closely because it was assumed to be trivial and to add nothing new. It had become an established part of the code that nobody was likely to modify. But, whatever the case, de Raadt concludes scathingly, “OpenSSL is not developed by a responsible team.” >Assuming that de Raadt is right, then one take-away for FOSS is that all the eyes in the world cannot be counted on to catch basic design problems. >Taken together, Segglemann’s and de Raadt’s comments also suggest that assuming no special effort is needed to discover bugs is a mistake. Perhaps more attention needs to be paid to formal reviews and software testing than FOSS traditionally has managed. The fact that FOSS development often involves remote cooperation does not mean that log-in test or in-person testing sessions could not be added to many project’s development cycle. >What Heartbleed proves is that FOSS needs at to examine the unexamined assumption it has held for years. Greg DeKoenigsberg, a vice president at Eucalyptus Systems, summed up the situation neatly on Facebook: “we don’t put enough eyes in the right places, because we assume [bug-detection] will just happen because of open source pixie dust — and now we’re paying the price for it.” [...] https://www.esecurityplanet.com/applications/why-all-linux-security-bugs-arent-shallow/ https://www.esecurityplanet.com/applications/why-all-linux-s... >Why All Linux (Security) Bugs Aren’t Shallow >Zemlin quoted the oft-repeated Linus’ law, which states that given enough eyes all bugs are shallow. That “law” essentially promises that many eyes provide a measure of quality and control and security to open source code. So if Linus’ law is true, Zemlin asked, why are damaging security issues being found now in open source code? >“In these cases the eyeballs weren’t really looking,” Zemlin said. https://www.zdnet.com/article/did-open-source-matter-for-heartbleed/ https://www.zdnet.com/article/did-open-source-matter-for-hea... >Did open source matter for Heartbleed? >Open source does not provide a meaningful inherent security benefit for OpenSSL and it may actually discourage some important testing techniques. Also, panhandling is not a good business model for important software like OpenSSL. >Let's stipulate that OpenSSL has a good reputation, perhaps even that it deserves that reputation (although this is not the first highly-critical vulnerability in OpenSSL). I would argue that the reputation is based largely on wishful thinking and open source mythology. >Before the word "mythology" gets me into too much trouble, I ought to say, as Nixon might have put it, "we're all open source activists now." For some purposes, open source is a good thing, or a necessary thing, or both. I agree, at least in part, with those who say that cryptography code needs to be open source, because it requires a high level of trust. >Ultimately, the logic of that last statement presumes that there are people analyzing the open source code of OpenSSL in order to confirm that it is deserving of trust. This is the "many eyeballs" effect described in The Cathedral and the Bazaar, by Eric Raymond, one of the early gospels in the theology of open source. The idea is that if enough people have access to source code then someone will notice the bugs.
- WesolyKubeczek 5y ago
  It's one of those things where what is said works in the way it has been worded, not in the way its author thought it worked. I mean, at its face value, the bugs have a better chance of being fixed because more people can see them. And the open source license, technically, empowers every user. Yes, even today, I can fix whatever I please and disregard whatever everyone says about "being a part of the community" and "decision process". And I can distribute the fixes on my own. In this sense, the "law" is working, it's just as obvious as the fact that it's way easier to run a company in a country with fewer regulations and simpler tax law than in a country with more regulations and more complex tax law. However, there are also other confounding factors which may negate this maxim. For example, the bystander effect is pretty big. Sure there's someone else fixing this. Or there's the fatigue of pull requests being enqueued for years. Or there is, on one hand, a stigma that you have to have a galaxy brain to work on X, while at the same time people making X may make mistakes but no one feels it's right to question them, because they are "galaxy brains" by definition. There's also this factor of eyeballs being tired, not really interested, not looking really carefully. And yet, once known, things do get fixed rather rapidly. In open source, I haven't seen yet an equivalent of the AWS status page. So, yeah, it doesn't guarantee you things. But when we're talking about all else being equivalent except for the license, an open-source X will be better, on average, than a closed-source X. Most areas of work benefit from transparency.
Seirdy 5y ago
I cited multiple pieces of FLOSS that employ and benefit from fuzzing, making an example out of a Linux bug that allowed sandbox escapes that was found through fuzzing rather than source analysis. Humans suck at finding subtle issues purely by reading source code. Running software is a more effective means to detect bugs. Once a bug has been detected, source code is really helpful to isolate and patch it.

user-the-name 5y ago

Does it work remarkably well? Are bugs significantly less common in open source software? I do not really see any indication of this in practice.

WesolyKubeczek 5y ago
It works well. It doesn't imply the bugs will be fixed. Sometimes it implies an exploit will be made. Sometimes the list of open bugs grows embarrassingly big, and that's it. Shallow. But, nevertheless, transparent it is.

DonHopkins 5y ago

>It doesn't imply that bugs get fixed, only that they are shallow. No, actually, it intentionally does imply that the bugs will get fixed: that's the whole point. Where did you get that idea -- have you even bothered to cast your own eyes on the original source of the quote, or any of the discussion around Heartbleed and other bugs, or is that just your "feeling"? If you go back and read the original source of the quote in context, you will see that it explicitly states that the key point is that bugs will get rapidly FIXED, in no uncertain terms: http://www.catb.org/~esr/writings/cathedral-bazaar/cathedral-bazaar/ar01s04.html http://www.catb.org/~esr/writings/cathedral-bazaar/cathedral... >[...] Linus was directly aiming to maximize the number of person-hours thrown at debugging and development, even at the possible cost of instability in the code and user-base burnout if any serious bug proved intractable. Linus was behaving as though he believed something like this: >8. Given a large enough beta-tester and co-developer base, almost every problem will be characterized quickly and the fix obvious to someone. [...] >But the key point is that both parts of the process (finding and fixing) tend to happen rapidly. [...] >In practice, the theoretical loss of efficiency due to duplication of work by debuggers almost never seems to be an issue in the Linux world. One effect of a ``release early and often'' policy is to minimize such duplication by propagating fed-back fixes quickly [JH]. [...] Notwithstanding the fact that ESR himself incompetently and dishonestly claimed to have found bugs in climate analysis software and didn't bother to fix them, because his actual point was to falsely discredit the scientists (whose exoneration he still has not acknowledged), not to altruistically advance science, but to mendaciously deny science. Is that what you mean by "bikeshedding"? Did the finding and fixing of the Heartbleed bug happen rapidly because it was open source? No, definitely not! Open source is not magical pixie dust. https://www.datamation.com/open-source/does-heartbleed-disprove-open-source-is-safer/ https://www.datamation.com/open-source/does-heartbleed-dispr... >Does Heartbleed Disprove ‘Open Source is Safer’? >[...] Or, as Eric Raymond famously said, “given enough eyeballs, all bugs are shallow.” Yet, somehow, Heartbleed appears to have existed for over two years before being discovered. It may even have been used by American security agencies in their surveillance of the public. [...] >Implicit in the description is not only the idea that peer review can substitute for software testing, but also that no special effort is needed to detect bugs. Simply by going about their business as developers, FOSS project members are likely to notice bugs so that they can be repaired. >This claim has not gone unchallenged. It is a statement of belief, not the conclusion of a scientific study, a rationalization of the fact that peer review in FOSS has always been easier than software testing. Moreover, in Facts and Fallacies about Software Engineering, Robert L. Glass claims that no correlation exists between the number of bugs reported and the number of reviewers. >Yet despite the claim’s weaknesses, it remains one of FOSS’s major assertions of superiority. Heartbleed seems an exception that at least challenges the widely believed rule, or maybe even overturns it completely. [...] >A more useful analysis has been offered by Theo de Raadt, the founder of OpenBSD and OpenSSH. De Raadt notes that malloc, a memory allocation library, was long ago patched to prevent Heartbleed-type exploitations. However, at the same time, OpenSSL added “a wrapper around malloc & free so that the library will cache memory on its own, and not free it to the protective malloc” — all in the name of improving performance on some systems. >In other words, the potential for a bug was detected and patched, but was by-passed by an engineering decision that favored efficiency over security. Perhaps, too, the wrapper was never examined closely because it was assumed to be trivial and to add nothing new. It had become an established part of the code that nobody was likely to modify. But, whatever the case, de Raadt concludes scathingly, “OpenSSL is not developed by a responsible team.” >Assuming that de Raadt is right, then one take-away for FOSS is that all the eyes in the world cannot be counted on to catch basic design problems. >Taken together, Segglemann’s and de Raadt’s comments also suggest that assuming no special effort is needed to discover bugs is a mistake. Perhaps more attention needs to be paid to formal reviews and software testing than FOSS traditionally has managed. The fact that FOSS development often involves remote cooperation does not mean that log-in test or in-person testing sessions could not be added to many project’s development cycle. >What Heartbleed proves is that FOSS needs at to examine the unexamined assumption it has held for years. Greg DeKoenigsberg, a vice president at Eucalyptus Systems, summed up the situation neatly on Facebook: “we don’t put enough eyes in the right places, because we assume [bug-detection] will just happen because of open source pixie dust — and now we’re paying the price for it.” [...] https://www.esecurityplanet.com/applications/why-all-linux-security-bugs-arent-shallow/ https://www.esecurityplanet.com/applications/why-all-linux-s... >Why All Linux (Security) Bugs Aren’t Shallow >Zemlin quoted the oft-repeated Linus’ law, which states that given enough eyes all bugs are shallow. That “law” essentially promises that many eyes provide a measure of quality and control and security to open source code. So if Linus’ law is true, Zemlin asked, why are damaging security issues being found now in open source code? >“In these cases the eyeballs weren’t really looking,” Zemlin said. https://www.zdnet.com/article/did-open-source-matter-for-heartbleed/ https://www.zdnet.com/article/did-open-source-matter-for-hea... >Did open source matter for Heartbleed? >Open source does not provide a meaningful inherent security benefit for OpenSSL and it may actually discourage some important testing techniques. Also, panhandling is not a good business model for important software like OpenSSL. >Let's stipulate that OpenSSL has a good reputation, perhaps even that it deserves that reputation (although this is not the first highly-critical vulnerability in OpenSSL). I would argue that the reputation is based largely on wishful thinking and open source mythology. >Before the word "mythology" gets me into too much trouble, I ought to say, as Nixon might have put it, "we're all open source activists now." For some purposes, open source is a good thing, or a necessary thing, or both. I agree, at least in part, with those who say that cryptography code needs to be open source, because it requires a high level of trust. >Ultimately, the logic of that last statement presumes that there are people analyzing the open source code of OpenSSL in order to confirm that it is deserving of trust. This is the "many eyeballs" effect described in The Cathedral and the Bazaar, by Eric Raymond, one of the early gospels in the theology of open source. The idea is that if enough people have access to source code then someone will notice the bugs.

WesolyKubeczek 5y ago
It's one of those things where what is said works in the way it has been worded, not in the way its author thought it worked. I mean, at its face value, the bugs have a better chance of being fixed because more people can see them. And the open source license, technically, empowers every user. Yes, even today, I can fix whatever I please and disregard whatever everyone says about "being a part of the community" and "decision process". And I can distribute the fixes on my own. In this sense, the "law" is working, it's just as obvious as the fact that it's way easier to run a company in a country with fewer regulations and simpler tax law than in a country with more regulations and more complex tax law. However, there are also other confounding factors which may negate this maxim. For example, the bystander effect is pretty big. Sure there's someone else fixing this. Or there's the fatigue of pull requests being enqueued for years. Or there is, on one hand, a stigma that you have to have a galaxy brain to work on X, while at the same time people making X may make mistakes but no one feels it's right to question them, because they are "galaxy brains" by definition. There's also this factor of eyeballs being tired, not really interested, not looking really carefully. And yet, once known, things do get fixed rather rapidly. In open source, I haven't seen yet an equivalent of the AWS status page. So, yeah, it doesn't guarantee you things. But when we're talking about all else being equivalent except for the license, an open-source X will be better, on average, than a closed-source X. Most areas of work benefit from transparency.

Seirdy 5y ago

I cited multiple pieces of FLOSS that employ and benefit from fuzzing, making an example out of a Linux bug that allowed sandbox escapes that was found through fuzzing rather than source analysis. Humans suck at finding subtle issues purely by reading source code. Running software is a more effective means to detect bugs. Once a bug has been detected, source code is really helpful to isolate and patch it.