8 ms·

> what's interesting isn't true ("Linus's law" being perhaps the most notorious example) > Even the high-level organizing notion of "cathedrals" and "bazaars",

> what's interesting isn't true ("Linus's law" being perhaps the most notorious example) > Even the high-level organizing notion of "cathedrals" and "bazaars", which should have been a lay-up, hasn't really proven out. you and i live oceans apart. a year of working around NixOS and especially linux on mobile phones has me working in very bazaar-like systems — where every component is swappable/composable — and seeing/using Linus’ law daily — in that people from out of the blue will patch things i’d been unable to figure out and when i hit edge cases that look familiar enough in any software i install i will debug it & send fixes upstream. i’m well aware the above is niche. on the other hand i’m fairly confident that niche would be inaccessible to most of us who are otherwise participating in it were it not leveraging these two concepts.

tptacek 3y ago
We do. As a vulnerability researcher, my take on "Linus's law" (which, like "Gell-mann Amnesia" isn't the product of the famous person it's named for) --- "given enough eyeballs, all bugs are shallow" --- is that has been effectively refuted by the recurrent discovery of grave vulnerabilities that have hidden in plain sight, sometimes for over a decade, and with the success of commercial vulnerability research work in eradicating those vulnerabilities.
- mjw1007 3y ago
  I don't think "bugs are shallow", or the "many eyeballs" section of this essay, are particularly talking about _discovering_ bugs. The author seems to have had a worldview in which bugs don't really matter if people aren't coming across them, and in which the difficult part of dealing with a bug is either reproducing it or getting from a symptom to a cause. If you were in a world where those two things are true then I think he's probably right that "many eyeballs" would help a great deal.
  - tptacek 3y ago
    It's not interesting to say that lots of interesting eyeballs are helpful. Anybody would have said that prior to this article's publication. Raymond makes a much stronger claim (which is why it has the force of "law"), and it hasn't borne out.
    - mjw1007 3y ago
      Again: the purported law says "shallow", not "soon discovered".
      - tptacek 3y ago
        No, you can't get away with this semantic dodge, because Raymond numbered what he believed were the most important lessons he was imparting, and the one corresponding to Linus' Law is: 8. Given a large enough beta-tester and co-developer base, almost every problem will be characterized quickly and the fix obvious to someone. He even attempted an axiomatic explanation: Maybe it shouldn't have been such a surprise, at that. Sociologists years ago discovered that the averaged opinion of a mass of equally expert (or equally ignorant) observers is quite a bit more reliable a predictor than the opinion of a single randomly-chosen one of the observers. They called this the Delphi effect. It appears that what Linus has shown is that this applies even to debugging an operating system—that the Delphi effect can tame development complexity even at the complexity level of an OS kernel.
        mjw1007 3y ago
        Well, I'm baffled then. From where I'm sitting that point 8 is clearly talking about what happens after a bug is discovered, and not about discovering bugs. The longer paragraph doesn't seem to contradict the notion either. My impression (based on the "How Many Eyeballs Tame Complexity" chapter) is that Raymond thought that "debugging" means "fixing bugs". If I were criticising this part of essay, I'd say the main weakness is that the things Raymond thought of as "taming complexity" weren't really addressing the hard problem of reducing the number of bugs.
        3y ago
  - pwdisswordfishc 3y ago
    > The author seems to have had a worldview in which bugs don't really matter That would explain so much… https://gitlab.com/esr/gif2png/-/commit/a8a761561b2a071e7452a00868efd5bdf795c443 https://gitlab.com/esr/gif2png/-/commit/a8a761561b2a071e7452...
    - DonHopkins 3y ago
      I don't think ESR would appreciate the irony of you directing many eyeballs to his carelessly bug-ridden code.
- lmm 3y ago
  None of the significant security vulnerabilities that I can remember have been "deep" - subtle things that require extensive familiarity with the specific codebase and could never have been found by a drive-by contributor, which is the idea that ESR is refuting. Debian keygen bug? Dumb one-liner. Heartbleed? Dumb one-liner. Goto fail? Technically a logic bug, but a two-line thing that could be understood by reading that one file without any additional context. I've seen cases where exploitation was complex, but even then, that tends to be because the full exploit is chaining together several bugs, each of which is individually simple and could be fixed by a drive-by contributor.
  - pron 3y ago
    If that was the intended meaning of "shallow", why does the number of eyeballs matter? I think the more obvious interpretation is "quickly discovered".
    - tptacek 3y ago
      Yes. We don't have to guess at this, because Raymond wrote that explicitly in the article.
      - lmm 3y ago
        In the article the number of eyeballs is pretty explicitly a minor caveat, and the main point is the shallowness in the sense of approachability: > In Linus's Law, I think, lies the core difference underlying the cathedral-builder and bazaar styles. In the cathedral-builder view of programming, bugs and development problems are tricky, insidious, deep phenomena. It takes months of scrutiny by a dedicated few to develop confidence that you've winkled them all out. Thus the long release intervals, and the inevitable disappointment when long-awaited releases are not perfect. > In the bazaar view, on the other hand, you assume that bugs are generally shallow phenomena—or, at least, that they turn shallow pretty quickly when exposed to a thousand eager co-developers pounding on every single new release. Accordingly you release often in order to get more corrections, and as a beneficial side effect you have less to lose if an occasional botch gets out the door.
- colinsane 3y ago
  >> you and i live oceans apart. > We do. which, just to clarify, i didn’t mean as “one of us contradicts the other” but that “we experience different slices of life which are guided by different priorities”. i very much wouldn’t want to be a security researcher in the open source areas i occupy now. likewise i wouldn’t want to use the development practices i experience in community open source were i instead building a fighter jet. looking at the bizarre overlap between “open source contributors who like {Nix,Haskell/FP,Rust}” and “people i know who work for defense contractors” though, i’m actually optimistic that our distance could shrink in time.
- DonHopkins 3y ago
  "My favorite part of the "many eyes" argument is how few bugs were found by the two eyes of Eric (the originator of the statement). All the many eyes are apparently attached to a lot of hands that type lots of words about many eyes, and never actually audit code." -Theo De Raadt
- musicale 3y ago
  It seems possible that some better eyeballs (possibly automated ones) have been developed and deployed en masse.

tptacek 3y ago

We do. As a vulnerability researcher, my take on "Linus's law" (which, like "Gell-mann Amnesia" isn't the product of the famous person it's named for) --- "given enough eyeballs, all bugs are shallow" --- is that has been effectively refuted by the recurrent discovery of grave vulnerabilities that have hidden in plain sight, sometimes for over a decade, and with the success of commercial vulnerability research work in eradicating those vulnerabilities.

mjw1007 3y ago
I don't think "bugs are shallow", or the "many eyeballs" section of this essay, are particularly talking about _discovering_ bugs. The author seems to have had a worldview in which bugs don't really matter if people aren't coming across them, and in which the difficult part of dealing with a bug is either reproducing it or getting from a symptom to a cause. If you were in a world where those two things are true then I think he's probably right that "many eyeballs" would help a great deal.
- tptacek 3y ago
  It's not interesting to say that lots of interesting eyeballs are helpful. Anybody would have said that prior to this article's publication. Raymond makes a much stronger claim (which is why it has the force of "law"), and it hasn't borne out.
  - mjw1007 3y ago
    Again: the purported law says "shallow", not "soon discovered".
    - tptacek 3y ago
      No, you can't get away with this semantic dodge, because Raymond numbered what he believed were the most important lessons he was imparting, and the one corresponding to Linus' Law is: 8. Given a large enough beta-tester and co-developer base, almost every problem will be characterized quickly and the fix obvious to someone. He even attempted an axiomatic explanation: Maybe it shouldn't have been such a surprise, at that. Sociologists years ago discovered that the averaged opinion of a mass of equally expert (or equally ignorant) observers is quite a bit more reliable a predictor than the opinion of a single randomly-chosen one of the observers. They called this the Delphi effect. It appears that what Linus has shown is that this applies even to debugging an operating system—that the Delphi effect can tame development complexity even at the complexity level of an OS kernel.
      - mjw1007 3y ago
        Well, I'm baffled then. From where I'm sitting that point 8 is clearly talking about what happens after a bug is discovered, and not about discovering bugs. The longer paragraph doesn't seem to contradict the notion either. My impression (based on the "How Many Eyeballs Tame Complexity" chapter) is that Raymond thought that "debugging" means "fixing bugs". If I were criticising this part of essay, I'd say the main weakness is that the things Raymond thought of as "taming complexity" weren't really addressing the hard problem of reducing the number of bugs.
        3y ago
- pwdisswordfishc 3y ago
  > The author seems to have had a worldview in which bugs don't really matter That would explain so much… https://gitlab.com/esr/gif2png/-/commit/a8a761561b2a071e7452a00868efd5bdf795c443 https://gitlab.com/esr/gif2png/-/commit/a8a761561b2a071e7452...
  - DonHopkins 3y ago
    I don't think ESR would appreciate the irony of you directing many eyeballs to his carelessly bug-ridden code.
lmm 3y ago
None of the significant security vulnerabilities that I can remember have been "deep" - subtle things that require extensive familiarity with the specific codebase and could never have been found by a drive-by contributor, which is the idea that ESR is refuting. Debian keygen bug? Dumb one-liner. Heartbleed? Dumb one-liner. Goto fail? Technically a logic bug, but a two-line thing that could be understood by reading that one file without any additional context. I've seen cases where exploitation was complex, but even then, that tends to be because the full exploit is chaining together several bugs, each of which is individually simple and could be fixed by a drive-by contributor.
- pron 3y ago
  If that was the intended meaning of "shallow", why does the number of eyeballs matter? I think the more obvious interpretation is "quickly discovered".
  - tptacek 3y ago
    Yes. We don't have to guess at this, because Raymond wrote that explicitly in the article.
    - lmm 3y ago
      In the article the number of eyeballs is pretty explicitly a minor caveat, and the main point is the shallowness in the sense of approachability: > In Linus's Law, I think, lies the core difference underlying the cathedral-builder and bazaar styles. In the cathedral-builder view of programming, bugs and development problems are tricky, insidious, deep phenomena. It takes months of scrutiny by a dedicated few to develop confidence that you've winkled them all out. Thus the long release intervals, and the inevitable disappointment when long-awaited releases are not perfect. > In the bazaar view, on the other hand, you assume that bugs are generally shallow phenomena—or, at least, that they turn shallow pretty quickly when exposed to a thousand eager co-developers pounding on every single new release. Accordingly you release often in order to get more corrections, and as a beneficial side effect you have less to lose if an occasional botch gets out the door.
colinsane 3y ago
>> you and i live oceans apart. > We do. which, just to clarify, i didn’t mean as “one of us contradicts the other” but that “we experience different slices of life which are guided by different priorities”. i very much wouldn’t want to be a security researcher in the open source areas i occupy now. likewise i wouldn’t want to use the development practices i experience in community open source were i instead building a fighter jet. looking at the bizarre overlap between “open source contributors who like {Nix,Haskell/FP,Rust}” and “people i know who work for defense contractors” though, i’m actually optimistic that our distance could shrink in time.
DonHopkins 3y ago
"My favorite part of the "many eyes" argument is how few bugs were found by the two eyes of Eric (the originator of the statement). All the many eyes are apparently attached to a lot of hands that type lots of words about many eyes, and never actually audit code." -Theo De Raadt
musicale 3y ago
It seems possible that some better eyeballs (possibly automated ones) have been developed and deployed en masse.