4 ms·

Anyone who has a couple of year into software engineering knows that there is diminishing return over the number of "eye balls" and that some problems are just

Anyone who has a couple of year into software engineering knows that there is diminishing return over the number of "eye balls" and that some problems are just god damn hard or need to be looked at with the problem in mind. Software quality is orthogonal to the openness or closeness of the source code. I think it is more about the will to do something great, the ability to listen to the user base and of course the technical skills of the authors.

netcan 12y ago
I don't think orthogonal is quite right either. I think it's just a matter of being complicated when you get dig into it. Open Source has certain strategies for quality that work, certain advantages (and disadvantages), different economies which means different abundances and scarcities of different resources. This "Linus’ Law" expresses something novel about the way open source projects can benefit from the advantages of being what they are. It's not an empty statement just not an airtight, always-right rule.
paulmd 12y ago
Yeah, number of eyeballs helps but it's not a catch-all. In some ways it could actually be a hindrance, since less-experienced eyeballs may not be able to mind all the necessary details, or may introduce further bugs/vulnerabilities. For our purposes, these are "bad programmers", whether or not they're good programmers on other stuff. A simple model for code quality is something like Q =(good programmers) / (good + bad programmers). I would actually argue that it's not even related to the average, and that bad programmers can degrade quality disproportionately to their actual number. I think it might be something closer to Q = (good programmers)^2 / (good + bad programmers)^2. This is what it seems like people are getting at with the whole "negative productivity" in the good/bad/10x programmer framework. Openness/closeness of the source code isn't the whole picture but the open-source model can more easily run into the too-many-cooks problem if the source is not properly gate-kept and reviewed in the aggregate. Thus I don't think it's entirely orthogonal. Of course closed-source software is typically commercial, which runs into its own set of pressures that degrade code quality. I think much more sophisticated testing systems are what will really boost code quality. More advanced detection suites that throw more warnings, mandated full unit test coverage, plus CI frameworks that make sure that your code isn't merged if it doesn't clear the warnings and pass the full-coverage unit test. Randomized address systems and mandated array-bounds checking that catch undefined behavior and off-by-one errors. Basically, making things fail noisily instead of silently and forcing contributors to pay attention. The C compiler in particular is really bad on the "throwing warnings" thing, even before you get yourself into other kinds of trouble. As in, according to the C standard, "rm -rf /" (or literally any other behavior the compiler wants) is a valid output behavior if you miss a closing quote([1]), rather than a compile-time error. That's an absurd definition in a security-minded world, and that's just the most egregious example of undefined behavior allowed by the C standard. JVM-style managed code and declared exceptions are annoying but do seem to be a step in the right direction from a security perspective. [1] http://blog.regehr.org/archives/213 http://blog.regehr.org/archives/213
lmm 12y ago
The article shows it's not about that > Perhaps you think these 30 bugs are really tricky, deeply hidden and complicated logic monsters that would explain the time they took to get found? Nope, I would say that every single one of them are pretty obvious once you spot them and none of them take a very long time for a reviewer to understand. For me the takeaway is that even something as basic and mature as curl can't get it right in C. Two buffer overflows in as many years, along with TLS and HTTP failures. It's past time to move onto better tools. I hope the "bitcoin piñata" calls attention to the fact that there's now a full SSL stack available in a substantially safer language.
- jacquesm 12y ago
  It's the devil that we know. New environments: new exploit classes.
  - schoen 12y ago
    There are some great examples of that (return-oriented programming as a way of getting around non-executable data segments), but surely there are some tools where adopting them was a pure win for safety and correctness over the status quo.
bch 12y ago
> or need to be looked at with the problem in mind. The article addresses how hard some of the bugs were (they were not hard), but additionally, sometimes (often?) code needs to be looked at without the problem in mind -- who hasn't written code (or a paper, or comment on HN) and proofed it immediately, committed it, and realized it has an error that your mind elided over because you were so sure "what you meant" was there that you glossed over a dumb error. Having a different mind review these things avoids that problem. Edit: I glossed over this -- I was also referring to proofing your work much later, so it's not still fresh in your mind "as you think it is". So, bring a "different mind" to the table. s/completely different mind/different mind/

netcan 12y ago

I don't think orthogonal is quite right either. I think it's just a matter of being complicated when you get dig into it. Open Source has certain strategies for quality that work, certain advantages (and disadvantages), different economies which means different abundances and scarcities of different resources. This "Linus’ Law" expresses something novel about the way open source projects can benefit from the advantages of being what they are. It's not an empty statement just not an airtight, always-right rule.

paulmd 12y ago

Yeah, number of eyeballs helps but it's not a catch-all. In some ways it could actually be a hindrance, since less-experienced eyeballs may not be able to mind all the necessary details, or may introduce further bugs/vulnerabilities. For our purposes, these are "bad programmers", whether or not they're good programmers on other stuff. A simple model for code quality is something like Q =(good programmers) / (good + bad programmers). I would actually argue that it's not even related to the average, and that bad programmers can degrade quality disproportionately to their actual number. I think it might be something closer to Q = (good programmers)^2 / (good + bad programmers)^2. This is what it seems like people are getting at with the whole "negative productivity" in the good/bad/10x programmer framework. Openness/closeness of the source code isn't the whole picture but the open-source model can more easily run into the too-many-cooks problem if the source is not properly gate-kept and reviewed in the aggregate. Thus I don't think it's entirely orthogonal. Of course closed-source software is typically commercial, which runs into its own set of pressures that degrade code quality. I think much more sophisticated testing systems are what will really boost code quality. More advanced detection suites that throw more warnings, mandated full unit test coverage, plus CI frameworks that make sure that your code isn't merged if it doesn't clear the warnings and pass the full-coverage unit test. Randomized address systems and mandated array-bounds checking that catch undefined behavior and off-by-one errors. Basically, making things fail noisily instead of silently and forcing contributors to pay attention. The C compiler in particular is really bad on the "throwing warnings" thing, even before you get yourself into other kinds of trouble. As in, according to the C standard, "rm -rf /" (or literally any other behavior the compiler wants) is a valid output behavior if you miss a closing quote([1]), rather than a compile-time error. That's an absurd definition in a security-minded world, and that's just the most egregious example of undefined behavior allowed by the C standard. JVM-style managed code and declared exceptions are annoying but do seem to be a step in the right direction from a security perspective. [1] http://blog.regehr.org/archives/213 http://blog.regehr.org/archives/213

lmm 12y ago

The article shows it's not about that > Perhaps you think these 30 bugs are really tricky, deeply hidden and complicated logic monsters that would explain the time they took to get found? Nope, I would say that every single one of them are pretty obvious once you spot them and none of them take a very long time for a reviewer to understand. For me the takeaway is that even something as basic and mature as curl can't get it right in C. Two buffer overflows in as many years, along with TLS and HTTP failures. It's past time to move onto better tools. I hope the "bitcoin piñata" calls attention to the fact that there's now a full SSL stack available in a substantially safer language.

jacquesm 12y ago
It's the devil that we know. New environments: new exploit classes.
- schoen 12y ago
  There are some great examples of that (return-oriented programming as a way of getting around non-executable data segments), but surely there are some tools where adopting them was a pure win for safety and correctness over the status quo.

bch 12y ago

> or need to be looked at with the problem in mind. The article addresses how hard some of the bugs were (they were not hard), but additionally, sometimes (often?) code needs to be looked at without the problem in mind -- who hasn't written code (or a paper, or comment on HN) and proofed it immediately, committed it, and realized it has an error that your mind elided over because you were so sure "what you meant" was there that you glossed over a dumb error. Having a different mind review these things avoids that problem. Edit: I glossed over this -- I was also referring to proofing your work much later, so it's not still fresh in your mind "as you think it is". So, bring a "different mind" to the table. s/completely different mind/different mind/