2 ms·

> the actual execution does not follow scientific principles. It can't really. The practitioners are generally brought into a project under heavy non-disclosur

> the actual execution does not follow scientific principles. It can't really. The practitioners are generally brought into a project under heavy non-disclosure and generally allowed to keep stats on the project as long as it is completely anonymized. This makes it hard to publish data in a scientific research mode, because it's hard to reproduce or validate independently. Most of the time the practitioners are called in (hypothetical example) b/c a manager says, "I have the requirements for this project, I have 11 developers on the team, and we need to produce a working model of X by January, which is sufficiently defect free that we can put it in autos for initial test without losing a lot of cars." Thresholds are identified. Then the practitioner goes to his/her database of similar projects and says, "OK, when using these tools, we found you can get this level of defects on projects of x LOCs within the time frame you're going for. However, if you forgo Y, you can get higher levels in 10% less time. Etc." That can be valuable information, rather than just throwing developers at a project and hoping for the best. The books that you dismiss as consultants just publishing their anecdotal info are often the product of hundreds of projects. Sure, the data is empirical, but it doesn't mean it's not rigorous. You'll note that the numbers in books and articles are often presented as ranges, which is due to the sensitivity of one factor to the presence of other factors (size of project being a particularly prominent factor). Going to your original point, all established practitioners agree on the leading causes of defects. And this is where I fault most developers today--they plain don't know what those are. They'll guess, but they don't know. Which is what I think I hear you lamenting at the top of this thread. Disclaimer: I'm not a specialist in this field nor a consultant, but I find its research interesting and have spoken to practitioners.

wsy 8y ago
Thank you for taking the time to explain the background. I totally agree that consultants working from quantitative data about past projects are much better than the usual cargo cults. However, there are a lot of dangers in the process: - If you are a consultant in automotive embedded software, you will be contracted for these projects, so your data will be quite useless to guide startup Web application development (for example, LOC can make sense for embedded C, but not for languages like Scala or Haskell, where you can implement the same feature elegantly in 100 lines or clumsily in 1000). - As consultant, you want to generate revenue. So there is a strong incentive to oversell how solid your insights are. There is no counter-force in place to balance that bias out. - While scientific publication is somewhat broken, it is (in CS) a quite good quality control with respect to scientific method. The book did not undergo any quality control by other experts. So now it becomes a matter of personal trust towards the author. Summarized, I think that practitioner's data collections and their personal reports about them are useful in some cases, but cannot replace scientific empiric research.

wsy 8y ago

Thank you for taking the time to explain the background. I totally agree that consultants working from quantitative data about past projects are much better than the usual cargo cults. However, there are a lot of dangers in the process: - If you are a consultant in automotive embedded software, you will be contracted for these projects, so your data will be quite useless to guide startup Web application development (for example, LOC can make sense for embedded C, but not for languages like Scala or Haskell, where you can implement the same feature elegantly in 100 lines or clumsily in 1000). - As consultant, you want to generate revenue. So there is a strong incentive to oversell how solid your insights are. There is no counter-force in place to balance that bias out. - While scientific publication is somewhat broken, it is (in CS) a quite good quality control with respect to scientific method. The book did not undergo any quality control by other experts. So now it becomes a matter of personal trust towards the author. Summarized, I think that practitioner's data collections and their personal reports about them are useful in some cases, but cannot replace scientific empiric research.