5 ms·

YAGNI is a sad example of the state of software engineering: our field follows rules of thumb, based on anecdotes as evidence. When will experimental validation

by wsy 8y ago

YAGNI is a sad example of the state of software engineering: our field follows rules of thumb, based on anecdotes as evidence. When will experimental validation become best practice, like in the other engineering disciplines?

ternaryoperator 8y ago
There is a whole field of study--what was originally called software engineering--that does quantitative, empirical analysis of software development. Its practitioners are often called by managers entering into large projects that have tight quality requirements so that the optimal choices can be made in terms of tooling and techniques. (For example, large-scale automotive software, where there is far less tolerance for error than in IT.)
- wsy 8y ago
  Do you have some pointers to important papers of this field of study?
  - ternaryoperator 8y ago
    [1] Article from Dr. Dobb's gives a very high-level overview of the kind of data that is available. [2] is one of the foundational books in this area. Its analysis of testing effectiveness is fascinating because it's so revealing. For example, comparing defect rates in projects that use TDD vs. non-TDD; the effectiveness of static analysis; etc. [1] http://www.drdobbs.com/225701139 http://www.drdobbs.com/225701139 [2] https://www.amazon.com/dp/0132582201/ https://www.amazon.com/dp/0132582201/
    - wsy 8y ago
      Thanks a lot!
      - wsy 8y ago
        After having read a bit, I'm a bit disappointed. While the intent goes in the direction I would like to see, the actual execution does not follow scientific principles. "Consultant collects some data, derives conclusions, and writes a book" is actually again only anecdotal. You need to think about your sample before collecting the data, otherwise your outcome is biased by who gave you a consulting gig. Consequently, the work has never been published as peer-reviewed paper with experiment and/or real-world data, so that someone else could validate or reproduce results. Is that assessment correct, or did I overlook scientific papers?
        andrewbinstock 8y ago
        > the actual execution does not follow scientific principles. It can't really. The practitioners are generally brought into a project under heavy non-disclosure and generally allowed to keep stats on the project as long as it is completely anonymized. This makes it hard to publish data in a scientific research mode, because it's hard to reproduce or validate independently. Most of the time the practitioners are called in (hypothetical example) b/c a manager says, "I have the requirements for this project, I have 11 developers on the team, and we need to produce a working model of X by January, which is sufficiently defect free that we can put it in autos for initial test without losing a lot of cars." Thresholds are identified. Then the practitioner goes to his/her database of similar projects and says, "OK, when using these tools, we found you can get this level of defects on projects of x LOCs within the time frame you're going for. However, if you forgo Y, you can get higher levels in 10% less time. Etc." That can be valuable information, rather than just throwing developers at a project and hoping for the best. The books that you dismiss as consultants just publishing their anecdotal info are often the product of hundreds of projects. Sure, the data is empirical, but it doesn't mean it's not rigorous. You'll note that the numbers in books and articles are often presented as ranges, which is due to the sensitivity of one factor to the presence of other factors (size of project being a particularly prominent factor). Going to your original point, all established practitioners agree on the leading causes of defects. And this is where I fault most developers today--they plain don't know what those are. They'll guess, but they don't know. Which is what I think I hear you lamenting at the top of this thread. Disclaimer: I'm not a specialist in this field nor a consultant, but I find its research interesting and have spoken to practitioners.
- curuinor 8y ago
  Those from that field have also often failed in order-of-magnitude ways.
  - ternaryoperator 8y ago
    > often failed in order-of-magnitude ways so, there should be lots of examples, right? References?
    - curuinor 8y ago
      these folks were the first to push CASE tools, which are a footnote in software history because they failed rather completely in actual production usage. they invented COCOMO, which never gets used for the easy reason of routine order-of-magnitude margins of error (60% in studies, 600% in practice...). they were the biggest pushers or UML and graphical programming, which dies and lives again and again in increasingly zombified states... have a dealie about cocomo specifically, I suppose. http://shape-of-code.coding-guidelines.com/2016/05/19/cocomo-how-not-to-fit-a-model-to-data/ http://shape-of-code.coding-guidelines.com/2016/05/19/cocomo...

by wsy 8y ago

ternaryoperator 8y ago

There is a whole field of study--what was originally called software engineering--that does quantitative, empirical analysis of software development. Its practitioners are often called by managers entering into large projects that have tight quality requirements so that the optimal choices can be made in terms of tooling and techniques. (For example, large-scale automotive software, where there is far less tolerance for error than in IT.)

wsy 8y ago
Do you have some pointers to important papers of this field of study?
- ternaryoperator 8y ago
  [1] Article from Dr. Dobb's gives a very high-level overview of the kind of data that is available. [2] is one of the foundational books in this area. Its analysis of testing effectiveness is fascinating because it's so revealing. For example, comparing defect rates in projects that use TDD vs. non-TDD; the effectiveness of static analysis; etc. [1] http://www.drdobbs.com/225701139 http://www.drdobbs.com/225701139 [2] https://www.amazon.com/dp/0132582201/ https://www.amazon.com/dp/0132582201/
  - wsy 8y ago
    Thanks a lot!
    - wsy 8y ago
      After having read a bit, I'm a bit disappointed. While the intent goes in the direction I would like to see, the actual execution does not follow scientific principles. "Consultant collects some data, derives conclusions, and writes a book" is actually again only anecdotal. You need to think about your sample before collecting the data, otherwise your outcome is biased by who gave you a consulting gig. Consequently, the work has never been published as peer-reviewed paper with experiment and/or real-world data, so that someone else could validate or reproduce results. Is that assessment correct, or did I overlook scientific papers?
      - andrewbinstock 8y ago
        > the actual execution does not follow scientific principles. It can't really. The practitioners are generally brought into a project under heavy non-disclosure and generally allowed to keep stats on the project as long as it is completely anonymized. This makes it hard to publish data in a scientific research mode, because it's hard to reproduce or validate independently. Most of the time the practitioners are called in (hypothetical example) b/c a manager says, "I have the requirements for this project, I have 11 developers on the team, and we need to produce a working model of X by January, which is sufficiently defect free that we can put it in autos for initial test without losing a lot of cars." Thresholds are identified. Then the practitioner goes to his/her database of similar projects and says, "OK, when using these tools, we found you can get this level of defects on projects of x LOCs within the time frame you're going for. However, if you forgo Y, you can get higher levels in 10% less time. Etc." That can be valuable information, rather than just throwing developers at a project and hoping for the best. The books that you dismiss as consultants just publishing their anecdotal info are often the product of hundreds of projects. Sure, the data is empirical, but it doesn't mean it's not rigorous. You'll note that the numbers in books and articles are often presented as ranges, which is due to the sensitivity of one factor to the presence of other factors (size of project being a particularly prominent factor). Going to your original point, all established practitioners agree on the leading causes of defects. And this is where I fault most developers today--they plain don't know what those are. They'll guess, but they don't know. Which is what I think I hear you lamenting at the top of this thread. Disclaimer: I'm not a specialist in this field nor a consultant, but I find its research interesting and have spoken to practitioners.
curuinor 8y ago
Those from that field have also often failed in order-of-magnitude ways.
- ternaryoperator 8y ago
  > often failed in order-of-magnitude ways so, there should be lots of examples, right? References?
  - curuinor 8y ago
    these folks were the first to push CASE tools, which are a footnote in software history because they failed rather completely in actual production usage. they invented COCOMO, which never gets used for the easy reason of routine order-of-magnitude margins of error (60% in studies, 600% in practice...). they were the biggest pushers or UML and graphical programming, which dies and lives again and again in increasingly zombified states... have a dealie about cocomo specifically, I suppose. http://shape-of-code.coding-guidelines.com/2016/05/19/cocomo-how-not-to-fit-a-model-to-data/ http://shape-of-code.coding-guidelines.com/2016/05/19/cocomo...