5 ms·

Evidence-based Software Engineering [pdf]

12 points by mblakele 15y ago

mblakele 15y ago
This paper is from 2004, but I don't think I've seen it discussed here before. Any experiences or insights to share on this topic?
- spenrose 15y ago
  Greg Wilson did a book, which I did not enjoy, on this topic: http://shop.oreilly.com/product/9780596808303.do http://shop.oreilly.com/product/9780596808303.do and a slideshow, which I recommend: http://www.slideshare.net/gvwilson/bits-of-evidence-2338367 http://www.slideshare.net/gvwilson/bits-of-evidence-2338367 . My take is that there is much to learn from science about how to evaluate propositions regarding software engineering (most, but not all, of them are unsupported) but few new useful ideas. Another reference along these lines: http://www.amazon.com/Facts-Fallacies-Software-Engineering-Robert/dp/0321117425 http://www.amazon.com/Facts-Fallacies-Software-Engineering-R...
- bhattisatish 15y ago
  I haven't yet read the paper but Joel of Joel on software fame, had discussed it once. See http://www.joelonsoftware.com/items/2007/10/26.html http://www.joelonsoftware.com/items/2007/10/26.html
- neuroelectronic 15y ago
  Too expensive.
andrewcooke 15y ago
the cornerstone of modern evidence-based medicine is the randomised trial. trials can produce statistically significant results because there are many humans and they are all quite similar. unfortunately, software projects are both more diverse and less common. this is a critical difference: you simply cannot replicate evidence-based medicine in the field of software development. section 5.2.2 addresses some of this, but is then pretty much ignored in the conclusion.
- mblakele 15y ago
  Yes, that struck me too. Some developers do randomized trials of user experience, but such trials aren't tests of development techniques - unless perhaps you are working on an IDE. And yet I think the points about negative evidence are relevant to software. Whenever a hot new trend surfaces, it seems like negative early experiences aren't reported. Later on there is usually a backlash, but it would be in our interest to shorten that cycle.
vanni 15y ago
According to HN Guidelines (http://ycombinator.com/newsguidelines.html http://ycombinator.com/newsguidelines.html), next time please append [pdf] to the title. Thank you.
- mblakele 15y ago
  Done, thanks.
  - vanni 15y ago
    Well done. :)
akkartik 15y ago
Is table 1 really just doing a search-and-replace of 'clinical' for 'software engineering'??

12 points by mblakele 15y ago

This paper is from 2004, but I don't think I've seen it discussed here before. Any experiences or insights to share on this topic?

spenrose 15y ago
Greg Wilson did a book, which I did not enjoy, on this topic: http://shop.oreilly.com/product/9780596808303.do http://shop.oreilly.com/product/9780596808303.do and a slideshow, which I recommend: http://www.slideshare.net/gvwilson/bits-of-evidence-2338367 http://www.slideshare.net/gvwilson/bits-of-evidence-2338367 . My take is that there is much to learn from science about how to evaluate propositions regarding software engineering (most, but not all, of them are unsupported) but few new useful ideas. Another reference along these lines: http://www.amazon.com/Facts-Fallacies-Software-Engineering-Robert/dp/0321117425 http://www.amazon.com/Facts-Fallacies-Software-Engineering-R...
bhattisatish 15y ago
I haven't yet read the paper but Joel of Joel on software fame, had discussed it once. See http://www.joelonsoftware.com/items/2007/10/26.html http://www.joelonsoftware.com/items/2007/10/26.html
neuroelectronic 15y ago
Too expensive.

andrewcooke 15y ago

the cornerstone of modern evidence-based medicine is the randomised trial. trials can produce statistically significant results because there are many humans and they are all quite similar. unfortunately, software projects are both more diverse and less common. this is a critical difference: you simply cannot replicate evidence-based medicine in the field of software development. section 5.2.2 addresses some of this, but is then pretty much ignored in the conclusion.

mblakele 15y ago
Yes, that struck me too. Some developers do randomized trials of user experience, but such trials aren't tests of development techniques - unless perhaps you are working on an IDE. And yet I think the points about negative evidence are relevant to software. Whenever a hot new trend surfaces, it seems like negative early experiences aren't reported. Later on there is usually a backlash, but it would be in our interest to shorten that cycle.

vanni 15y ago

According to HN Guidelines (http://ycombinator.com/newsguidelines.html http://ycombinator.com/newsguidelines.html), next time please append [pdf] to the title. Thank you.

mblakele 15y ago
Done, thanks.
- vanni 15y ago
  Well done. :)

akkartik 15y ago

Is table 1 really just doing a search-and-replace of 'clinical' for 'software engineering'??