3 ms·

These studies are bs because they capture corellations and confuse them with causations. Good studies need controlled laboratory conditions to remove external

These studies are bs because they capture corellations and confuse them with causations. Good studies need controlled laboratory conditions to remove external variables. For example, maybe FP projects are written by coders who have more years of experience on average. So in that case it's not about the language paradigm, any facts derived from the study would actually be about the individuals who write in that language. And it's not surprising at all that more experienced individuals write fewer bugs. This is just one factor that may be ignored; there are probably hundreds that can cause significant distortions in the results. It would be like saying that Norwegian is a highly effective language because they have a very high GDP per capita. This completely ignores the more significant fact that the country has vast amounts of offshore oil to capitalize on.

xgk 7y ago
studies are bs because We should not let the perfect be the enemy of the good. The problem is that we do not currently know how to carry out studies on the efficacy of programming language paradigms under "controlled laboratory conditions". It's known to be hard. Feel free to change this and become famous! The studies discussed in the article are but first steps towards: - a better empirical grounding of programming language & software engineering research; - more emphasis on reproducibility in science. I'm glad that Vitek, Berger et al are starting serious empirical PL/SE research, and care about reproducibility! Bravo!
calpaterson 7y ago
A study that captures a correlation instead of causation directly is not completely useless. It seems wise to try to draw some conclusions from the massive corpus of OSS software that is available given that it's hard to imagine any researcher will ever get the budget to do even one controlled test on a meaningfully sized software project. Anyway, it's not hard to imagine an improved version of the study that knows about the years of experience of each contributor (even if that necessitates reducing the size of the dataset a bit) and controls for it. I suppose that would be the "empirical software engineering" equivalent of econometrics.

xgk 7y ago

studies are bs because We should not let the perfect be the enemy of the good. The problem is that we do not currently know how to carry out studies on the efficacy of programming language paradigms under "controlled laboratory conditions". It's known to be hard. Feel free to change this and become famous! The studies discussed in the article are but first steps towards: - a better empirical grounding of programming language & software engineering research; - more emphasis on reproducibility in science. I'm glad that Vitek, Berger et al are starting serious empirical PL/SE research, and care about reproducibility! Bravo!

calpaterson 7y ago

A study that captures a correlation instead of causation directly is not completely useless. It seems wise to try to draw some conclusions from the massive corpus of OSS software that is available given that it's hard to imagine any researcher will ever get the budget to do even one controlled test on a meaningfully sized software project. Anyway, it's not hard to imagine an improved version of the study that knows about the years of experience of each contributor (even if that necessitates reducing the size of the dataset a bit) and controls for it. I suppose that would be the "empirical software engineering" equivalent of econometrics.