3 ms·

This paper is emblematic of a serious problem in the software development field: lack of empirical research. It is unsurprisingly filled with opinion and anecdo

This paper is emblematic of a serious problem in the software development field: lack of empirical research. It is unsurprisingly filled with opinion and anecdote, with little mention of research in the area. And in the one actual study cited, "Does Test-Driven Development Really Improve Software Design Quality?", the author mis-characterizes the findings. Not that there is a huge set of research on unit testing to refer to--there isn't--and hence the problem. "Software development has been characterized from its origins by a serious want of empirical facts tested against reality, which provide evidence of the advantages or disadvantages of the different methods, techniques or tools used in the development of software systems."[1] In my view, the best thing we could do is to adopt "Evidence-Based Software Engineering"[2] as other disciplines have. This is more likely to have a major positive impact than the newest and hottest language, tool, or technique. [1] Reviewing 25 Years of Testing Technique Experiments. https://www.researchgate.net/publication/220277637_Reviewing_25_Years_of_Testing_Technique_Experiments https://www.researchgate.net/publication/220277637_Reviewing... [2] Evidence-Based Software Engineering. https://dl.acm.org/citation.cfm?id=999432 https://dl.acm.org/citation.cfm?id=999432

Frost1x 7y ago
I agree entirely. The more tendency there is to push development as an "engineering" discipline, the more evidence based research needs to be used to solidify or disprove certain development dogmas repeated over and over again, which in many cases, seemingly have no empirical evidence beyond a few anecdotal success cases. This is the norm in the industry, "you're doing it wrong, the best way is this way..."--based on what evidence that pertains to this case or has proven generalized success applicable here? In most cases, it's someone's successful anecdotal experiences which worked for the specific cases they were involved with. That doesn't mean those approaches can be abstracted away and generalized to all cases, but many in this industry do that regularly and critique others' approaches based on that. It becomes this competitive ego contest: well my work was at x, y, z solving q and was successful--making me an authority on a, b, c's problems solving p because x, y, z is a leader (appeal to authority fallacy)... etc. It's one thing to treat development as much of an art (which to me, it very well still is) but once you start treating it as a concrete discipline, you need to provide the lyme, cement, aggregates, water... evidence and studies showing approaches and how they faired across controls and varied cases.
- 0x445442 7y ago
  Agree with you and parent poster. I remember 15 years ago or more going through CMMI certification and there was a push to get to level 3. Our company hired a consultant who came in every eight weeks or so to track our progress, give direction etc. On one of his visits he started interviewing engineers individually to see how things were going and I asked him what the point of it all was and I could tell he was quite taken back. But then, not surprisingly, he said to be more efficient developing software. I then asked him if he or any organization he'd worked with had ever actually tracked the time it took to implement the process, to which he answered no. Then I asked him, then how the hell do you know if any of this is making software development more efficient?
  - deleted 7y ago
    [deleted]
- Retra 7y ago
  Is there a good way to fix it? I mean, if you were building bridges, certainly another engineer could show you data about material properties and successful builds, and you could reasonably extrapolate to your new (unique) design fairly well, but that's because you're building with the same physical materials and operating under the same laws. With software, it is a bit harder to find that common ground, because the 'laws' change if you are on a different architecture or handling different kinds of 'traffic', etc. For a bridge, you can just make a bigger beam because you would never try to build close to the scale which would make the material properties irrelevant, but with a computer, you can do this, and you might start to invalidate prior evidence. A large amount of software development dogma is built around scaling up, but that's also a huge problem in every engineering & scientific discipline. Though, to counter my own counter-point, it would be nice to have better analytical tools; more informative & accurate ways to understand & visualize how resources are actually being used when a program runs.
  - qznc 7y ago
    It is actually simple. Just look into any business where safety and security matters. I'm in automotive [0]. There is somebody who has to sign that the software is safe. Without someone to make that signature the car does not get released. For Free Software, look at Sqlite. There are some nice slides from 2009 [1]. In general it results in some simple rules like "100% test coverage". Of course, these rules (while simple) are not easy to satisfy and certainly expensive. [0] http://beza1e1.tuxen.de/aspice.html http://beza1e1.tuxen.de/aspice.html [1] https://www.sqlite.org/talks/wroclaw-20090310.pdf https://www.sqlite.org/talks/wroclaw-20090310.pdf
diego_moita 7y ago
I strongly agree but I warn that evidence is a lot harder than it seems because a lot of what we do is very hard to measure or even define. How do you reliably measure programmer productivity? Lines of code, inflection points, units of functionally... all have their flaws.

Frost1x 7y ago

I agree entirely. The more tendency there is to push development as an "engineering" discipline, the more evidence based research needs to be used to solidify or disprove certain development dogmas repeated over and over again, which in many cases, seemingly have no empirical evidence beyond a few anecdotal success cases. This is the norm in the industry, "you're doing it wrong, the best way is this way..."--based on what evidence that pertains to this case or has proven generalized success applicable here? In most cases, it's someone's successful anecdotal experiences which worked for the specific cases they were involved with. That doesn't mean those approaches can be abstracted away and generalized to all cases, but many in this industry do that regularly and critique others' approaches based on that. It becomes this competitive ego contest: well my work was at x, y, z solving q and was successful--making me an authority on a, b, c's problems solving p because x, y, z is a leader (appeal to authority fallacy)... etc. It's one thing to treat development as much of an art (which to me, it very well still is) but once you start treating it as a concrete discipline, you need to provide the lyme, cement, aggregates, water... evidence and studies showing approaches and how they faired across controls and varied cases.

0x445442 7y ago
Agree with you and parent poster. I remember 15 years ago or more going through CMMI certification and there was a push to get to level 3. Our company hired a consultant who came in every eight weeks or so to track our progress, give direction etc. On one of his visits he started interviewing engineers individually to see how things were going and I asked him what the point of it all was and I could tell he was quite taken back. But then, not surprisingly, he said to be more efficient developing software. I then asked him if he or any organization he'd worked with had ever actually tracked the time it took to implement the process, to which he answered no. Then I asked him, then how the hell do you know if any of this is making software development more efficient?
- deleted 7y ago
  [deleted]
Retra 7y ago
Is there a good way to fix it? I mean, if you were building bridges, certainly another engineer could show you data about material properties and successful builds, and you could reasonably extrapolate to your new (unique) design fairly well, but that's because you're building with the same physical materials and operating under the same laws. With software, it is a bit harder to find that common ground, because the 'laws' change if you are on a different architecture or handling different kinds of 'traffic', etc. For a bridge, you can just make a bigger beam because you would never try to build close to the scale which would make the material properties irrelevant, but with a computer, you can do this, and you might start to invalidate prior evidence. A large amount of software development dogma is built around scaling up, but that's also a huge problem in every engineering & scientific discipline. Though, to counter my own counter-point, it would be nice to have better analytical tools; more informative & accurate ways to understand & visualize how resources are actually being used when a program runs.
- qznc 7y ago
  It is actually simple. Just look into any business where safety and security matters. I'm in automotive [0]. There is somebody who has to sign that the software is safe. Without someone to make that signature the car does not get released. For Free Software, look at Sqlite. There are some nice slides from 2009 [1]. In general it results in some simple rules like "100% test coverage". Of course, these rules (while simple) are not easy to satisfy and certainly expensive. [0] http://beza1e1.tuxen.de/aspice.html http://beza1e1.tuxen.de/aspice.html [1] https://www.sqlite.org/talks/wroclaw-20090310.pdf https://www.sqlite.org/talks/wroclaw-20090310.pdf

diego_moita 7y ago

I strongly agree but I warn that evidence is a lot harder than it seems because a lot of what we do is very hard to measure or even define. How do you reliably measure programmer productivity? Lines of code, inflection points, units of functionally... all have their flaws.