3 ms·

There are so many compounding factors here that the conclusion seems unreliable. It also reminds me a lot of the streetlight effect[1], in that defects may simp

by sgk284 6y ago

There are so many compounding factors here that the conclusion seems unreliable. It also reminds me a lot of the streetlight effect[1], in that defects may simply be easier to detect in smaller functions so that's where we find them. Many of the conclusions center around density and lines of code. This is weird because if smaller functions lead to less overall code, then if you have 2 bugs in a 1,000 sloc codebase it'll measure worse than 6 bugs in a 10,000 sloc codebase that does the exact same thing. It might be more valuable to compute the defect rate per cyclomatic or kolmogorov complexity. Or some other measure that's independent of line length and then figure out how function size impacts those complexity measures. [1] https://en.wikipedia.org/wiki/Streetlight_effect https://en.wikipedia.org/wiki/Streetlight_effect

eesmith 6y ago
What do you consider to be the conclusion(s)? I ask because the conclusion is only firm about very short functions, otherwise saying "[f]or longer functions, the picture is less clear" - which you agree with, yes? Part of the conclusion concerns how short functions appear to be harder to debug, which seems rather the opposite of your hypothesis that defects are easier to find in them. (One of the contributing arguments is that the indirection of a function name adds its own mental overhead.) Nor do I see a conclusion that smaller functions lead to less overall code. The article seems more concerned about questions like "should [we] refactor our long methods into short ones to avoid defects", with that study concluding no. And the article addresses code complexity several times, most directly as: """The literature doesn’t provide a straightforward way to measure which feature (length or number of methods) is more significant when predicting defects. It’s quite possible that, for example, measures such as “average method complexity” and “number of methods in a class” simply act as 2nd order estimators for number of lines, in which case we’d simply be comparing which measure correlates better with the underlying metric.""" I don't see how your answer fits in, so did I misread the article that badly?
leto_ii 6y ago
I think you make a valuable point. If somebody writes concise, clear code with little duplication, but still has a bug, their stats might look worse simply because their code is shorter. It's also not clear that bug rates should be the only/main thing to look at. Complexity measures can also help to indicate if the code is easy to read/extend. In my experience low cyclomatic complexity does correlate with more readable code.
- tsimionescu 6y ago
  > If somebody writes concise, clear code with little duplication, but still has a bug, their stats might look worse simply because their code is shorter. Fortunately, the science here me to indicate the opposite: the less code you have, the fewer defects per LoC you'll have. This is also indicated in the article, which mentions that both function size and function length were found to be correlated to higher defect density, meaning that both may in fact simply be estimators of overall code length - that overall code length correlates with defect density per LoC. > It's also not clear that bug rates should be the only/main thing to look at. Complexity measures can also help to indicate if the code is easy to read/extend. In my experience low cyclomatic complexity does correlate with more readable code. The article does actually cite some (necessarily small-scale) studies on this as well, which have also tended to find that larger functions are easier to read for debugging purposes, and that it was easier to add new functionality to the system with larger functions; but that it was easier to modify the existing functionality of the system with smaller functions. This is all in the section entitled 'Practical Effects'.
  - leto_ii 6y ago
    > Fortunately, the science here me to indicate the opposite: the less code you have, the fewer defects per LoC you'll have I judged things conditional on the fact that a bug exists. If you apply the technique of counting bugs/loc blindly you may end up favouring longer, more tangled code. Think about it this way: if your goal is low bugs/loc then when a bug shows up you will be incentivized to fix it in a way that increases loc, instead maybe of simplifying thing. This will lead to fewer bugs and lower bug density.

by sgk284 6y ago

eesmith 6y ago

What do you consider to be the conclusion(s)? I ask because the conclusion is only firm about very short functions, otherwise saying "[f]or longer functions, the picture is less clear" - which you agree with, yes? Part of the conclusion concerns how short functions appear to be harder to debug, which seems rather the opposite of your hypothesis that defects are easier to find in them. (One of the contributing arguments is that the indirection of a function name adds its own mental overhead.) Nor do I see a conclusion that smaller functions lead to less overall code. The article seems more concerned about questions like "should [we] refactor our long methods into short ones to avoid defects", with that study concluding no. And the article addresses code complexity several times, most directly as: """The literature doesn’t provide a straightforward way to measure which feature (length or number of methods) is more significant when predicting defects. It’s quite possible that, for example, measures such as “average method complexity” and “number of methods in a class” simply act as 2nd order estimators for number of lines, in which case we’d simply be comparing which measure correlates better with the underlying metric.""" I don't see how your answer fits in, so did I misread the article that badly?

leto_ii 6y ago

I think you make a valuable point. If somebody writes concise, clear code with little duplication, but still has a bug, their stats might look worse simply because their code is shorter. It's also not clear that bug rates should be the only/main thing to look at. Complexity measures can also help to indicate if the code is easy to read/extend. In my experience low cyclomatic complexity does correlate with more readable code.

tsimionescu 6y ago
> If somebody writes concise, clear code with little duplication, but still has a bug, their stats might look worse simply because their code is shorter. Fortunately, the science here me to indicate the opposite: the less code you have, the fewer defects per LoC you'll have. This is also indicated in the article, which mentions that both function size and function length were found to be correlated to higher defect density, meaning that both may in fact simply be estimators of overall code length - that overall code length correlates with defect density per LoC. > It's also not clear that bug rates should be the only/main thing to look at. Complexity measures can also help to indicate if the code is easy to read/extend. In my experience low cyclomatic complexity does correlate with more readable code. The article does actually cite some (necessarily small-scale) studies on this as well, which have also tended to find that larger functions are easier to read for debugging purposes, and that it was easier to add new functionality to the system with larger functions; but that it was easier to modify the existing functionality of the system with smaller functions. This is all in the section entitled 'Practical Effects'.
- leto_ii 6y ago
  > Fortunately, the science here me to indicate the opposite: the less code you have, the fewer defects per LoC you'll have I judged things conditional on the fact that a bug exists. If you apply the technique of counting bugs/loc blindly you may end up favouring longer, more tangled code. Think about it this way: if your goal is low bugs/loc then when a bug shows up you will be incentivized to fix it in a way that increases loc, instead maybe of simplifying thing. This will lead to fewer bugs and lower bug density.