4 ms·

This is a truly terrific post, but it says: > They’re underselling the effect here. While the dominant factor is number of commits, language still matters a lo

by pron 7y ago

This is a truly terrific post, but it says: > They’re underselling the effect here. While the dominant factor is number of commits, language still matters a lot, too. If you choose C++ over TypeScript, you’re going to have twice as many DFCs! That doesn’t necessarily mean twice as many bugs, but it is suggestive. Further, while they say the effect is “overwhelmingly dominated by […] project size, team size, and commit size”, that doesn’t actually bear out. Only the number of commits is a bigger factor in language choice. This is inaccurate. "Effect" is not the expected difference (i.e. difference between the means) but, roughly, the expected difference divided by variance. Just looking at the expected difference is insufficient to determine if the effect is large (and undersold) or small. Expected difference is not an interesting statistical property, just as the mean isn't (by itself). If you're looking at ten Clojure projects and ten C++ projects, and all ten Clojure projects have 10 DFCs, while eight C++ projects have 8 DFCs and two have 500, then the expected difference is huge, but the effect is small. Indeed, when looking at the variance, Clojure, the "best"-performing language in this dataset, and C++, the "worst"-performing language in this dataset, the two were not very distinguishable, supporting everyone's finding of a very small effect.

afandian 7y ago
DFC = "Defect-Fixing Commits"
closed 7y ago
> Expected difference is not an interesting statistical property, just as the mean isn't (by itself). Difference in means can meet the definition of effect size, and are listed as an example right away in the wikipedia article on it [0]! The key is that it capture the phenomena of interest (and generally, the magnitude shouldn't be a function of number of observations). In psychology, where scales often have arbitrarily defined ranges (eg IQ scales), means usually are not useful effect sizes. The case you list is a separate issue from effect size (probably the violation of distributional assumptions in some implicit or explicit model). Even using difference in means / variance (say, cohen's d), two observations that extreme would make the interpretation of both mean and variance calculations pretty dubious (a problem not solved by dividing them). https://en.wikipedia.org/wiki/Effect_size https://en.wikipedia.org/wiki/Effect_size
- pron 7y ago
  Which example? The six examples that talk about "difference in means" really show the difference in means divided by some measure of variance. Who would consider the same expected difference to mean the same effect size when the population distributions are narrow and when they're wide? > two observations that extreme would make the interpretation of both mean and variance calculations pretty dubious Yeah, it's a bad example, but I think it at least gives an intuition for why expected difference is not a good measure of effect. This shows two data sets with the same expected difference but one shows a large effect and one shows a small one: https://imgur.com/a/hg2NNl2 https://imgur.com/a/hg2NNl2 (the rebuttal paper also draws the expected value with variance and shows a similar situation).
  - closed 7y ago
    > Which example? First paragraph: "Examples of effect sizes include the correlation between two variables,[2] the regression coefficient in a regression, the mean difference, or the ..." > Who would consider the same expected difference to mean the same effect size when the population distributions are narrow and when they're wide? For the case where the unit of measurement has a sensible, relevant interpretation (e.g. if a study measured dollar value of two interventions), and attempts to capture uncertainty via CI or another means, I would consider it one meausure of effect size. The key to understanding effect size is that its most common focus is around making measures over arbitrary scales become scale invariant. But scales are not always arbitrary. Daniel Lakens has a great article on ES and puts the motivation for calculating them well.. > First, they allow researchers to present the magnitude of the reported effects in a standardized metric which can be understood regardless of the scale that was used to measure the dependent variable. Such standardized effect sizes allow researchers to communicate the practical significance of their results (what are the practical consequences of the findings for daily life), instead of only reporting the statistical significance (how likely is the pattern of results observed in an experiment, given the assumption that there is no effect in the population). https://www.frontiersin.org/articles/10.3389/fpsyg.2013.00863/full https://www.frontiersin.org/articles/10.3389/fpsyg.2013.0086... To the degree that the thing measured has an inherently meaningful scale to the researchers (eg sometimes dollars, time), then it is already an effect size measure. There is some nuance here, since the level on which you might want to interpret something (eg the spread could be part of the value, especially to an individual who will have 1 and not many codebases). You might also want to render it comparable to other studies that measure something else, and it's unclear how to convert it to your scale, so you use standardized ES measures (eg many meta analyses). But an important point is that whether something qualifies is really a question of the scale. (What the best ES is for your specific question is another important issue!)

by pron 7y ago

afandian 7y ago

DFC = "Defect-Fixing Commits"

closed 7y ago

> Expected difference is not an interesting statistical property, just as the mean isn't (by itself). Difference in means can meet the definition of effect size, and are listed as an example right away in the wikipedia article on it [0]! The key is that it capture the phenomena of interest (and generally, the magnitude shouldn't be a function of number of observations). In psychology, where scales often have arbitrarily defined ranges (eg IQ scales), means usually are not useful effect sizes. The case you list is a separate issue from effect size (probably the violation of distributional assumptions in some implicit or explicit model). Even using difference in means / variance (say, cohen's d), two observations that extreme would make the interpretation of both mean and variance calculations pretty dubious (a problem not solved by dividing them). https://en.wikipedia.org/wiki/Effect_size https://en.wikipedia.org/wiki/Effect_size

pron 7y ago
Which example? The six examples that talk about "difference in means" really show the difference in means divided by some measure of variance. Who would consider the same expected difference to mean the same effect size when the population distributions are narrow and when they're wide? > two observations that extreme would make the interpretation of both mean and variance calculations pretty dubious Yeah, it's a bad example, but I think it at least gives an intuition for why expected difference is not a good measure of effect. This shows two data sets with the same expected difference but one shows a large effect and one shows a small one: https://imgur.com/a/hg2NNl2 https://imgur.com/a/hg2NNl2 (the rebuttal paper also draws the expected value with variance and shows a similar situation).
- closed 7y ago
  > Which example? First paragraph: "Examples of effect sizes include the correlation between two variables,[2] the regression coefficient in a regression, the mean difference, or the ..." > Who would consider the same expected difference to mean the same effect size when the population distributions are narrow and when they're wide? For the case where the unit of measurement has a sensible, relevant interpretation (e.g. if a study measured dollar value of two interventions), and attempts to capture uncertainty via CI or another means, I would consider it one meausure of effect size. The key to understanding effect size is that its most common focus is around making measures over arbitrary scales become scale invariant. But scales are not always arbitrary. Daniel Lakens has a great article on ES and puts the motivation for calculating them well.. > First, they allow researchers to present the magnitude of the reported effects in a standardized metric which can be understood regardless of the scale that was used to measure the dependent variable. Such standardized effect sizes allow researchers to communicate the practical significance of their results (what are the practical consequences of the findings for daily life), instead of only reporting the statistical significance (how likely is the pattern of results observed in an experiment, given the assumption that there is no effect in the population). https://www.frontiersin.org/articles/10.3389/fpsyg.2013.00863/full https://www.frontiersin.org/articles/10.3389/fpsyg.2013.0086... To the degree that the thing measured has an inherently meaningful scale to the researchers (eg sometimes dollars, time), then it is already an effect size measure. There is some nuance here, since the level on which you might want to interpret something (eg the spread could be part of the value, especially to an individual who will have 1 and not many codebases). You might also want to render it comparable to other studies that measure something else, and it's unclear how to convert it to your scale, so you use standardized ES measures (eg many meta analyses). But an important point is that whether something qualifies is really a question of the scale. (What the best ES is for your specific question is another important issue!)