3 ms·

> Expected difference is not an interesting statistical property, just as the mean isn't (by itself). Difference in means can meet the definition of effect siz

by closed 7y ago

> Expected difference is not an interesting statistical property, just as the mean isn't (by itself). Difference in means can meet the definition of effect size, and are listed as an example right away in the wikipedia article on it [0]! The key is that it capture the phenomena of interest (and generally, the magnitude shouldn't be a function of number of observations). In psychology, where scales often have arbitrarily defined ranges (eg IQ scales), means usually are not useful effect sizes. The case you list is a separate issue from effect size (probably the violation of distributional assumptions in some implicit or explicit model). Even using difference in means / variance (say, cohen's d), two observations that extreme would make the interpretation of both mean and variance calculations pretty dubious (a problem not solved by dividing them). https://en.wikipedia.org/wiki/Effect_size https://en.wikipedia.org/wiki/Effect_size

pron 7y ago
Which example? The six examples that talk about "difference in means" really show the difference in means divided by some measure of variance. Who would consider the same expected difference to mean the same effect size when the population distributions are narrow and when they're wide? > two observations that extreme would make the interpretation of both mean and variance calculations pretty dubious Yeah, it's a bad example, but I think it at least gives an intuition for why expected difference is not a good measure of effect. This shows two data sets with the same expected difference but one shows a large effect and one shows a small one: https://imgur.com/a/hg2NNl2 https://imgur.com/a/hg2NNl2 (the rebuttal paper also draws the expected value with variance and shows a similar situation).
- closed 7y ago
  > Which example? First paragraph: "Examples of effect sizes include the correlation between two variables,[2] the regression coefficient in a regression, the mean difference, or the ..." > Who would consider the same expected difference to mean the same effect size when the population distributions are narrow and when they're wide? For the case where the unit of measurement has a sensible, relevant interpretation (e.g. if a study measured dollar value of two interventions), and attempts to capture uncertainty via CI or another means, I would consider it one meausure of effect size. The key to understanding effect size is that its most common focus is around making measures over arbitrary scales become scale invariant. But scales are not always arbitrary. Daniel Lakens has a great article on ES and puts the motivation for calculating them well.. > First, they allow researchers to present the magnitude of the reported effects in a standardized metric which can be understood regardless of the scale that was used to measure the dependent variable. Such standardized effect sizes allow researchers to communicate the practical significance of their results (what are the practical consequences of the findings for daily life), instead of only reporting the statistical significance (how likely is the pattern of results observed in an experiment, given the assumption that there is no effect in the population). https://www.frontiersin.org/articles/10.3389/fpsyg.2013.00863/full https://www.frontiersin.org/articles/10.3389/fpsyg.2013.0086... To the degree that the thing measured has an inherently meaningful scale to the researchers (eg sometimes dollars, time), then it is already an effect size measure. There is some nuance here, since the level on which you might want to interpret something (eg the spread could be part of the value, especially to an individual who will have 1 and not many codebases). You might also want to render it comparable to other studies that measure something else, and it's unclear how to convert it to your scale, so you use standardized ES measures (eg many meta analyses). But an important point is that whether something qualifies is really a question of the scale. (What the best ES is for your specific question is another important issue!)
  - pron 7y ago
    > the mean difference Yes, but later the article shows that "mean difference" is really mean difference divided by variance. > an important point is that whether something qualifies is really a question of the scale It's one of the concerns, but I would say it's the main one. An effect size needs to distinguish between the two cases here, both having the same expected difference: https://imgur.com/a/hg2NNl2 https://imgur.com/a/hg2NNl2 If you care about units, you can talk about the expected value/difference, but that doesn't make that a meaningful effect size. What you need to do in those cases is to, at least, mention both the expected difference and the variance.
    - closed 7y ago
      > Yes, but later the article shows that "mean difference" is really mean difference divided by variance. I would say later the article shows examples of standardized effect size, so divides by variance. Whether that means effect size can't be a difference in means (and explains the wiki intro comment) I disagree with--see my Lakens comment for why. > If you care about units, you can talk about the expected value/difference, but that doesn't make that a meaningful effect size. What you need to do in those cases is to, at least, mention both the expected difference and the variance. This seems like it is begging the question. If you look at the definition and rationale for effect size, both in the wikipedia and in Lakens, a claim this strong is not there. For example, a CI over difference in means is one way to capture what a standardized effect size that uses a specific variance term might be doing (what Lakens describes as the second ES viewpoint: statistical significance, distinguishing between effect and power). In any event, thanks for discussing--it's been really helpful to think about and revisit this topic!
ramblenode 7y ago
The difference in means might be a useful measure of effect size if you are interested in comparing the results of two experiments, each with large n. It tells you what difference to expect if you repeated those experiments with new samples. If, however, you are interested in the effect on a single observation, then the variability in samples/populations needs to be taken into account.

by closed 7y ago

pron 7y ago

Which example? The six examples that talk about "difference in means" really show the difference in means divided by some measure of variance. Who would consider the same expected difference to mean the same effect size when the population distributions are narrow and when they're wide? > two observations that extreme would make the interpretation of both mean and variance calculations pretty dubious Yeah, it's a bad example, but I think it at least gives an intuition for why expected difference is not a good measure of effect. This shows two data sets with the same expected difference but one shows a large effect and one shows a small one: https://imgur.com/a/hg2NNl2 https://imgur.com/a/hg2NNl2 (the rebuttal paper also draws the expected value with variance and shows a similar situation).

closed 7y ago
> Which example? First paragraph: "Examples of effect sizes include the correlation between two variables,[2] the regression coefficient in a regression, the mean difference, or the ..." > Who would consider the same expected difference to mean the same effect size when the population distributions are narrow and when they're wide? For the case where the unit of measurement has a sensible, relevant interpretation (e.g. if a study measured dollar value of two interventions), and attempts to capture uncertainty via CI or another means, I would consider it one meausure of effect size. The key to understanding effect size is that its most common focus is around making measures over arbitrary scales become scale invariant. But scales are not always arbitrary. Daniel Lakens has a great article on ES and puts the motivation for calculating them well.. > First, they allow researchers to present the magnitude of the reported effects in a standardized metric which can be understood regardless of the scale that was used to measure the dependent variable. Such standardized effect sizes allow researchers to communicate the practical significance of their results (what are the practical consequences of the findings for daily life), instead of only reporting the statistical significance (how likely is the pattern of results observed in an experiment, given the assumption that there is no effect in the population). https://www.frontiersin.org/articles/10.3389/fpsyg.2013.00863/full https://www.frontiersin.org/articles/10.3389/fpsyg.2013.0086... To the degree that the thing measured has an inherently meaningful scale to the researchers (eg sometimes dollars, time), then it is already an effect size measure. There is some nuance here, since the level on which you might want to interpret something (eg the spread could be part of the value, especially to an individual who will have 1 and not many codebases). You might also want to render it comparable to other studies that measure something else, and it's unclear how to convert it to your scale, so you use standardized ES measures (eg many meta analyses). But an important point is that whether something qualifies is really a question of the scale. (What the best ES is for your specific question is another important issue!)
- pron 7y ago
  > the mean difference Yes, but later the article shows that "mean difference" is really mean difference divided by variance. > an important point is that whether something qualifies is really a question of the scale It's one of the concerns, but I would say it's the main one. An effect size needs to distinguish between the two cases here, both having the same expected difference: https://imgur.com/a/hg2NNl2 https://imgur.com/a/hg2NNl2 If you care about units, you can talk about the expected value/difference, but that doesn't make that a meaningful effect size. What you need to do in those cases is to, at least, mention both the expected difference and the variance.
  - closed 7y ago
    > Yes, but later the article shows that "mean difference" is really mean difference divided by variance. I would say later the article shows examples of standardized effect size, so divides by variance. Whether that means effect size can't be a difference in means (and explains the wiki intro comment) I disagree with--see my Lakens comment for why. > If you care about units, you can talk about the expected value/difference, but that doesn't make that a meaningful effect size. What you need to do in those cases is to, at least, mention both the expected difference and the variance. This seems like it is begging the question. If you look at the definition and rationale for effect size, both in the wikipedia and in Lakens, a claim this strong is not there. For example, a CI over difference in means is one way to capture what a standardized effect size that uses a specific variance term might be doing (what Lakens describes as the second ES viewpoint: statistical significance, distinguishing between effect and power). In any event, thanks for discussing--it's been really helpful to think about and revisit this topic!

ramblenode 7y ago

The difference in means might be a useful measure of effect size if you are interested in comparing the results of two experiments, each with large n. It tells you what difference to expect if you repeated those experiments with new samples. If, however, you are interested in the effect on a single observation, then the variability in samples/populations needs to be taken into account.