4 ms·

part of this has to do with the fact that our intuitive sense of effect sizes don't really use proportions and we subconsciously start including sample sizes.

part of this has to do with the fact that our intuitive sense of effect sizes don't really use proportions and we subconsciously start including sample sizes. If a state had an election with millions of voters and got a 55-45 result, it would be a decisive landslide victory; If a elementary school classroom had an election 20 voters had got a 55-45 split, it's be the narrowest possible margin of victory. Most would likely say that the effect in the former 'feels' much larger even though proportions are identical, which suggests that under the hood we're factoring in sample size to our intuition about effect sizes (probably something chi-square-ish). The result is that the framing of the problem can change our sense of how big the effects are. When we hear that these are state-level elections, we think it's a huge effect and feel that we should be able to do reverse inference. If it was reframed as an election on a much smaller sample, the paradox disappears and you'd say "of course you wouldn't be able to reverse that inference"

thaumasiotes 2y ago
This ties in to the difference between whether an effect is statistically significant ("does the effect exist?") and whether it's significant ("does the effect matter?"). It's very common to confuse the two ideas. In particular, in an election with many millions of votes and a 55-45 margin, it's common to describe the winner as receiving a mandate to rule, because it was so easy to determine who the winner was, despite the fact that they appear to be extremely unpopular. That's not a mandate in any ordinary sense.
- fjkdlsjflkds 2y ago
  > whether an effect is statistically significant ("does the effect exist?") and whether it's significant ("does the effect matter?") The more accurate terms to describe this is whether an effect is significant (i.e., "do we have enough information to be able to claim it is different from zero?") vs. whether an effect is relevant (i.e., "is [our estimate of] the size of the effect meaningfully different from zero?"). Statistics can only address the first issue (effect significance); the second issue (effect relevance) requires domain knowledge beyond statistics (in some contexts, a difference of 0.01 units can be irrelevant, while it may be relevant in other contexts).

thaumasiotes 2y ago

This ties in to the difference between whether an effect is statistically significant ("does the effect exist?") and whether it's significant ("does the effect matter?"). It's very common to confuse the two ideas. In particular, in an election with many millions of votes and a 55-45 margin, it's common to describe the winner as receiving a mandate to rule, because it was so easy to determine who the winner was, despite the fact that they appear to be extremely unpopular. That's not a mandate in any ordinary sense.

fjkdlsjflkds 2y ago
> whether an effect is statistically significant ("does the effect exist?") and whether it's significant ("does the effect matter?") The more accurate terms to describe this is whether an effect is significant (i.e., "do we have enough information to be able to claim it is different from zero?") vs. whether an effect is relevant (i.e., "is [our estimate of] the size of the effect meaningfully different from zero?"). Statistics can only address the first issue (effect significance); the second issue (effect relevance) requires domain knowledge beyond statistics (in some contexts, a difference of 0.01 units can be irrelevant, while it may be relevant in other contexts).