4 ms·

>"'It is highly unlikely that you will find a large number of statistically significant results. The sample sizes are just too small'." This is just a tautolog

by nonbel 8y ago

>"'It is highly unlikely that you will find a large number of statistically significant results. The sample sizes are just too small'." This is just a tautology. In social science large sample size = statistical significance (there are only false negatives since the null model being tested is always wrong).

chrisfinazzo 8y ago
If this is true, how does that work if the fundamental conclusion re the relationship of 2 entities is the inverse of what you expect? To me, that result doesn't call for adding participants to get to an acceptable p value, it means reevaluate your assumptions, because your basis of understanding is wrong. @epistasis' using the term 'p-hack' is incredibly apt in this case. The ability for a study to be deemed "correct" in proving a hypothesis might be true of small samples if you are sufficiently rigorous in designing the experiment, but it could still fall apart at larger sizes - which only proves that part of your hypothesis was not statistically meaningful regardless of the conclusion you reached. Again, a thorough discussion of limitations around any research should address this. E.g, "Our findings indicate x, but there is no way to know without further study if y is actually true instead"
- nonbel 8y ago
  >"If this is true, how does that work if the fundamental conclusion re the relationship of 2 entities is the inverse of what you expect?" Not quite sure what you are trying to say but this doesnt sound like any statistical model I've seen used in social science. Usually they assume some distribution (eg normal, or "t") and then assume both samples are taken from that distribution. From that they derive the proportion of times you would see the observed difference in means between samples.
  - chrisfinazzo 8y ago
    It's really the exceptional case I'm thinking of here. Basically, you screw up such that the data you've collected is meaningless. Sure, you could do the statistical analysis correctly and get a result, but it ends up being the opposite not because the relationship is actually different, but because some part of your design is flawed. In that case, you would never end up with a correct result. (This is of course the worst possible outcome, but I would be remiss if I didn't mention it here.) Although I have no background in it, I suppose the same could happen in the other sciences if there are side effects that you don't anticipate and don't measure accurately enough. ^ This again is why reproducibility is important. Earlier in this thread, @majidazmi mentioned that a major focus of their thesis was to prove that a prior experiment was incorrect in its findings.

by nonbel 8y ago

chrisfinazzo 8y ago

If this is true, how does that work if the fundamental conclusion re the relationship of 2 entities is the inverse of what you expect? To me, that result doesn't call for adding participants to get to an acceptable p value, it means reevaluate your assumptions, because your basis of understanding is wrong. @epistasis' using the term 'p-hack' is incredibly apt in this case. The ability for a study to be deemed "correct" in proving a hypothesis might be true of small samples if you are sufficiently rigorous in designing the experiment, but it could still fall apart at larger sizes - which only proves that part of your hypothesis was not statistically meaningful regardless of the conclusion you reached. Again, a thorough discussion of limitations around any research should address this. E.g, "Our findings indicate x, but there is no way to know without further study if y is actually true instead"

nonbel 8y ago
>"If this is true, how does that work if the fundamental conclusion re the relationship of 2 entities is the inverse of what you expect?" Not quite sure what you are trying to say but this doesnt sound like any statistical model I've seen used in social science. Usually they assume some distribution (eg normal, or "t") and then assume both samples are taken from that distribution. From that they derive the proportion of times you would see the observed difference in means between samples.
- chrisfinazzo 8y ago
  It's really the exceptional case I'm thinking of here. Basically, you screw up such that the data you've collected is meaningless. Sure, you could do the statistical analysis correctly and get a result, but it ends up being the opposite not because the relationship is actually different, but because some part of your design is flawed. In that case, you would never end up with a correct result. (This is of course the worst possible outcome, but I would be remiss if I didn't mention it here.) Although I have no background in it, I suppose the same could happen in the other sciences if there are side effects that you don't anticipate and don't measure accurately enough. ^ This again is why reproducibility is important. Earlier in this thread, @majidazmi mentioned that a major focus of their thesis was to prove that a prior experiment was incorrect in its findings.