4 ms·

Not surprised to hear this at all. I can remember being specifically told during both undergrad and grad research seminars 'It is highly unlikely that you will

Not surprised to hear this at all. I can remember being specifically told during both undergrad and grad research seminars 'It is highly unlikely that you will find a large number of statistically significant results. The sample sizes are just too small'. In social sciences, this is an astoundingly common problem. Then, of course, when I have to defend my comp. exam (for a second time) in front of the faculty, I spent the last 5 minutes lamenting the very phenomenon you described. Research can be deeply engrossing, but it doesn't mean shit when you go out into the world looking to land your next job. Of my 3 reviewers, only 1 had spent significant time in industry before teaching so, think of this as a wake up call for people to get outside the bubble and broaden their horizons once in a while. Tl;dr The program I was (Communications) in changed their requirements right after I left to eliminate this distinction and now students are required to develop and implement an idea (Marketing campaign, Training seminar, Comms. Audit w/ recommendations, etc.). You might get, 'that's nice' or even 'interesting', but this is usually followed up with 'how can you help me solve problems x and y right now on deadline?'

nonbel 8y ago
>"'It is highly unlikely that you will find a large number of statistically significant results. The sample sizes are just too small'." This is just a tautology. In social science large sample size = statistical significance (there are only false negatives since the null model being tested is always wrong).
- chrisfinazzo 8y ago
  If this is true, how does that work if the fundamental conclusion re the relationship of 2 entities is the inverse of what you expect? To me, that result doesn't call for adding participants to get to an acceptable p value, it means reevaluate your assumptions, because your basis of understanding is wrong. @epistasis' using the term 'p-hack' is incredibly apt in this case. The ability for a study to be deemed "correct" in proving a hypothesis might be true of small samples if you are sufficiently rigorous in designing the experiment, but it could still fall apart at larger sizes - which only proves that part of your hypothesis was not statistically meaningful regardless of the conclusion you reached. Again, a thorough discussion of limitations around any research should address this. E.g, "Our findings indicate x, but there is no way to know without further study if y is actually true instead"
  - nonbel 8y ago
    >"If this is true, how does that work if the fundamental conclusion re the relationship of 2 entities is the inverse of what you expect?" Not quite sure what you are trying to say but this doesnt sound like any statistical model I've seen used in social science. Usually they assume some distribution (eg normal, or "t") and then assume both samples are taken from that distribution. From that they derive the proportion of times you would see the observed difference in means between samples.
    - chrisfinazzo 8y ago
      It's really the exceptional case I'm thinking of here. Basically, you screw up such that the data you've collected is meaningless. Sure, you could do the statistical analysis correctly and get a result, but it ends up being the opposite not because the relationship is actually different, but because some part of your design is flawed. In that case, you would never end up with a correct result. (This is of course the worst possible outcome, but I would be remiss if I didn't mention it here.) Although I have no background in it, I suppose the same could happen in the other sciences if there are side effects that you don't anticipate and don't measure accurately enough. ^ This again is why reproducibility is important. Earlier in this thread, @majidazmi mentioned that a major focus of their thesis was to prove that a prior experiment was incorrect in its findings.

nonbel 8y ago

>"'It is highly unlikely that you will find a large number of statistically significant results. The sample sizes are just too small'." This is just a tautology. In social science large sample size = statistical significance (there are only false negatives since the null model being tested is always wrong).

chrisfinazzo 8y ago
If this is true, how does that work if the fundamental conclusion re the relationship of 2 entities is the inverse of what you expect? To me, that result doesn't call for adding participants to get to an acceptable p value, it means reevaluate your assumptions, because your basis of understanding is wrong. @epistasis' using the term 'p-hack' is incredibly apt in this case. The ability for a study to be deemed "correct" in proving a hypothesis might be true of small samples if you are sufficiently rigorous in designing the experiment, but it could still fall apart at larger sizes - which only proves that part of your hypothesis was not statistically meaningful regardless of the conclusion you reached. Again, a thorough discussion of limitations around any research should address this. E.g, "Our findings indicate x, but there is no way to know without further study if y is actually true instead"
- nonbel 8y ago
  >"If this is true, how does that work if the fundamental conclusion re the relationship of 2 entities is the inverse of what you expect?" Not quite sure what you are trying to say but this doesnt sound like any statistical model I've seen used in social science. Usually they assume some distribution (eg normal, or "t") and then assume both samples are taken from that distribution. From that they derive the proportion of times you would see the observed difference in means between samples.
  - chrisfinazzo 8y ago
    It's really the exceptional case I'm thinking of here. Basically, you screw up such that the data you've collected is meaningless. Sure, you could do the statistical analysis correctly and get a result, but it ends up being the opposite not because the relationship is actually different, but because some part of your design is flawed. In that case, you would never end up with a correct result. (This is of course the worst possible outcome, but I would be remiss if I didn't mention it here.) Although I have no background in it, I suppose the same could happen in the other sciences if there are side effects that you don't anticipate and don't measure accurately enough. ^ This again is why reproducibility is important. Earlier in this thread, @majidazmi mentioned that a major focus of their thesis was to prove that a prior experiment was incorrect in its findings.