6 ms·

The issue really surfaces when you dissect "statistical training". I had a conversation with a colleague getting the a PhD and about to publish a paper. She was

The issue really surfaces when you dissect "statistical training". I had a conversation with a colleague getting the a PhD and about to publish a paper. She was mentioning to me that she tried "a bunch" of different specifications and the one that was significant was [I forget what the variables were, but this is in the education field]. I pointed out to her that, since she tried a bunch of specifications and chose the one which yielded the "positive" result, wouldn't that fundamentally alter the interpretation of the p-value, and possibly invalidate her claim? She looked at me like I was a moron, uneducated, and trying to be difficult. I'm forever indebted to my econometrics professor who helped me build a tremendous mental model about how to think about these issues. But between the really poor training (most people I encounter in the field just run STATA/SPSS commands with limited/no understanding of what they do mathematically) and the terrible incentives facing researchers (you don't see many negative results published, do you), it's no surprise that when people come in and attempt to seriously replicate even foundational findings, they often come away disappointed (thinking about the Reproducibility project - https://en.wikipedia.org/wiki/Reproducibility_Project https://en.wikipedia.org/wiki/Reproducibility_Project) I, too, don't have an answer to this. But I can only imagine that the way to prevent is to push the industry in the direction that truth-seeking, not novelty, is rewarded. I've mentioned this before here, but I know professors who established their entire careers on the backs of research which was later completely refuted -- even in cases where their actions where fradulent. And they are still teaching -- and successful. As academics - like media and enveryone else -- are chasing our ever-dimming attention spans, I think this'll be a really tough nut to meaningfully crack

nerdponx 9y ago
There's really no excuse for p-value fishing when procedures for controlling false-discovery and familywise error rates exist and can be implemented in a single line of code in R [0]. 0: https://stats.stackexchange.com/q/63441/36229 https://stats.stackexchange.com/q/63441/36229
nabla9 9y ago
It's really sad because there are more tools and methods to squeeze statistically meaningful results from data than before. Even if the researchers could use these new tools, in many areas peer reviewers are not equipped to understand them. Maybe its the time to divide research into different specialties. Main subject area research team must always consult statistical experts for model selection, experiment design, and interpretation of the results. Time when one group can do all these things alone may be over.
- tnecniv 9y ago
  This already happens. It's not uncommon for a grant for, say, something in medicine to include as a budget item funds to hire a biostats person.
ryanmonroe 9y ago
It seems like you're using this example to imply statistical education is likely to blind people to obvious logical problems with their interpretation of evidence. If that's the case it isn't a very good example since multiple comparisons is a standard issue commonly covered in a first or second course in statistics.
ams6110 9y ago
When you've invested 4 or more years in Ph.D. research, there are powerful motivations to "find" results that support your thesis. Perhaps they are subconscious, but they are powerful.
- nonbel 9y ago
  The people who end up like that are the exact people who the PhD process is supposed to filter out...
  - opportune 9y ago
    No university or professor wants to be known for having any sizable number of their PhD candidates filtered out in the middle of the process. They'll delay poor performers for sure, but it just looks really bad when their candidates essentially fail out. Also consider that often it's not even the candidate's fault that their research went nowhere. Sometimes it's just due to chance that a particular path became a dead-end, but other times it's due to a professor obstinately pushing them to focus on something unproductive, perhaps well after it's almost entirely evident that the dead end is coming. It's a messed up system. I think the entire field of academic research needs a complete overhaul.
    - nonbel 9y ago
      Yes, the current system rewards that behavior and as a side effect rewards misunderstanding stats, etc. The people who have no qualms about "statistically significant" = "my theory is correct" (either due to maliciousness or ignorance) will graduate much sooner and be able to pump out more papers.
  - Klockan 9y ago
    Fake it till you make it significant (p < 0.05).
- fredophile 9y ago
  I think the real problem is that positive results are seen as better than negative results. Running a bunch of studies and showing that your theory doesn't work is just as much work as proving that it does work and is still a significant contribution.
Chris2048 9y ago
> I'm forever indebted to my econometrics professor who helped me build a tremendous mental model about how to think about these issues Can you tell me what that is? Stats/probability is fairly unintuitive to me, but I can't help thinking there is a better way of framing it. Like a 'probability' is often stated like it where a concrete thing, when it's really more of a 'best guess' based on the evidence, possibly with in an in-built assumption that the most conceptually likely thing (itself a 'best guess') did indeed happen, and inferring events (the most likely sequence of events) that led to observed results.. This is quite hard to unpack, and reason about in the same way as 'logic', which many minds seems to gravitate to...

nerdponx 9y ago

There's really no excuse for p-value fishing when procedures for controlling false-discovery and familywise error rates exist and can be implemented in a single line of code in R [0]. 0: https://stats.stackexchange.com/q/63441/36229 https://stats.stackexchange.com/q/63441/36229

nabla9 9y ago

It's really sad because there are more tools and methods to squeeze statistically meaningful results from data than before. Even if the researchers could use these new tools, in many areas peer reviewers are not equipped to understand them. Maybe its the time to divide research into different specialties. Main subject area research team must always consult statistical experts for model selection, experiment design, and interpretation of the results. Time when one group can do all these things alone may be over.

tnecniv 9y ago
This already happens. It's not uncommon for a grant for, say, something in medicine to include as a budget item funds to hire a biostats person.

ryanmonroe 9y ago

It seems like you're using this example to imply statistical education is likely to blind people to obvious logical problems with their interpretation of evidence. If that's the case it isn't a very good example since multiple comparisons is a standard issue commonly covered in a first or second course in statistics.

ams6110 9y ago

When you've invested 4 or more years in Ph.D. research, there are powerful motivations to "find" results that support your thesis. Perhaps they are subconscious, but they are powerful.

nonbel 9y ago
The people who end up like that are the exact people who the PhD process is supposed to filter out...
- opportune 9y ago
  No university or professor wants to be known for having any sizable number of their PhD candidates filtered out in the middle of the process. They'll delay poor performers for sure, but it just looks really bad when their candidates essentially fail out. Also consider that often it's not even the candidate's fault that their research went nowhere. Sometimes it's just due to chance that a particular path became a dead-end, but other times it's due to a professor obstinately pushing them to focus on something unproductive, perhaps well after it's almost entirely evident that the dead end is coming. It's a messed up system. I think the entire field of academic research needs a complete overhaul.
  - nonbel 9y ago
    Yes, the current system rewards that behavior and as a side effect rewards misunderstanding stats, etc. The people who have no qualms about "statistically significant" = "my theory is correct" (either due to maliciousness or ignorance) will graduate much sooner and be able to pump out more papers.
- Klockan 9y ago
  Fake it till you make it significant (p < 0.05).
fredophile 9y ago
I think the real problem is that positive results are seen as better than negative results. Running a bunch of studies and showing that your theory doesn't work is just as much work as proving that it does work and is still a significant contribution.

Chris2048 9y ago

> I'm forever indebted to my econometrics professor who helped me build a tremendous mental model about how to think about these issues Can you tell me what that is? Stats/probability is fairly unintuitive to me, but I can't help thinking there is a better way of framing it. Like a 'probability' is often stated like it where a concrete thing, when it's really more of a 'best guess' based on the evidence, possibly with in an in-built assumption that the most conceptually likely thing (itself a 'best guess') did indeed happen, and inferring events (the most likely sequence of events) that led to observed results.. This is quite hard to unpack, and reason about in the same way as 'logic', which many minds seems to gravitate to...