3 ms·

Why most published research findings are false: https://en.m.wikipedia.org/wiki/Why_Most_Published_Research_Findings_Are_False https://en.m.wikipedia.org/wiki/

Why most published research findings are false: https://en.m.wikipedia.org/wiki/Why_Most_Published_Research_Findings_Are_False https://en.m.wikipedia.org/wiki/Why_Most_Published_Research_...

ChrisLomont 5y ago
That study has been debunked. For example, in it he guesses at some factors fundamental to his thesis. Later studies to empirically measure those factors found his guesses to be wildly incorrect, and correcting for the empirical data, Ioannis claims are found to be unsupportable.
- hamilyon2 5y ago
  Ironic, isn't it?
- bigfudge 5y ago
  I'm not aware of those studies related to the Ioannides paper, but the story on replication in psychology and specifically social psychology is not at all pretty. My sense is that there are research domains that are functional, but others (social, clinical psychology) which are not at all. Given the level of motivated cognition and sloppy thinking evident in decades of psychiatric and other research into cannabis use, I wouldn't be at all surprised to find low standards in studies of this type. I'm not saying that's the case, but it's certainly my prior when reading any dietary studies, or studies of "bad effect x" of recreational drugs. In general, when publishing any study like this the research should be contextualised properly. For example, it would make sense to state the effect size in relation to the risk of drinking 10% more than the recommended units of alcohol, or smoking 20 cigarettes per day.
- throwanem 5y ago
  Debunked where, by whom? Not that Ioannidis isn't a publicity hound of the first water, but if nothing else it'd be useful to have cites for the next time this comes up.
  - ChrisLomont 5y ago
    I posted 2 citations in this thread. Or simply read his paper, and you can see how it doesn't prove without fail what it claims - it relies on numbers he made up - later groups measured these numbers and found the made up ones invalid.
- buescher 5y ago
  A fair criticism of Ionnidis would be "statisticians always knew this about every field that reports statistical significance with p-values, but no one was going to listen, I don't know what that guy's got". I'm not aware of any substantive "debunking" of his famous paper, which is not a "study".
  - ChrisLomont 5y ago
    You are correct - it was not a study, since that would require studying some empirical data instead of making up values. It was actually an essay, akin to a guest blog post, but for a journal [0]. Look at his paper - it clearly relies on values that he made up - there is not a single measurement of actual papers to fill in his formulas. What he did was show it was possible for stuff to be wrong, given values he pulled out the air. When people did studies to measure those values from real papers, which Ioannidis did not, they found his made up values far from reality. For example, this paper shows how to do actual measurements of the values Ioannidis made up [1]. Here's one [2] (of many) paper showing that Ioannidis did not prove the claim people think he did for other reasons. Carefully read the actual original paper, and note that there are many values to input to his model, that he hand waves over and does not actually measure. [0] https://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.0020124&xid=17259,15700019,15700186,15700190,15700248 https://journals.plos.org/plosmedicine/article?id=10.1371/jo... [1] https://arxiv.org/ftp/arxiv/papers/1301/1301.3718.pdf https://arxiv.org/ftp/arxiv/papers/1301/1301.3718.pdf [2] https://econpapers.repec.org/article/plopmed00/0040168.htm https://econpapers.repec.org/article/plopmed00/0040168.htm
    - buescher 5y ago
      Yeah, the papers you cite both basically start "Look, we know you all abuse the heck out of p-values and you really should stop (wink wink nudge nudge) doing that" and go on to "gee, if you start with much more optimistic assumptions than Ioannidis, then you don't come to such rhetorically devastating conclusions". Of course, he's a very bad man for pointing all this out, isn't he?
      - 5y ago

Why most published research findings are false: https://en.m.wikipedia.org/wiki/Why_Most_Published_Research_Findings_Are_False https://en.m.wikipedia.org/wiki/Why_Most_Published_Research_...

ChrisLomont 5y ago

That study has been debunked. For example, in it he guesses at some factors fundamental to his thesis. Later studies to empirically measure those factors found his guesses to be wildly incorrect, and correcting for the empirical data, Ioannis claims are found to be unsupportable.

hamilyon2 5y ago
Ironic, isn't it?
bigfudge 5y ago
I'm not aware of those studies related to the Ioannides paper, but the story on replication in psychology and specifically social psychology is not at all pretty. My sense is that there are research domains that are functional, but others (social, clinical psychology) which are not at all. Given the level of motivated cognition and sloppy thinking evident in decades of psychiatric and other research into cannabis use, I wouldn't be at all surprised to find low standards in studies of this type. I'm not saying that's the case, but it's certainly my prior when reading any dietary studies, or studies of "bad effect x" of recreational drugs. In general, when publishing any study like this the research should be contextualised properly. For example, it would make sense to state the effect size in relation to the risk of drinking 10% more than the recommended units of alcohol, or smoking 20 cigarettes per day.
throwanem 5y ago
Debunked where, by whom? Not that Ioannidis isn't a publicity hound of the first water, but if nothing else it'd be useful to have cites for the next time this comes up.
- ChrisLomont 5y ago
  I posted 2 citations in this thread. Or simply read his paper, and you can see how it doesn't prove without fail what it claims - it relies on numbers he made up - later groups measured these numbers and found the made up ones invalid.
buescher 5y ago
A fair criticism of Ionnidis would be "statisticians always knew this about every field that reports statistical significance with p-values, but no one was going to listen, I don't know what that guy's got". I'm not aware of any substantive "debunking" of his famous paper, which is not a "study".
- ChrisLomont 5y ago
  You are correct - it was not a study, since that would require studying some empirical data instead of making up values. It was actually an essay, akin to a guest blog post, but for a journal [0]. Look at his paper - it clearly relies on values that he made up - there is not a single measurement of actual papers to fill in his formulas. What he did was show it was possible for stuff to be wrong, given values he pulled out the air. When people did studies to measure those values from real papers, which Ioannidis did not, they found his made up values far from reality. For example, this paper shows how to do actual measurements of the values Ioannidis made up [1]. Here's one [2] (of many) paper showing that Ioannidis did not prove the claim people think he did for other reasons. Carefully read the actual original paper, and note that there are many values to input to his model, that he hand waves over and does not actually measure. [0] https://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.0020124&xid=17259,15700019,15700186,15700190,15700248 https://journals.plos.org/plosmedicine/article?id=10.1371/jo... [1] https://arxiv.org/ftp/arxiv/papers/1301/1301.3718.pdf https://arxiv.org/ftp/arxiv/papers/1301/1301.3718.pdf [2] https://econpapers.repec.org/article/plopmed00/0040168.htm https://econpapers.repec.org/article/plopmed00/0040168.htm
  - buescher 5y ago
    Yeah, the papers you cite both basically start "Look, we know you all abuse the heck out of p-values and you really should stop (wink wink nudge nudge) doing that" and go on to "gee, if you start with much more optimistic assumptions than Ioannidis, then you don't come to such rhetorically devastating conclusions". Of course, he's a very bad man for pointing all this out, isn't he?
    - 5y ago