5 ms·

> You'll find plenty of information if you actually seek it out rather than simply making a knee-jerk, snarky comment, but here is one example article which art

by aab0 10y ago

> You'll find plenty of information if you actually seek it out rather than simply making a knee-jerk, snarky comment, but here is one example article which articulates some of the issues that have been under consideration in recent years: http://m.ije.oxfordjournals.org/content/41/1/273.full http://m.ije.oxfordjournals.org/content/41/1/273.full I assume you are referring to " If the seven associations that did not reach P ≤ 5 × 10−8 when additional data were considered are assumed to have been false-positives, the false-discovery rate for borderline associations is estimated to be 27% [95% confidence interval (CI) 12–48%]. For five associations, the current P-value is > 10−6 [corresponding false-discovery rate 19% (95% CI 7–39%)]." That doesn't show anything relevant. Failure to replicate at 10-8 is a ludicrous way to define non-replication; to paraphrase Cohen, surely God loves the 10-7.99 almost as much as the 10-8... This paper needs to adjust for power, and ask how many hits one would expect to not replicate at 10-8 given the power of the replicating studies. If you do remember power, GWASes replicate fantastically, for example https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3681663/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3681663/ "Replicability rates are high within Europeans, with 155 successful out of 181 attempts (85.6%), when only 9 positive replications (∼5%) would be expected under the null hypothesis of no association (binomial test, P<10−16). This excess was robust to the significance threshold (e.g. 122 observed vs. 0.18 expected if only replication attempts achieving P<0.001 are considered successful and 56 observed vs. 1.8×10−5 expected for a threshold of P<10−7, Table S5). Moreover, replicability rates within Europeans approach 100% when accounting for statistical power. For the 168 attempts for which we could calculate the power to replicate the original finding (Table S5), we observed 147 positive replications, which is almost identical to the expectation of 149.1 positive replications given that average power is 89.1% (see Materials and Methods). This is expected, since most GWAS already contain an internal replication phase [1], [24]." > and, believe me, many GWA studies have been published with much less significant p-values I don't think they have. Ever since Ioannidis and others demonstrated what a total debacle the early candidate-gene studies were around 2009-2011, using the first GWASes to demonstrate that, GWASes have been pretty standardly done at 10-8.

erdevs 10y ago
> > and, believe me, many GWA studies have been published with much less significant p-values > I don't think they have. Ever since Ioannidis and others demonstrated what a total debacle the early candidate-gene studies were around 2009-2011 Yes, pre-Ioannidis is the period I was referring to. Things cleaned up a lot in 2012+. What you said does not conflict with what I said... the field, especially early on, has published some spurious results. It seems like you're simultaneously saying "I don't think they have [published low-quality results]" and then immediately admitting what a "debacle" early studies sometimes were. > That doesn't show anything relevant. Failure to replicate at 10-8 is a ludicrous way to define non-replication This is a silly statement. Doesn't the significance level being "ludicrous" depend on things like the degree of multiple testing happening? Obviously, yes. The reason why the field (not just this one paper) has pushed for significance in the 10^-7 to 10^-8 range is partially for this reason. So... are you questioning the entire field's movement over the past few years? If so, on what basis? As high-throughput, low-cost full genome sequencing begins to replace SNP-based techniques, GWAS will have to wrestle with this issue even more. I'm not sure what exactly you're debating me on here. I'm not saying anything controversial in the field. Again, even the wikipedia article cites well-known studies and quotations from respected sources in the literature, including "Particularly the statistical issue of multiple testing wherein it has been noted that "the GWA approach can be problematic because the massive number of statistical tests performed presents an unprecedented potential for false-positive results"... which is what I originally pointed out. This is an issue the field has struggled with from the get go. It's matured and is much better now (as I've noted), but the field still struggles with the issue. And there are still low-quality papers being published. I also gave this particular paper praise for holding to a higher standard than some other GWA studies.
- aab0 10y ago
  > Yes, pre-Ioannidis is the period I was referring to. Things cleaned up a lot in 2012+. What you said does not conflict with what I said.. A candidate-gene study != GWAS. It's particularly bizarre to criticize GWASes for the sins of candidate-gene study when GWASes were literally partially designed to avoid those problems. Don't equivocate. If you have criticisms of actual GWASes as they are run and good reason to doubt that the hits are noise and will not replicate in well-powered followups (contrary to what we actually see, in this GWAS and others...), give them; don't swap in criticisms of candidate-gene studies and pretend they're the same thing, because they're not. > Doesn't the significance level being "ludicrous" depend on things like the degree of multiple testing happening? It does. And that's exactly why holding replications to 10-8 is ludicrous. The error rate when testing 5 or 15 SNPs at 10-8 is much much much smaller than when testing 500,000 SNPs. Why would you hold a multiple-test of 5 tests to the same standard as 500,000+ tests? > I'm not saying anything controversial in the field. If you're insinuating that GWASes are as bad as candidate-gene studies were, or that a p-value threshold of 10-8 should be used for everything, or that we cannot have high confidence in any given hit at 10-8 (especially when replicated at 10-5) you certainly are saying controversial things. > I also gave this particular paper praise for holding to a higher standard than some other GWA studies. Such as?

by aab0 10y ago

erdevs 10y ago

> > and, believe me, many GWA studies have been published with much less significant p-values > I don't think they have. Ever since Ioannidis and others demonstrated what a total debacle the early candidate-gene studies were around 2009-2011 Yes, pre-Ioannidis is the period I was referring to. Things cleaned up a lot in 2012+. What you said does not conflict with what I said... the field, especially early on, has published some spurious results. It seems like you're simultaneously saying "I don't think they have [published low-quality results]" and then immediately admitting what a "debacle" early studies sometimes were. > That doesn't show anything relevant. Failure to replicate at 10-8 is a ludicrous way to define non-replication This is a silly statement. Doesn't the significance level being "ludicrous" depend on things like the degree of multiple testing happening? Obviously, yes. The reason why the field (not just this one paper) has pushed for significance in the 10^-7 to 10^-8 range is partially for this reason. So... are you questioning the entire field's movement over the past few years? If so, on what basis? As high-throughput, low-cost full genome sequencing begins to replace SNP-based techniques, GWAS will have to wrestle with this issue even more. I'm not sure what exactly you're debating me on here. I'm not saying anything controversial in the field. Again, even the wikipedia article cites well-known studies and quotations from respected sources in the literature, including "Particularly the statistical issue of multiple testing wherein it has been noted that "the GWA approach can be problematic because the massive number of statistical tests performed presents an unprecedented potential for false-positive results"... which is what I originally pointed out. This is an issue the field has struggled with from the get go. It's matured and is much better now (as I've noted), but the field still struggles with the issue. And there are still low-quality papers being published. I also gave this particular paper praise for holding to a higher standard than some other GWA studies.

aab0 10y ago
> Yes, pre-Ioannidis is the period I was referring to. Things cleaned up a lot in 2012+. What you said does not conflict with what I said.. A candidate-gene study != GWAS. It's particularly bizarre to criticize GWASes for the sins of candidate-gene study when GWASes were literally partially designed to avoid those problems. Don't equivocate. If you have criticisms of actual GWASes as they are run and good reason to doubt that the hits are noise and will not replicate in well-powered followups (contrary to what we actually see, in this GWAS and others...), give them; don't swap in criticisms of candidate-gene studies and pretend they're the same thing, because they're not. > Doesn't the significance level being "ludicrous" depend on things like the degree of multiple testing happening? It does. And that's exactly why holding replications to 10-8 is ludicrous. The error rate when testing 5 or 15 SNPs at 10-8 is much much much smaller than when testing 500,000 SNPs. Why would you hold a multiple-test of 5 tests to the same standard as 500,000+ tests? > I'm not saying anything controversial in the field. If you're insinuating that GWASes are as bad as candidate-gene studies were, or that a p-value threshold of 10-8 should be used for everything, or that we cannot have high confidence in any given hit at 10-8 (especially when replicated at 10-5) you certainly are saying controversial things. > I also gave this particular paper praise for holding to a higher standard than some other GWA studies. Such as?