6 ms·

You should try reading up on the subject before being blithely dismissive. False discovery rates, false positive rates, and family-wise error rates and how bes

by erdevs 10y ago

You should try reading up on the subject before being blithely dismissive. False discovery rates, false positive rates, and family-wise error rates and how best to control for them are ongoing areas of interest and research in GWAS. There are calls for p-value requirements in the 10^-8 range to help avoid this. There are calls to address stratification (which can affect both type I and type II errors). A lot of research and debate on this topic over the past several years. Who are you exactly to dismiss all of this scientific inquiry? It's great if you have expertise in another field, but it seems odd to dismiss scientific questions and ongoing research in this particular field. You'll find plenty of information if you actually seek it out rather than simply making a knee-jerk, snarky comment, but here is one example article which articulates some of the issues that have been under consideration in recent years: http://m.ije.oxfordjournals.org/content/41/1/273.full http://m.ije.oxfordjournals.org/content/41/1/273.full Note, there, how the level of significance is discussed. A p-value of 10^-7 to 10^-8 is suggested (as compared to this study's 10^-5 level of significance... and, believe me, many GWA studies have been published with much less significant p-values). This is an ongoing and active area of discussion in the field. I'm not an expert, but some of my colleagues are, and it's a topic they sometimes discuss and brainstorm over lunch, etc. Actually, even the wikipedia page on GWAS mentions some of this inquiry and debate, as well as the erroneous publication that has plagued this nascent field. It'll all be worked out over time and great discussions are happening here. Vast improvement in processes and standards has been made over the past few years in particular. But we don't move the ball forward by dismissing questions or incorrectly assuming all must simply be right and well.

aab0 10y ago
> You'll find plenty of information if you actually seek it out rather than simply making a knee-jerk, snarky comment, but here is one example article which articulates some of the issues that have been under consideration in recent years: http://m.ije.oxfordjournals.org/content/41/1/273.full http://m.ije.oxfordjournals.org/content/41/1/273.full I assume you are referring to " If the seven associations that did not reach P ≤ 5 × 10−8 when additional data were considered are assumed to have been false-positives, the false-discovery rate for borderline associations is estimated to be 27% [95% confidence interval (CI) 12–48%]. For five associations, the current P-value is > 10−6 [corresponding false-discovery rate 19% (95% CI 7–39%)]." That doesn't show anything relevant. Failure to replicate at 10-8 is a ludicrous way to define non-replication; to paraphrase Cohen, surely God loves the 10-7.99 almost as much as the 10-8... This paper needs to adjust for power, and ask how many hits one would expect to not replicate at 10-8 given the power of the replicating studies. If you do remember power, GWASes replicate fantastically, for example https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3681663/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3681663/ "Replicability rates are high within Europeans, with 155 successful out of 181 attempts (85.6%), when only 9 positive replications (∼5%) would be expected under the null hypothesis of no association (binomial test, P<10−16). This excess was robust to the significance threshold (e.g. 122 observed vs. 0.18 expected if only replication attempts achieving P<0.001 are considered successful and 56 observed vs. 1.8×10−5 expected for a threshold of P<10−7, Table S5). Moreover, replicability rates within Europeans approach 100% when accounting for statistical power. For the 168 attempts for which we could calculate the power to replicate the original finding (Table S5), we observed 147 positive replications, which is almost identical to the expectation of 149.1 positive replications given that average power is 89.1% (see Materials and Methods). This is expected, since most GWAS already contain an internal replication phase [1], [24]." > and, believe me, many GWA studies have been published with much less significant p-values I don't think they have. Ever since Ioannidis and others demonstrated what a total debacle the early candidate-gene studies were around 2009-2011, using the first GWASes to demonstrate that, GWASes have been pretty standardly done at 10-8.
- erdevs 10y ago
  > > and, believe me, many GWA studies have been published with much less significant p-values > I don't think they have. Ever since Ioannidis and others demonstrated what a total debacle the early candidate-gene studies were around 2009-2011 Yes, pre-Ioannidis is the period I was referring to. Things cleaned up a lot in 2012+. What you said does not conflict with what I said... the field, especially early on, has published some spurious results. It seems like you're simultaneously saying "I don't think they have [published low-quality results]" and then immediately admitting what a "debacle" early studies sometimes were. > That doesn't show anything relevant. Failure to replicate at 10-8 is a ludicrous way to define non-replication This is a silly statement. Doesn't the significance level being "ludicrous" depend on things like the degree of multiple testing happening? Obviously, yes. The reason why the field (not just this one paper) has pushed for significance in the 10^-7 to 10^-8 range is partially for this reason. So... are you questioning the entire field's movement over the past few years? If so, on what basis? As high-throughput, low-cost full genome sequencing begins to replace SNP-based techniques, GWAS will have to wrestle with this issue even more. I'm not sure what exactly you're debating me on here. I'm not saying anything controversial in the field. Again, even the wikipedia article cites well-known studies and quotations from respected sources in the literature, including "Particularly the statistical issue of multiple testing wherein it has been noted that "the GWA approach can be problematic because the massive number of statistical tests performed presents an unprecedented potential for false-positive results"... which is what I originally pointed out. This is an issue the field has struggled with from the get go. It's matured and is much better now (as I've noted), but the field still struggles with the issue. And there are still low-quality papers being published. I also gave this particular paper praise for holding to a higher standard than some other GWA studies.
  - aab0 10y ago
    > Yes, pre-Ioannidis is the period I was referring to. Things cleaned up a lot in 2012+. What you said does not conflict with what I said.. A candidate-gene study != GWAS. It's particularly bizarre to criticize GWASes for the sins of candidate-gene study when GWASes were literally partially designed to avoid those problems. Don't equivocate. If you have criticisms of actual GWASes as they are run and good reason to doubt that the hits are noise and will not replicate in well-powered followups (contrary to what we actually see, in this GWAS and others...), give them; don't swap in criticisms of candidate-gene studies and pretend they're the same thing, because they're not. > Doesn't the significance level being "ludicrous" depend on things like the degree of multiple testing happening? It does. And that's exactly why holding replications to 10-8 is ludicrous. The error rate when testing 5 or 15 SNPs at 10-8 is much much much smaller than when testing 500,000 SNPs. Why would you hold a multiple-test of 5 tests to the same standard as 500,000+ tests? > I'm not saying anything controversial in the field. If you're insinuating that GWASes are as bad as candidate-gene studies were, or that a p-value threshold of 10-8 should be used for everything, or that we cannot have high confidence in any given hit at 10-8 (especially when replicated at 10-5) you certainly are saying controversial things. > I also gave this particular paper praise for holding to a higher standard than some other GWA studies. Such as?
jessriedel 10y ago
My comment was not intended to have the barbs on it that appear to have been read into it. ("Who are you exactly to dismiss all of this scientific inquiry?" "...knee-jerk, snarky comment...") I'm sorry that it came off that way. In any case, I still don't know what your concrete claim is. > here is one example article which articulates some of the issues that have been under consideration in recent years: http://m.ije.oxfordjournals.org/content/41/1/273.full http://m.ije.oxfordjournals.org/content/41/1/273.full From the abstract of that article: "Currently, associations of common variants reaching P ≤ 5 × 10−8 are considered replicated. However, there is some ambiguity about the most suitable threshold for claiming genome-wide significance." So, people do calculate genome-wide significance, and there is some ambiguity over where exactly that line should be. This is in line with my understanding of the situation. The exactly analogous statement can be made within particle physics ("there is some ambiguity about the most suitable threshold for claiming significance"), where it is often called the "look-elsewhere effect". But this prudent caution doesn't cause to people to say FUD like "particle physics studies should be treated with great caution" or "Most particle physics studies leave a lot to be desired". Such statements may absolutely in fact be true for GWAS studies, but you didn't give any good reasons for it. Indeed the very article you link to ends this way: "Conclusion: A substantial proportion, but not all, of the associations with borderline genome-wide significance represent replicable, possibly genuine associations. Our empirical evaluation suggests a possible relaxation in the current GWS threshold." How should one square that conclusion with your original comment?
- erdevs 10y ago
  Thanks for the reply, and my turn to apologize for misinterpreting your tone. You asked initially what exactly I was saying, and here again how to square what I'm saying with my original comment. So, let me try to explain, and I would hope we're not on different sides of this as I think what I'm saying is reasonable, given the context. I originally said: "GWA studies general should be treated with great caution. The way they work generally is based on a simple p-value test of association among outcome (in this case, depression) and all genes based on SNPs. There is a high degree of mere chance association and false positives. Most GWA studies leave a lot to be desired." For context: genome-association studies have had a history of being blown out of proportion in the press. And often for outcomes which greatly affect people's lives. Depression is one such issue and it'd be a shame if people were led into thinking there is necessarily a great breakthrough here in understanding possible gene-linkages to depression outcomes. It'd also be a shame if the result was ignored. I tried to provide some praise to this paper for being fairly rigorous, but also note that GWAS studies should be treated with caution generally. Why should GWAS studies be treated with caution generally? (Besides that results of any study should be treated with some degree of caution.) Well, firstly, GWAS is a fairly nascent field. Unlike physics or even particle physics, it hasn't had that much time to mature. This is doubly so when applying GWAS to mental health. I'm sure the paper covers some of these risks (or at least it hopefully does)... but relying on self-reporting introduces potential selection bias in the population sample, as does using people seeking help vs a general population study. Quickly reading the paper, it looks like it relied on self-reports and analyzed only people who'd been diagnosed with major depression (meaning they'd sought help). We should be cautious in over-generalizing based on this. Secondly, GWAS has had a rough history of overstating results and misapplying analyses. It's much better today than it was even, say, 5 years ago. Ioannidis and others made some heroic efforts to convince the field to clean up it's act, in effect starting ~7-8 years ago. Thirdly, there is a historical pattern of results in this field being overhyped in the press. Finally, there are active and sometimes heated debates in the field about how best to do GWAS. This is getting worse as a high-throughput, low-cost full-genome sequencing comes online to a greater and greater degree and SNP-based data sets fall by the wayside. Some question taking a frequentist approach at all in the face of such a huge degree of multiple testing. Others call for much, much higher requirements for holdout data sets, cross-validation, and replication before a study is published or considered final, especially when dealing with things like mental health (and their likely application to the field of pharmacology). This is serious stuff that could end up affecting people's mental health treatments and lives, so caution is warranted. Especially given the field's relative nascence, self-admitted history of publishing low-quality results, the rapidly changing techniques, and the fact that there are ongoing debates within the field of how best to do GWAS analysis and how to effectively replicate results.
hyperbovine 10y ago
FDR was practically invented to handle the multiple testing issues that accompany analyzing array data. I think what OP was implying was that there is no way that you are going to get a GWAS paper into Nature Genetics in 2016 without performing the appropriate statistical corrections. Literally everyone in the field is aware of this issue. In other words, give the authors and referees a little credit.
- erdevs 10y ago
  I did not mean to imply that this paper was due extra skepticism in particular. Indeed, I tried to give the paper due praise for seeming to be more rigorous than many other GWASs. And, yes indeed, false discovery rate has been a huge issue which the field has highlighted... that is my point. Caution has been warranted here historically, and it continues to be warranted even today.

by erdevs 10y ago

aab0 10y ago

> You'll find plenty of information if you actually seek it out rather than simply making a knee-jerk, snarky comment, but here is one example article which articulates some of the issues that have been under consideration in recent years: http://m.ije.oxfordjournals.org/content/41/1/273.full http://m.ije.oxfordjournals.org/content/41/1/273.full I assume you are referring to " If the seven associations that did not reach P ≤ 5 × 10−8 when additional data were considered are assumed to have been false-positives, the false-discovery rate for borderline associations is estimated to be 27% [95% confidence interval (CI) 12–48%]. For five associations, the current P-value is > 10−6 [corresponding false-discovery rate 19% (95% CI 7–39%)]." That doesn't show anything relevant. Failure to replicate at 10-8 is a ludicrous way to define non-replication; to paraphrase Cohen, surely God loves the 10-7.99 almost as much as the 10-8... This paper needs to adjust for power, and ask how many hits one would expect to not replicate at 10-8 given the power of the replicating studies. If you do remember power, GWASes replicate fantastically, for example https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3681663/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3681663/ "Replicability rates are high within Europeans, with 155 successful out of 181 attempts (85.6%), when only 9 positive replications (∼5%) would be expected under the null hypothesis of no association (binomial test, P<10−16). This excess was robust to the significance threshold (e.g. 122 observed vs. 0.18 expected if only replication attempts achieving P<0.001 are considered successful and 56 observed vs. 1.8×10−5 expected for a threshold of P<10−7, Table S5). Moreover, replicability rates within Europeans approach 100% when accounting for statistical power. For the 168 attempts for which we could calculate the power to replicate the original finding (Table S5), we observed 147 positive replications, which is almost identical to the expectation of 149.1 positive replications given that average power is 89.1% (see Materials and Methods). This is expected, since most GWAS already contain an internal replication phase [1], [24]." > and, believe me, many GWA studies have been published with much less significant p-values I don't think they have. Ever since Ioannidis and others demonstrated what a total debacle the early candidate-gene studies were around 2009-2011, using the first GWASes to demonstrate that, GWASes have been pretty standardly done at 10-8.

erdevs 10y ago
> > and, believe me, many GWA studies have been published with much less significant p-values > I don't think they have. Ever since Ioannidis and others demonstrated what a total debacle the early candidate-gene studies were around 2009-2011 Yes, pre-Ioannidis is the period I was referring to. Things cleaned up a lot in 2012+. What you said does not conflict with what I said... the field, especially early on, has published some spurious results. It seems like you're simultaneously saying "I don't think they have [published low-quality results]" and then immediately admitting what a "debacle" early studies sometimes were. > That doesn't show anything relevant. Failure to replicate at 10-8 is a ludicrous way to define non-replication This is a silly statement. Doesn't the significance level being "ludicrous" depend on things like the degree of multiple testing happening? Obviously, yes. The reason why the field (not just this one paper) has pushed for significance in the 10^-7 to 10^-8 range is partially for this reason. So... are you questioning the entire field's movement over the past few years? If so, on what basis? As high-throughput, low-cost full genome sequencing begins to replace SNP-based techniques, GWAS will have to wrestle with this issue even more. I'm not sure what exactly you're debating me on here. I'm not saying anything controversial in the field. Again, even the wikipedia article cites well-known studies and quotations from respected sources in the literature, including "Particularly the statistical issue of multiple testing wherein it has been noted that "the GWA approach can be problematic because the massive number of statistical tests performed presents an unprecedented potential for false-positive results"... which is what I originally pointed out. This is an issue the field has struggled with from the get go. It's matured and is much better now (as I've noted), but the field still struggles with the issue. And there are still low-quality papers being published. I also gave this particular paper praise for holding to a higher standard than some other GWA studies.
- aab0 10y ago
  > Yes, pre-Ioannidis is the period I was referring to. Things cleaned up a lot in 2012+. What you said does not conflict with what I said.. A candidate-gene study != GWAS. It's particularly bizarre to criticize GWASes for the sins of candidate-gene study when GWASes were literally partially designed to avoid those problems. Don't equivocate. If you have criticisms of actual GWASes as they are run and good reason to doubt that the hits are noise and will not replicate in well-powered followups (contrary to what we actually see, in this GWAS and others...), give them; don't swap in criticisms of candidate-gene studies and pretend they're the same thing, because they're not. > Doesn't the significance level being "ludicrous" depend on things like the degree of multiple testing happening? It does. And that's exactly why holding replications to 10-8 is ludicrous. The error rate when testing 5 or 15 SNPs at 10-8 is much much much smaller than when testing 500,000 SNPs. Why would you hold a multiple-test of 5 tests to the same standard as 500,000+ tests? > I'm not saying anything controversial in the field. If you're insinuating that GWASes are as bad as candidate-gene studies were, or that a p-value threshold of 10-8 should be used for everything, or that we cannot have high confidence in any given hit at 10-8 (especially when replicated at 10-5) you certainly are saying controversial things. > I also gave this particular paper praise for holding to a higher standard than some other GWA studies. Such as?

jessriedel 10y ago

My comment was not intended to have the barbs on it that appear to have been read into it. ("Who are you exactly to dismiss all of this scientific inquiry?" "...knee-jerk, snarky comment...") I'm sorry that it came off that way. In any case, I still don't know what your concrete claim is. > here is one example article which articulates some of the issues that have been under consideration in recent years: http://m.ije.oxfordjournals.org/content/41/1/273.full http://m.ije.oxfordjournals.org/content/41/1/273.full From the abstract of that article: "Currently, associations of common variants reaching P ≤ 5 × 10−8 are considered replicated. However, there is some ambiguity about the most suitable threshold for claiming genome-wide significance." So, people do calculate genome-wide significance, and there is some ambiguity over where exactly that line should be. This is in line with my understanding of the situation. The exactly analogous statement can be made within particle physics ("there is some ambiguity about the most suitable threshold for claiming significance"), where it is often called the "look-elsewhere effect". But this prudent caution doesn't cause to people to say FUD like "particle physics studies should be treated with great caution" or "Most particle physics studies leave a lot to be desired". Such statements may absolutely in fact be true for GWAS studies, but you didn't give any good reasons for it. Indeed the very article you link to ends this way: "Conclusion: A substantial proportion, but not all, of the associations with borderline genome-wide significance represent replicable, possibly genuine associations. Our empirical evaluation suggests a possible relaxation in the current GWS threshold." How should one square that conclusion with your original comment?

erdevs 10y ago
Thanks for the reply, and my turn to apologize for misinterpreting your tone. You asked initially what exactly I was saying, and here again how to square what I'm saying with my original comment. So, let me try to explain, and I would hope we're not on different sides of this as I think what I'm saying is reasonable, given the context. I originally said: "GWA studies general should be treated with great caution. The way they work generally is based on a simple p-value test of association among outcome (in this case, depression) and all genes based on SNPs. There is a high degree of mere chance association and false positives. Most GWA studies leave a lot to be desired." For context: genome-association studies have had a history of being blown out of proportion in the press. And often for outcomes which greatly affect people's lives. Depression is one such issue and it'd be a shame if people were led into thinking there is necessarily a great breakthrough here in understanding possible gene-linkages to depression outcomes. It'd also be a shame if the result was ignored. I tried to provide some praise to this paper for being fairly rigorous, but also note that GWAS studies should be treated with caution generally. Why should GWAS studies be treated with caution generally? (Besides that results of any study should be treated with some degree of caution.) Well, firstly, GWAS is a fairly nascent field. Unlike physics or even particle physics, it hasn't had that much time to mature. This is doubly so when applying GWAS to mental health. I'm sure the paper covers some of these risks (or at least it hopefully does)... but relying on self-reporting introduces potential selection bias in the population sample, as does using people seeking help vs a general population study. Quickly reading the paper, it looks like it relied on self-reports and analyzed only people who'd been diagnosed with major depression (meaning they'd sought help). We should be cautious in over-generalizing based on this. Secondly, GWAS has had a rough history of overstating results and misapplying analyses. It's much better today than it was even, say, 5 years ago. Ioannidis and others made some heroic efforts to convince the field to clean up it's act, in effect starting ~7-8 years ago. Thirdly, there is a historical pattern of results in this field being overhyped in the press. Finally, there are active and sometimes heated debates in the field about how best to do GWAS. This is getting worse as a high-throughput, low-cost full-genome sequencing comes online to a greater and greater degree and SNP-based data sets fall by the wayside. Some question taking a frequentist approach at all in the face of such a huge degree of multiple testing. Others call for much, much higher requirements for holdout data sets, cross-validation, and replication before a study is published or considered final, especially when dealing with things like mental health (and their likely application to the field of pharmacology). This is serious stuff that could end up affecting people's mental health treatments and lives, so caution is warranted. Especially given the field's relative nascence, self-admitted history of publishing low-quality results, the rapidly changing techniques, and the fact that there are ongoing debates within the field of how best to do GWAS analysis and how to effectively replicate results.

hyperbovine 10y ago

FDR was practically invented to handle the multiple testing issues that accompany analyzing array data. I think what OP was implying was that there is no way that you are going to get a GWAS paper into Nature Genetics in 2016 without performing the appropriate statistical corrections. Literally everyone in the field is aware of this issue. In other words, give the authors and referees a little credit.

erdevs 10y ago
I did not mean to imply that this paper was due extra skepticism in particular. Indeed, I tried to give the paper due praise for seeming to be more rigorous than many other GWASs. And, yes indeed, false discovery rate has been a huge issue which the field has highlighted... that is my point. Caution has been warranted here historically, and it continues to be warranted even today.