5 ms·

I believe this is the underlying paper: http://www.nature.com/ng/journal/vaop/ncurrent/full/ng.3623.html http://www.nature.com/ng/journal/vaop/ncurrent/full/ng.

by erdevs 10y ago

I believe this is the underlying paper: http://www.nature.com/ng/journal/vaop/ncurrent/full/ng.3623.html http://www.nature.com/ng/journal/vaop/ncurrent/full/ng.3623.... GWA studies general should be treated with great caution. The way they work generally is based on a simple p-value test of association among outcome (in this case, depression) and all genes based on SNPs. There is a high degree of mere chance association and false positives. Most GWA studies leave a lot to be desired. This one looks more solid than most. The p-value appears to be 10^-5 and they ran a replication data set as well. Many GWASs report much less stringent p-value and many don't run sub-replications. One interesting aside about GWA studies: this may have changed recently, but I believe it used to be that any GWA study involving funding from the NIH was requires to post its data to a public, freely available database. That always seemed a good practice to me and one that should be emulated in other sorts of studies involving government funding. I wonder if this study is subject to that requirement.

jessriedel 10y ago
> There is a high degree of mere chance association and false positives. Are you really suggesting that most GWAS studies don't calculate genome-wide significance? This is wrong. If that's not what you're suggesting, I don't know what you're saying.
- erdevs 10y ago
  You should try reading up on the subject before being blithely dismissive. False discovery rates, false positive rates, and family-wise error rates and how best to control for them are ongoing areas of interest and research in GWAS. There are calls for p-value requirements in the 10^-8 range to help avoid this. There are calls to address stratification (which can affect both type I and type II errors). A lot of research and debate on this topic over the past several years. Who are you exactly to dismiss all of this scientific inquiry? It's great if you have expertise in another field, but it seems odd to dismiss scientific questions and ongoing research in this particular field. You'll find plenty of information if you actually seek it out rather than simply making a knee-jerk, snarky comment, but here is one example article which articulates some of the issues that have been under consideration in recent years: http://m.ije.oxfordjournals.org/content/41/1/273.full http://m.ije.oxfordjournals.org/content/41/1/273.full Note, there, how the level of significance is discussed. A p-value of 10^-7 to 10^-8 is suggested (as compared to this study's 10^-5 level of significance... and, believe me, many GWA studies have been published with much less significant p-values). This is an ongoing and active area of discussion in the field. I'm not an expert, but some of my colleagues are, and it's a topic they sometimes discuss and brainstorm over lunch, etc. Actually, even the wikipedia page on GWAS mentions some of this inquiry and debate, as well as the erroneous publication that has plagued this nascent field. It'll all be worked out over time and great discussions are happening here. Vast improvement in processes and standards has been made over the past few years in particular. But we don't move the ball forward by dismissing questions or incorrectly assuming all must simply be right and well.
  - aab0 10y ago
    > You'll find plenty of information if you actually seek it out rather than simply making a knee-jerk, snarky comment, but here is one example article which articulates some of the issues that have been under consideration in recent years: http://m.ije.oxfordjournals.org/content/41/1/273.full http://m.ije.oxfordjournals.org/content/41/1/273.full I assume you are referring to " If the seven associations that did not reach P ≤ 5 × 10−8 when additional data were considered are assumed to have been false-positives, the false-discovery rate for borderline associations is estimated to be 27% [95% confidence interval (CI) 12–48%]. For five associations, the current P-value is > 10−6 [corresponding false-discovery rate 19% (95% CI 7–39%)]." That doesn't show anything relevant. Failure to replicate at 10-8 is a ludicrous way to define non-replication; to paraphrase Cohen, surely God loves the 10-7.99 almost as much as the 10-8... This paper needs to adjust for power, and ask how many hits one would expect to not replicate at 10-8 given the power of the replicating studies. If you do remember power, GWASes replicate fantastically, for example https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3681663/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3681663/ "Replicability rates are high within Europeans, with 155 successful out of 181 attempts (85.6%), when only 9 positive replications (∼5%) would be expected under the null hypothesis of no association (binomial test, P<10−16). This excess was robust to the significance threshold (e.g. 122 observed vs. 0.18 expected if only replication attempts achieving P<0.001 are considered successful and 56 observed vs. 1.8×10−5 expected for a threshold of P<10−7, Table S5). Moreover, replicability rates within Europeans approach 100% when accounting for statistical power. For the 168 attempts for which we could calculate the power to replicate the original finding (Table S5), we observed 147 positive replications, which is almost identical to the expectation of 149.1 positive replications given that average power is 89.1% (see Materials and Methods). This is expected, since most GWAS already contain an internal replication phase [1], [24]." > and, believe me, many GWA studies have been published with much less significant p-values I don't think they have. Ever since Ioannidis and others demonstrated what a total debacle the early candidate-gene studies were around 2009-2011, using the first GWASes to demonstrate that, GWASes have been pretty standardly done at 10-8.
    - erdevs 10y ago
      > > and, believe me, many GWA studies have been published with much less significant p-values > I don't think they have. Ever since Ioannidis and others demonstrated what a total debacle the early candidate-gene studies were around 2009-2011 Yes, pre-Ioannidis is the period I was referring to. Things cleaned up a lot in 2012+. What you said does not conflict with what I said... the field, especially early on, has published some spurious results. It seems like you're simultaneously saying "I don't think they have [published low-quality results]" and then immediately admitting what a "debacle" early studies sometimes were. > That doesn't show anything relevant. Failure to replicate at 10-8 is a ludicrous way to define non-replication This is a silly statement. Doesn't the significance level being "ludicrous" depend on things like the degree of multiple testing happening? Obviously, yes. The reason why the field (not just this one paper) has pushed for significance in the 10^-7 to 10^-8 range is partially for this reason. So... are you questioning the entire field's movement over the past few years? If so, on what basis? As high-throughput, low-cost full genome sequencing begins to replace SNP-based techniques, GWAS will have to wrestle with this issue even more. I'm not sure what exactly you're debating me on here. I'm not saying anything controversial in the field. Again, even the wikipedia article cites well-known studies and quotations from respected sources in the literature, including "Particularly the statistical issue of multiple testing wherein it has been noted that "the GWA approach can be problematic because the massive number of statistical tests performed presents an unprecedented potential for false-positive results"... which is what I originally pointed out. This is an issue the field has struggled with from the get go. It's matured and is much better now (as I've noted), but the field still struggles with the issue. And there are still low-quality papers being published. I also gave this particular paper praise for holding to a higher standard than some other GWA studies.
- dekhn 10y ago
  The history of GWAS has shown that the community producing it took quite some time to reach barely acceptable levels of statistical analysis.
aab0 10y ago
There are 3 datasets here, the 23andMe discovery dataset (n=300k), the previous Psychiatric Genetic Consortium ('PGC'/'MDD' in the paper), and then a second 23andMe heldout validation sample (n=150k). The set of 5 hits discovered in the discovery dataset were at the usual 10^-8 threshold. They then meta-analytically pooled those results with the older PGC results and got a set of hits at 10^-5. Finally, they took those two datasets and pooled it with the held out replication dataset, yielding the final set of 17 hits at 10^-8. The final results are at the significance level you want, and almost all of the signs for the top SNPs are the same between datasets/cohorts (presumably why they reported broken-out sub-analyses rather than skipping straight to the final results, to demonstrate consistency). Those 17 are the ones used in the rest of the paper. > Many GWASs report much less stringent p-value and many don't run sub-replications. This isn't true. Most GWASes use the standard genome-wide significance level of 10-8. If they do not, it's because of well-motivated other considerations such as being replications of previous hits. (If you are testing replication of 5 earlier hits, rather than 500,000 SNPs, your p-value threshold ought to be looser.) > I wonder if this study is subject to that requirement. The supplementary gives the top 10k hits, which is the most critical part of the data, which you can use for polygenic scores, gene sets, heritability & genetic correlations via LD score regression etc. It's only top 10k SNPs because I believe 23andMe imposes that as a requirement on people using its data - something about possible reidentification if too many SNPs' values are released. (There are, of course, other cynical business-related reasons for why they might impose such a requirement.) I've seen that done in a few other GWAS studies like educational attainment, and they said it was because of 23andMe.
- erdevs 10y ago
  > There are 3 datasets here... Yes, understood. Part of the reason I dug up the actual paper was so that others could see it, as the news source linked didn't go into much detail. I didn't read the entire paper, and appreciate the additional explanation here. I'm sure it's helpful for others as a condensed summary as well. > > Many GWASs report much less stringent p-value and many don't run sub-replications. > This isn't true. What is not true? Do all GWAS studies do similar holdout replications and compare to other results? No. Have all GWAS studies over time always adjusted appropriately for multiple testing and held to a high significance standard? No. That is a more recent development and set of standards in the field over the past few years. I didn't make an outrageous claim; I simply noted that many GWASs have used much lower significance thresholds (especially in the past) and many don't do thorough holdout replications nor replicate in other ways. > Most GWASes use the standard genome-wide significance level of 10-8. Indeed. I see that you are aware of this push in the field for greater degrees of significance and that this is now fairly standardized... yet in another comment you appear to be debating me when I stated that this has been the case.
nonbel 10y ago
>"The p-value appears to be 10^-5 and they ran a replication data set as well. Many GWASs report much less stringent p-value and many don't run sub-replications." The p-value really should be totally irrelevant to your assessment, unless you believe the null model they used is literally true. I didn't check the paper, so maybe this is one of those rare times. However, in general I've seen that everything is correlated with everything.

by erdevs 10y ago

jessriedel 10y ago

> There is a high degree of mere chance association and false positives. Are you really suggesting that most GWAS studies don't calculate genome-wide significance? This is wrong. If that's not what you're suggesting, I don't know what you're saying.

erdevs 10y ago
You should try reading up on the subject before being blithely dismissive. False discovery rates, false positive rates, and family-wise error rates and how best to control for them are ongoing areas of interest and research in GWAS. There are calls for p-value requirements in the 10^-8 range to help avoid this. There are calls to address stratification (which can affect both type I and type II errors). A lot of research and debate on this topic over the past several years. Who are you exactly to dismiss all of this scientific inquiry? It's great if you have expertise in another field, but it seems odd to dismiss scientific questions and ongoing research in this particular field. You'll find plenty of information if you actually seek it out rather than simply making a knee-jerk, snarky comment, but here is one example article which articulates some of the issues that have been under consideration in recent years: http://m.ije.oxfordjournals.org/content/41/1/273.full http://m.ije.oxfordjournals.org/content/41/1/273.full Note, there, how the level of significance is discussed. A p-value of 10^-7 to 10^-8 is suggested (as compared to this study's 10^-5 level of significance... and, believe me, many GWA studies have been published with much less significant p-values). This is an ongoing and active area of discussion in the field. I'm not an expert, but some of my colleagues are, and it's a topic they sometimes discuss and brainstorm over lunch, etc. Actually, even the wikipedia page on GWAS mentions some of this inquiry and debate, as well as the erroneous publication that has plagued this nascent field. It'll all be worked out over time and great discussions are happening here. Vast improvement in processes and standards has been made over the past few years in particular. But we don't move the ball forward by dismissing questions or incorrectly assuming all must simply be right and well.
- aab0 10y ago
  > You'll find plenty of information if you actually seek it out rather than simply making a knee-jerk, snarky comment, but here is one example article which articulates some of the issues that have been under consideration in recent years: http://m.ije.oxfordjournals.org/content/41/1/273.full http://m.ije.oxfordjournals.org/content/41/1/273.full I assume you are referring to " If the seven associations that did not reach P ≤ 5 × 10−8 when additional data were considered are assumed to have been false-positives, the false-discovery rate for borderline associations is estimated to be 27% [95% confidence interval (CI) 12–48%]. For five associations, the current P-value is > 10−6 [corresponding false-discovery rate 19% (95% CI 7–39%)]." That doesn't show anything relevant. Failure to replicate at 10-8 is a ludicrous way to define non-replication; to paraphrase Cohen, surely God loves the 10-7.99 almost as much as the 10-8... This paper needs to adjust for power, and ask how many hits one would expect to not replicate at 10-8 given the power of the replicating studies. If you do remember power, GWASes replicate fantastically, for example https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3681663/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3681663/ "Replicability rates are high within Europeans, with 155 successful out of 181 attempts (85.6%), when only 9 positive replications (∼5%) would be expected under the null hypothesis of no association (binomial test, P<10−16). This excess was robust to the significance threshold (e.g. 122 observed vs. 0.18 expected if only replication attempts achieving P<0.001 are considered successful and 56 observed vs. 1.8×10−5 expected for a threshold of P<10−7, Table S5). Moreover, replicability rates within Europeans approach 100% when accounting for statistical power. For the 168 attempts for which we could calculate the power to replicate the original finding (Table S5), we observed 147 positive replications, which is almost identical to the expectation of 149.1 positive replications given that average power is 89.1% (see Materials and Methods). This is expected, since most GWAS already contain an internal replication phase [1], [24]." > and, believe me, many GWA studies have been published with much less significant p-values I don't think they have. Ever since Ioannidis and others demonstrated what a total debacle the early candidate-gene studies were around 2009-2011, using the first GWASes to demonstrate that, GWASes have been pretty standardly done at 10-8.
  - erdevs 10y ago
    > > and, believe me, many GWA studies have been published with much less significant p-values > I don't think they have. Ever since Ioannidis and others demonstrated what a total debacle the early candidate-gene studies were around 2009-2011 Yes, pre-Ioannidis is the period I was referring to. Things cleaned up a lot in 2012+. What you said does not conflict with what I said... the field, especially early on, has published some spurious results. It seems like you're simultaneously saying "I don't think they have [published low-quality results]" and then immediately admitting what a "debacle" early studies sometimes were. > That doesn't show anything relevant. Failure to replicate at 10-8 is a ludicrous way to define non-replication This is a silly statement. Doesn't the significance level being "ludicrous" depend on things like the degree of multiple testing happening? Obviously, yes. The reason why the field (not just this one paper) has pushed for significance in the 10^-7 to 10^-8 range is partially for this reason. So... are you questioning the entire field's movement over the past few years? If so, on what basis? As high-throughput, low-cost full genome sequencing begins to replace SNP-based techniques, GWAS will have to wrestle with this issue even more. I'm not sure what exactly you're debating me on here. I'm not saying anything controversial in the field. Again, even the wikipedia article cites well-known studies and quotations from respected sources in the literature, including "Particularly the statistical issue of multiple testing wherein it has been noted that "the GWA approach can be problematic because the massive number of statistical tests performed presents an unprecedented potential for false-positive results"... which is what I originally pointed out. This is an issue the field has struggled with from the get go. It's matured and is much better now (as I've noted), but the field still struggles with the issue. And there are still low-quality papers being published. I also gave this particular paper praise for holding to a higher standard than some other GWA studies.
dekhn 10y ago
The history of GWAS has shown that the community producing it took quite some time to reach barely acceptable levels of statistical analysis.

aab0 10y ago

There are 3 datasets here, the 23andMe discovery dataset (n=300k), the previous Psychiatric Genetic Consortium ('PGC'/'MDD' in the paper), and then a second 23andMe heldout validation sample (n=150k). The set of 5 hits discovered in the discovery dataset were at the usual 10^-8 threshold. They then meta-analytically pooled those results with the older PGC results and got a set of hits at 10^-5. Finally, they took those two datasets and pooled it with the held out replication dataset, yielding the final set of 17 hits at 10^-8. The final results are at the significance level you want, and almost all of the signs for the top SNPs are the same between datasets/cohorts (presumably why they reported broken-out sub-analyses rather than skipping straight to the final results, to demonstrate consistency). Those 17 are the ones used in the rest of the paper. > Many GWASs report much less stringent p-value and many don't run sub-replications. This isn't true. Most GWASes use the standard genome-wide significance level of 10-8. If they do not, it's because of well-motivated other considerations such as being replications of previous hits. (If you are testing replication of 5 earlier hits, rather than 500,000 SNPs, your p-value threshold ought to be looser.) > I wonder if this study is subject to that requirement. The supplementary gives the top 10k hits, which is the most critical part of the data, which you can use for polygenic scores, gene sets, heritability & genetic correlations via LD score regression etc. It's only top 10k SNPs because I believe 23andMe imposes that as a requirement on people using its data - something about possible reidentification if too many SNPs' values are released. (There are, of course, other cynical business-related reasons for why they might impose such a requirement.) I've seen that done in a few other GWAS studies like educational attainment, and they said it was because of 23andMe.

erdevs 10y ago
> There are 3 datasets here... Yes, understood. Part of the reason I dug up the actual paper was so that others could see it, as the news source linked didn't go into much detail. I didn't read the entire paper, and appreciate the additional explanation here. I'm sure it's helpful for others as a condensed summary as well. > > Many GWASs report much less stringent p-value and many don't run sub-replications. > This isn't true. What is not true? Do all GWAS studies do similar holdout replications and compare to other results? No. Have all GWAS studies over time always adjusted appropriately for multiple testing and held to a high significance standard? No. That is a more recent development and set of standards in the field over the past few years. I didn't make an outrageous claim; I simply noted that many GWASs have used much lower significance thresholds (especially in the past) and many don't do thorough holdout replications nor replicate in other ways. > Most GWASes use the standard genome-wide significance level of 10-8. Indeed. I see that you are aware of this push in the field for greater degrees of significance and that this is now fairly standardized... yet in another comment you appear to be debating me when I stated that this has been the case.

nonbel 10y ago

>"The p-value appears to be 10^-5 and they ran a replication data set as well. Many GWASs report much less stringent p-value and many don't run sub-replications." The p-value really should be totally irrelevant to your assessment, unless you believe the null model they used is literally true. I didn't check the paper, so maybe this is one of those rare times. However, in general I've seen that everything is correlated with everything.