3 ms·

Thanks for the reply, and my turn to apologize for misinterpreting your tone. You asked initially what exactly I was saying, and here again how to square what

by erdevs 10y ago

Thanks for the reply, and my turn to apologize for misinterpreting your tone. You asked initially what exactly I was saying, and here again how to square what I'm saying with my original comment. So, let me try to explain, and I would hope we're not on different sides of this as I think what I'm saying is reasonable, given the context. I originally said: "GWA studies general should be treated with great caution. The way they work generally is based on a simple p-value test of association among outcome (in this case, depression) and all genes based on SNPs. There is a high degree of mere chance association and false positives. Most GWA studies leave a lot to be desired." For context: genome-association studies have had a history of being blown out of proportion in the press. And often for outcomes which greatly affect people's lives. Depression is one such issue and it'd be a shame if people were led into thinking there is necessarily a great breakthrough here in understanding possible gene-linkages to depression outcomes. It'd also be a shame if the result was ignored. I tried to provide some praise to this paper for being fairly rigorous, but also note that GWAS studies should be treated with caution generally. Why should GWAS studies be treated with caution generally? (Besides that results of any study should be treated with some degree of caution.) Well, firstly, GWAS is a fairly nascent field. Unlike physics or even particle physics, it hasn't had that much time to mature. This is doubly so when applying GWAS to mental health. I'm sure the paper covers some of these risks (or at least it hopefully does)... but relying on self-reporting introduces potential selection bias in the population sample, as does using people seeking help vs a general population study. Quickly reading the paper, it looks like it relied on self-reports and analyzed only people who'd been diagnosed with major depression (meaning they'd sought help). We should be cautious in over-generalizing based on this. Secondly, GWAS has had a rough history of overstating results and misapplying analyses. It's much better today than it was even, say, 5 years ago. Ioannidis and others made some heroic efforts to convince the field to clean up it's act, in effect starting ~7-8 years ago. Thirdly, there is a historical pattern of results in this field being overhyped in the press. Finally, there are active and sometimes heated debates in the field about how best to do GWAS. This is getting worse as a high-throughput, low-cost full-genome sequencing comes online to a greater and greater degree and SNP-based data sets fall by the wayside. Some question taking a frequentist approach at all in the face of such a huge degree of multiple testing. Others call for much, much higher requirements for holdout data sets, cross-validation, and replication before a study is published or considered final, especially when dealing with things like mental health (and their likely application to the field of pharmacology). This is serious stuff that could end up affecting people's mental health treatments and lives, so caution is warranted. Especially given the field's relative nascence, self-admitted history of publishing low-quality results, the rapidly changing techniques, and the fact that there are ongoing debates within the field of how best to do GWAS analysis and how to effectively replicate results.

erdevs 10y ago
Here is an example of the sort of press reaction I'm referring to (this isn't a particularly egregious example, but it demonstrates the point): https://www.washingtonpost.com/news/to-your-health/wp/2016/08/01/large-dna-study-using-23andme-data-finds-15-sites-linked-to-depression/ https://www.washingtonpost.com/news/to-your-health/wp/2016/0... Look at the language there. Scientists "pinpointed 15 locations in our DNA that are associated with depression..." [emphases added]. There is no sense of nuance or any caution in conclusion here. Even reading the whole article, which perhaps most people won't even bother with, you don't get much sense that there's any degree of uncertainty here. There is no indication that the field at large is still wrestling with how best to analyze SNP studies at all, despite publishing studies "practically ever week". There is little qualification around the data set here (it only briefly mentions that 23andme's data is based on saliva, not blood or other cells... which may be problematic. But also it should be noted that disease outcomes in this study were self-reported and from a self-selected subset of the general population who sought professional help... and there are many, many other nuances to consider). Instead, we get the fairly flat-out impression that this is a definitive discovery and, not only that, but it is of huge significance in the field as well. It may well be. But caution is warranted until time and replication hopefully do their work. Unfortunately, this is a pattern in GWAS studies and their public relations. And while the underlying methodology in the field has improved tremendously over the past 5 years, there is still a lot of debate within the field about how much more improvement needs to be made (on data sources, on analytic techniques, on review standards), and how to adapt to forthcoming changes in available study data. I think this study appears to be more robust than many others. So, I'm not picking on it. In fact I gave it particular praise for being relatively robust compared to some other GWASs. But I do think generally people should be more cautious about interpreting GWA studies or over-generalizing them.

by erdevs 10y ago

erdevs 10y ago

Here is an example of the sort of press reaction I'm referring to (this isn't a particularly egregious example, but it demonstrates the point): https://www.washingtonpost.com/news/to-your-health/wp/2016/08/01/large-dna-study-using-23andme-data-finds-15-sites-linked-to-depression/ https://www.washingtonpost.com/news/to-your-health/wp/2016/0... Look at the language there. Scientists "pinpointed 15 locations in our DNA that are associated with depression..." [emphases added]. There is no sense of nuance or any caution in conclusion here. Even reading the whole article, which perhaps most people won't even bother with, you don't get much sense that there's any degree of uncertainty here. There is no indication that the field at large is still wrestling with how best to analyze SNP studies at all, despite publishing studies "practically ever week". There is little qualification around the data set here (it only briefly mentions that 23andme's data is based on saliva, not blood or other cells... which may be problematic. But also it should be noted that disease outcomes in this study were self-reported and from a self-selected subset of the general population who sought professional help... and there are many, many other nuances to consider). Instead, we get the fairly flat-out impression that this is a definitive discovery and, not only that, but it is of huge significance in the field as well. It may well be. But caution is warranted until time and replication hopefully do their work. Unfortunately, this is a pattern in GWAS studies and their public relations. And while the underlying methodology in the field has improved tremendously over the past 5 years, there is still a lot of debate within the field about how much more improvement needs to be made (on data sources, on analytic techniques, on review standards), and how to adapt to forthcoming changes in available study data. I think this study appears to be more robust than many others. So, I'm not picking on it. In fact I gave it particular praise for being relatively robust compared to some other GWASs. But I do think generally people should be more cautious about interpreting GWA studies or over-generalizing them.