3 ms·

My comment was not intended to have the barbs on it that appear to have been read into it. ("Who are you exactly to dismiss all of this scientific inquiry?" "

My comment was not intended to have the barbs on it that appear to have been read into it. ("Who are you exactly to dismiss all of this scientific inquiry?" "...knee-jerk, snarky comment...") I'm sorry that it came off that way. In any case, I still don't know what your concrete claim is. > here is one example article which articulates some of the issues that have been under consideration in recent years: http://m.ije.oxfordjournals.org/content/41/1/273.full http://m.ije.oxfordjournals.org/content/41/1/273.full From the abstract of that article: "Currently, associations of common variants reaching P ≤ 5 × 10−8 are considered replicated. However, there is some ambiguity about the most suitable threshold for claiming genome-wide significance." So, people do calculate genome-wide significance, and there is some ambiguity over where exactly that line should be. This is in line with my understanding of the situation. The exactly analogous statement can be made within particle physics ("there is some ambiguity about the most suitable threshold for claiming significance"), where it is often called the "look-elsewhere effect". But this prudent caution doesn't cause to people to say FUD like "particle physics studies should be treated with great caution" or "Most particle physics studies leave a lot to be desired". Such statements may absolutely in fact be true for GWAS studies, but you didn't give any good reasons for it. Indeed the very article you link to ends this way: "Conclusion: A substantial proportion, but not all, of the associations with borderline genome-wide significance represent replicable, possibly genuine associations. Our empirical evaluation suggests a possible relaxation in the current GWS threshold." How should one square that conclusion with your original comment?

erdevs 10y ago
Thanks for the reply, and my turn to apologize for misinterpreting your tone. You asked initially what exactly I was saying, and here again how to square what I'm saying with my original comment. So, let me try to explain, and I would hope we're not on different sides of this as I think what I'm saying is reasonable, given the context. I originally said: "GWA studies general should be treated with great caution. The way they work generally is based on a simple p-value test of association among outcome (in this case, depression) and all genes based on SNPs. There is a high degree of mere chance association and false positives. Most GWA studies leave a lot to be desired." For context: genome-association studies have had a history of being blown out of proportion in the press. And often for outcomes which greatly affect people's lives. Depression is one such issue and it'd be a shame if people were led into thinking there is necessarily a great breakthrough here in understanding possible gene-linkages to depression outcomes. It'd also be a shame if the result was ignored. I tried to provide some praise to this paper for being fairly rigorous, but also note that GWAS studies should be treated with caution generally. Why should GWAS studies be treated with caution generally? (Besides that results of any study should be treated with some degree of caution.) Well, firstly, GWAS is a fairly nascent field. Unlike physics or even particle physics, it hasn't had that much time to mature. This is doubly so when applying GWAS to mental health. I'm sure the paper covers some of these risks (or at least it hopefully does)... but relying on self-reporting introduces potential selection bias in the population sample, as does using people seeking help vs a general population study. Quickly reading the paper, it looks like it relied on self-reports and analyzed only people who'd been diagnosed with major depression (meaning they'd sought help). We should be cautious in over-generalizing based on this. Secondly, GWAS has had a rough history of overstating results and misapplying analyses. It's much better today than it was even, say, 5 years ago. Ioannidis and others made some heroic efforts to convince the field to clean up it's act, in effect starting ~7-8 years ago. Thirdly, there is a historical pattern of results in this field being overhyped in the press. Finally, there are active and sometimes heated debates in the field about how best to do GWAS. This is getting worse as a high-throughput, low-cost full-genome sequencing comes online to a greater and greater degree and SNP-based data sets fall by the wayside. Some question taking a frequentist approach at all in the face of such a huge degree of multiple testing. Others call for much, much higher requirements for holdout data sets, cross-validation, and replication before a study is published or considered final, especially when dealing with things like mental health (and their likely application to the field of pharmacology). This is serious stuff that could end up affecting people's mental health treatments and lives, so caution is warranted. Especially given the field's relative nascence, self-admitted history of publishing low-quality results, the rapidly changing techniques, and the fact that there are ongoing debates within the field of how best to do GWAS analysis and how to effectively replicate results.
- erdevs 10y ago
  Here is an example of the sort of press reaction I'm referring to (this isn't a particularly egregious example, but it demonstrates the point): https://www.washingtonpost.com/news/to-your-health/wp/2016/08/01/large-dna-study-using-23andme-data-finds-15-sites-linked-to-depression/ https://www.washingtonpost.com/news/to-your-health/wp/2016/0... Look at the language there. Scientists "pinpointed 15 locations in our DNA that are associated with depression..." [emphases added]. There is no sense of nuance or any caution in conclusion here. Even reading the whole article, which perhaps most people won't even bother with, you don't get much sense that there's any degree of uncertainty here. There is no indication that the field at large is still wrestling with how best to analyze SNP studies at all, despite publishing studies "practically ever week". There is little qualification around the data set here (it only briefly mentions that 23andme's data is based on saliva, not blood or other cells... which may be problematic. But also it should be noted that disease outcomes in this study were self-reported and from a self-selected subset of the general population who sought professional help... and there are many, many other nuances to consider). Instead, we get the fairly flat-out impression that this is a definitive discovery and, not only that, but it is of huge significance in the field as well. It may well be. But caution is warranted until time and replication hopefully do their work. Unfortunately, this is a pattern in GWAS studies and their public relations. And while the underlying methodology in the field has improved tremendously over the past 5 years, there is still a lot of debate within the field about how much more improvement needs to be made (on data sources, on analytic techniques, on review standards), and how to adapt to forthcoming changes in available study data. I think this study appears to be more robust than many others. So, I'm not picking on it. In fact I gave it particular praise for being relatively robust compared to some other GWASs. But I do think generally people should be more cautious about interpreting GWA studies or over-generalizing them.

erdevs 10y ago

Thanks for the reply, and my turn to apologize for misinterpreting your tone. You asked initially what exactly I was saying, and here again how to square what I'm saying with my original comment. So, let me try to explain, and I would hope we're not on different sides of this as I think what I'm saying is reasonable, given the context. I originally said: "GWA studies general should be treated with great caution. The way they work generally is based on a simple p-value test of association among outcome (in this case, depression) and all genes based on SNPs. There is a high degree of mere chance association and false positives. Most GWA studies leave a lot to be desired." For context: genome-association studies have had a history of being blown out of proportion in the press. And often for outcomes which greatly affect people's lives. Depression is one such issue and it'd be a shame if people were led into thinking there is necessarily a great breakthrough here in understanding possible gene-linkages to depression outcomes. It'd also be a shame if the result was ignored. I tried to provide some praise to this paper for being fairly rigorous, but also note that GWAS studies should be treated with caution generally. Why should GWAS studies be treated with caution generally? (Besides that results of any study should be treated with some degree of caution.) Well, firstly, GWAS is a fairly nascent field. Unlike physics or even particle physics, it hasn't had that much time to mature. This is doubly so when applying GWAS to mental health. I'm sure the paper covers some of these risks (or at least it hopefully does)... but relying on self-reporting introduces potential selection bias in the population sample, as does using people seeking help vs a general population study. Quickly reading the paper, it looks like it relied on self-reports and analyzed only people who'd been diagnosed with major depression (meaning they'd sought help). We should be cautious in over-generalizing based on this. Secondly, GWAS has had a rough history of overstating results and misapplying analyses. It's much better today than it was even, say, 5 years ago. Ioannidis and others made some heroic efforts to convince the field to clean up it's act, in effect starting ~7-8 years ago. Thirdly, there is a historical pattern of results in this field being overhyped in the press. Finally, there are active and sometimes heated debates in the field about how best to do GWAS. This is getting worse as a high-throughput, low-cost full-genome sequencing comes online to a greater and greater degree and SNP-based data sets fall by the wayside. Some question taking a frequentist approach at all in the face of such a huge degree of multiple testing. Others call for much, much higher requirements for holdout data sets, cross-validation, and replication before a study is published or considered final, especially when dealing with things like mental health (and their likely application to the field of pharmacology). This is serious stuff that could end up affecting people's mental health treatments and lives, so caution is warranted. Especially given the field's relative nascence, self-admitted history of publishing low-quality results, the rapidly changing techniques, and the fact that there are ongoing debates within the field of how best to do GWAS analysis and how to effectively replicate results.

erdevs 10y ago
Here is an example of the sort of press reaction I'm referring to (this isn't a particularly egregious example, but it demonstrates the point): https://www.washingtonpost.com/news/to-your-health/wp/2016/08/01/large-dna-study-using-23andme-data-finds-15-sites-linked-to-depression/ https://www.washingtonpost.com/news/to-your-health/wp/2016/0... Look at the language there. Scientists "pinpointed 15 locations in our DNA that are associated with depression..." [emphases added]. There is no sense of nuance or any caution in conclusion here. Even reading the whole article, which perhaps most people won't even bother with, you don't get much sense that there's any degree of uncertainty here. There is no indication that the field at large is still wrestling with how best to analyze SNP studies at all, despite publishing studies "practically ever week". There is little qualification around the data set here (it only briefly mentions that 23andme's data is based on saliva, not blood or other cells... which may be problematic. But also it should be noted that disease outcomes in this study were self-reported and from a self-selected subset of the general population who sought professional help... and there are many, many other nuances to consider). Instead, we get the fairly flat-out impression that this is a definitive discovery and, not only that, but it is of huge significance in the field as well. It may well be. But caution is warranted until time and replication hopefully do their work. Unfortunately, this is a pattern in GWAS studies and their public relations. And while the underlying methodology in the field has improved tremendously over the past 5 years, there is still a lot of debate within the field about how much more improvement needs to be made (on data sources, on analytic techniques, on review standards), and how to adapt to forthcoming changes in available study data. I think this study appears to be more robust than many others. So, I'm not picking on it. In fact I gave it particular praise for being relatively robust compared to some other GWASs. But I do think generally people should be more cautious about interpreting GWA studies or over-generalizing them.