4 ms·

Robust research needs many lines of evidence

51 points by lainon 9y ago

nonbel 9y ago
This paper really sounds like it advocates the status quo of attempting to take the "sum of the evidence", which usually leads to the generation of many different weak (low sample size, etc) lines of evidence. They don't seem to get that what made science so successful was its ability to generate simpler models of phenomenon. To do this you need to come up with (most often quantitative) theories that can synthesize various lines of evidence. The practice of "counting heads" seen in fields like medicine is totally inappropriate (eg 2 out of 3 results support the theory so it looks good). The correct thing to do is more like what is seen in physics. Eg, take the idea that "cancer is due to mutations that accumulate as cells divide, therefore cancer rates should be related to number of divisions and mutation rate." The status quo is to do something like: 1) "cancer is positively correlated with number of divisions" 2) "cancer is positively correlated with mutation rate" 3) "cancer is positively correlated with number of accumulated mutations" Instead, we should say "this means the probability of accumulating at least n mutations after d divisions given mutation rate r should follow a law similar to": p(x >= n) = (1 - (1 - r)^d)^n The two approaches are similar in that you measure mutation rate, number of divisions, accumulated mutations and frequency of cancer cells. The difference is that for the latter approach all the lines of evidence combine in a nonlinear fashion. This method allows us to subsume a myriad of factors such as genetic disposition, exposure to toxins, etc to be incorporated via a single value (mutation rate). So rather than an ever expanding apparent complexity as more factors are incorporated, there is an accumulation of knowledge as we estimate the model parameters more and more accurately and either reject, modify, or keep our models of the phenomenon.
- SiempreViernes 9y ago
  Which paper are you talking about? Do you actually mean the linked article? Because it contains the following: > But replication alone will get us only so far. In some cases, routine replication might actually make matters worse. Consistent findings could take on the status of confirmed truths, when they actually reflect failings in study design, methods or analytical tools. This is a strong criticism against the counting heads method you also object to.
  - nonbel 9y ago
    Look at their example. Theory: "One example in which triangulation has helped is in establishing that smoking during pregnancy results in babies with lower birth weights" Lines of evidence: 1) "women who smoke are more likely to have babies who weigh less" 2) "if a woman’s partner smokes during her pregnancy, many of the same confounders apply as in maternal smoking, but the association with lower birth weight is much weaker" 3) "Birth weight can also be analysed according to levels of cigarette taxation across US states, which reduces the effects of confounders." 4) "the birth weights of siblings whose mother smoked during one pregnancy but not another." 5) "In cohorts grouped according to whether or not people carry a genetic variant associated with greater cigarette consumption in those who smoke, mothers who smoke and carry the variant tended to have babies who weighed less; non-smokers with the same variant did not." Conclusion: "Taken together, these studies make it clear that maternal smoking affects birth weight directly" This looks exactly like what I described as "counting heads". There is no synthesis of this evidence, just a disconnected list of things (and who knows whether contrary evidence has been left out or never checked).
- aaavl2821 9y ago
  that's definitely a good approach in theory, but messier in reality. if we knew enough about which variables impacted disease, and could measure them well enough, and had comprehensive quantitative models of each variable interacts with a biological system, we could do this, but we dont. the reason we have to triangulate is because the types of modeling you do in physics just doesn't translate to fields like life science / medicine there are too many unknowns in life sci and error bars are too high compared to physics to enable this kind of model to be too accurate. collecting all of that data (ie mutation rate, number of divisions, accumuated mutations etc) across whole tumors in enough people to matter would be incredibly costly and time consuming, and the result probably wouldnt be as useful as we'd hope.
  - nonbel 9y ago
    >"there are too many unknowns in life sci and error bars are too high compared to physics to enable this kind of model to be too accurate. collecting all of that data (ie mutation rate, number of divisions, accumuated mutations etc) across whole tumors in enough people to matter would be incredibly costly and time consuming, and the result probably wouldnt be as useful as we'd hope." Researchers will never cure cancer if they don't collect basic information like mutation rates, number of divisions since zygote as a function of age, cells in each tissue as a function of age, etc. It is like trying to develop astronomy without recording the positions of the planets... Also, I have seen that an only-slightly-more-complicated model than that extremely simple one (assuming a decaying curve for division rate as a function of age) is able to fit the age-specific incidence of many cancers (data from SEER). Have you checked that such approaches are too difficult or are you assuming so? Can you point me to info on any failed attempts? It really sounds like giving up to me. If that is the attitude of cancer researchers, I suspect eventually people with a less defeatist attitude will take it on (possibly coming from outside the healthcare industry).
    - aaavl2821 9y ago
      I think that new approaches to doing drug R&D are always welcome and if you know of any particular interesting and useful approaches id love to learn more. The "bottoms up" approach you mention seems like it could be really powerful and definitely complementary to whats being done now. I have however seen a lot of approaches "sort of" like that not work that well. I also know of a lot of less quantitative approaches taht actually work really well, and thats how a lot of R&D is done today I disagree w the statement that we'll never cure cancer w/o info on mutation rates. Many of the most impactful medicines we have were discovered without this kind of low level, quantitative info. Many through serendipity. Penicillin is the classic example. We have actually made huge progress in treating many cancers in the last five years. We can turn many melanomas from death sentences into relatively manageable chronic diseases, and a lot of blood cancers are being completely eradicated in some patients (at least for a time). We have a long way to go, but most of these insights come from triangulating various pieces of molecular biological and chemical information to create testable insights There has been a move lately to more "genomic" medicine where you identify a specific genentic mutation, and target that to treat disease. Most diseases aren't that simple though, and these are rarely curative There was also a move in the early 2000s to measure expression levels of hundreds of thousands of cancer genes to try to understand which ones would metastasize, how to treat them, etc. In some cases algorithms got pretty good at predicting rates of metastasis, but most didnt, and even fewer yielded clinically actionable insights. And I dont know of any drugs that were developed based on this work The idea of the moment is the "liquid biopsy", measuring pieces of DNA or proteins in the blood, and using machine learning to correlate these "biomarkers" with risk of getting cancer or other endpoints. The jury is still out on whether this works to my knowledge, though frankly i havent followed the science too closely No one is giving up, not in the least -- in fact more money is being invested into drug research than ever before. Its just really hard, and "triangulation" and insight are more powerful right now than purely quantitative methods. I hope that changes. But for things to change, we probably need better tools for collecting data before algorithms could be really powerful
aaavl2821 9y ago
Often the first thing companies do when they license tech from academia is try to reproduce the findings in a rigorous setting, rather than jumping straight into novel work. Behind closed doors, big pharma scientists complain about the quality of academic work even published in leading journals This problem is real and is probably not going to be addressed within the academic community. However, as more scientists realize that the path to running your own lab and directing your own research agenda lies in working at startups rather than in academia / big pharma, hopefully the way science is done will change In CA at least, venture capital funds more life science R&D than do NIH dollars, and probably more than big biotech (Genentech / Amgen / Gilead etc)
- SiempreViernes 9y ago
  If big pharma are fine with paying for shitty research instead of spending money on doing it themselves that's their choice I guess.
  - aaavl2821 9y ago
    That's true, but what's happening now is that startups are licensing tech from academia, making it closer to "industrial grade" in terms of scale and quality, proving the concept more definitively (usually in humans) then selling to big pharma. Big pharma does license stuff from academia all the time, but for orders of magnitude less money than what they pay for startups with good research
- novaRom 9y ago
  > Behind closed doors, big pharma scientists complain about the quality of academic work Same in Deep Learning research - 99% of papers today is just an obfuscated description of their "alchemy". Even from famous universities and companies.
  - deleted 9y ago
    [deleted]
- epistasis 9y ago
SubiculumCode 9y ago
I wanted to say that converging evidence is highly valued in neuroscience already. We know every single study is flawed in some way, but attacking the same problem from 3 angles and getting similar answers, you might really have something.
- SubiculumCode 9y ago
  And also, those three angles might not happen in a single paper, but across a series of papers, then synthesized in a review article.
- mturmon 9y ago
  Yes, you can see this in astronomy and planetary science as well. As a basic example, the wikipedia pages on dark matter and dark energy cite various sources of converging evidence. In planetary science, one example is the multiple lines of evidence that there were once significant water flows on Mars.

51 points by lainon 9y ago

This paper really sounds like it advocates the status quo of attempting to take the "sum of the evidence", which usually leads to the generation of many different weak (low sample size, etc) lines of evidence. They don't seem to get that what made science so successful was its ability to generate simpler models of phenomenon. To do this you need to come up with (most often quantitative) theories that can synthesize various lines of evidence. The practice of "counting heads" seen in fields like medicine is totally inappropriate (eg 2 out of 3 results support the theory so it looks good). The correct thing to do is more like what is seen in physics. Eg, take the idea that "cancer is due to mutations that accumulate as cells divide, therefore cancer rates should be related to number of divisions and mutation rate." The status quo is to do something like: 1) "cancer is positively correlated with number of divisions" 2) "cancer is positively correlated with mutation rate" 3) "cancer is positively correlated with number of accumulated mutations" Instead, we should say "this means the probability of accumulating at least n mutations after d divisions given mutation rate r should follow a law similar to": p(x >= n) = (1 - (1 - r)^d)^n The two approaches are similar in that you measure mutation rate, number of divisions, accumulated mutations and frequency of cancer cells. The difference is that for the latter approach all the lines of evidence combine in a nonlinear fashion. This method allows us to subsume a myriad of factors such as genetic disposition, exposure to toxins, etc to be incorporated via a single value (mutation rate). So rather than an ever expanding apparent complexity as more factors are incorporated, there is an accumulation of knowledge as we estimate the model parameters more and more accurately and either reject, modify, or keep our models of the phenomenon.

SiempreViernes 9y ago
Which paper are you talking about? Do you actually mean the linked article? Because it contains the following: > But replication alone will get us only so far. In some cases, routine replication might actually make matters worse. Consistent findings could take on the status of confirmed truths, when they actually reflect failings in study design, methods or analytical tools. This is a strong criticism against the counting heads method you also object to.
- nonbel 9y ago
  Look at their example. Theory: "One example in which triangulation has helped is in establishing that smoking during pregnancy results in babies with lower birth weights" Lines of evidence: 1) "women who smoke are more likely to have babies who weigh less" 2) "if a woman’s partner smokes during her pregnancy, many of the same confounders apply as in maternal smoking, but the association with lower birth weight is much weaker" 3) "Birth weight can also be analysed according to levels of cigarette taxation across US states, which reduces the effects of confounders." 4) "the birth weights of siblings whose mother smoked during one pregnancy but not another." 5) "In cohorts grouped according to whether or not people carry a genetic variant associated with greater cigarette consumption in those who smoke, mothers who smoke and carry the variant tended to have babies who weighed less; non-smokers with the same variant did not." Conclusion: "Taken together, these studies make it clear that maternal smoking affects birth weight directly" This looks exactly like what I described as "counting heads". There is no synthesis of this evidence, just a disconnected list of things (and who knows whether contrary evidence has been left out or never checked).
aaavl2821 9y ago
that's definitely a good approach in theory, but messier in reality. if we knew enough about which variables impacted disease, and could measure them well enough, and had comprehensive quantitative models of each variable interacts with a biological system, we could do this, but we dont. the reason we have to triangulate is because the types of modeling you do in physics just doesn't translate to fields like life science / medicine there are too many unknowns in life sci and error bars are too high compared to physics to enable this kind of model to be too accurate. collecting all of that data (ie mutation rate, number of divisions, accumuated mutations etc) across whole tumors in enough people to matter would be incredibly costly and time consuming, and the result probably wouldnt be as useful as we'd hope.
- nonbel 9y ago
  >"there are too many unknowns in life sci and error bars are too high compared to physics to enable this kind of model to be too accurate. collecting all of that data (ie mutation rate, number of divisions, accumuated mutations etc) across whole tumors in enough people to matter would be incredibly costly and time consuming, and the result probably wouldnt be as useful as we'd hope." Researchers will never cure cancer if they don't collect basic information like mutation rates, number of divisions since zygote as a function of age, cells in each tissue as a function of age, etc. It is like trying to develop astronomy without recording the positions of the planets... Also, I have seen that an only-slightly-more-complicated model than that extremely simple one (assuming a decaying curve for division rate as a function of age) is able to fit the age-specific incidence of many cancers (data from SEER). Have you checked that such approaches are too difficult or are you assuming so? Can you point me to info on any failed attempts? It really sounds like giving up to me. If that is the attitude of cancer researchers, I suspect eventually people with a less defeatist attitude will take it on (possibly coming from outside the healthcare industry).
  - aaavl2821 9y ago
    I think that new approaches to doing drug R&D are always welcome and if you know of any particular interesting and useful approaches id love to learn more. The "bottoms up" approach you mention seems like it could be really powerful and definitely complementary to whats being done now. I have however seen a lot of approaches "sort of" like that not work that well. I also know of a lot of less quantitative approaches taht actually work really well, and thats how a lot of R&D is done today I disagree w the statement that we'll never cure cancer w/o info on mutation rates. Many of the most impactful medicines we have were discovered without this kind of low level, quantitative info. Many through serendipity. Penicillin is the classic example. We have actually made huge progress in treating many cancers in the last five years. We can turn many melanomas from death sentences into relatively manageable chronic diseases, and a lot of blood cancers are being completely eradicated in some patients (at least for a time). We have a long way to go, but most of these insights come from triangulating various pieces of molecular biological and chemical information to create testable insights There has been a move lately to more "genomic" medicine where you identify a specific genentic mutation, and target that to treat disease. Most diseases aren't that simple though, and these are rarely curative There was also a move in the early 2000s to measure expression levels of hundreds of thousands of cancer genes to try to understand which ones would metastasize, how to treat them, etc. In some cases algorithms got pretty good at predicting rates of metastasis, but most didnt, and even fewer yielded clinically actionable insights. And I dont know of any drugs that were developed based on this work The idea of the moment is the "liquid biopsy", measuring pieces of DNA or proteins in the blood, and using machine learning to correlate these "biomarkers" with risk of getting cancer or other endpoints. The jury is still out on whether this works to my knowledge, though frankly i havent followed the science too closely No one is giving up, not in the least -- in fact more money is being invested into drug research than ever before. Its just really hard, and "triangulation" and insight are more powerful right now than purely quantitative methods. I hope that changes. But for things to change, we probably need better tools for collecting data before algorithms could be really powerful

aaavl2821 9y ago

Often the first thing companies do when they license tech from academia is try to reproduce the findings in a rigorous setting, rather than jumping straight into novel work. Behind closed doors, big pharma scientists complain about the quality of academic work even published in leading journals This problem is real and is probably not going to be addressed within the academic community. However, as more scientists realize that the path to running your own lab and directing your own research agenda lies in working at startups rather than in academia / big pharma, hopefully the way science is done will change In CA at least, venture capital funds more life science R&D than do NIH dollars, and probably more than big biotech (Genentech / Amgen / Gilead etc)

SiempreViernes 9y ago
If big pharma are fine with paying for shitty research instead of spending money on doing it themselves that's their choice I guess.
- aaavl2821 9y ago
  That's true, but what's happening now is that startups are licensing tech from academia, making it closer to "industrial grade" in terms of scale and quality, proving the concept more definitively (usually in humans) then selling to big pharma. Big pharma does license stuff from academia all the time, but for orders of magnitude less money than what they pay for startups with good research
novaRom 9y ago
> Behind closed doors, big pharma scientists complain about the quality of academic work Same in Deep Learning research - 99% of papers today is just an obfuscated description of their "alchemy". Even from famous universities and companies.
- deleted 9y ago
  [deleted]
epistasis 9y ago

SubiculumCode 9y ago

I wanted to say that converging evidence is highly valued in neuroscience already. We know every single study is flawed in some way, but attacking the same problem from 3 angles and getting similar answers, you might really have something.

SubiculumCode 9y ago
And also, those three angles might not happen in a single paper, but across a series of papers, then synthesized in a review article.
mturmon 9y ago
Yes, you can see this in astronomy and planetary science as well. As a basic example, the wikipedia pages on dark matter and dark energy cite various sources of converging evidence. In planetary science, one example is the multiple lines of evidence that there were once significant water flows on Mars.