6 ms·

I came to the biological sciences trained as a chemist. As I began working on my Ph.D. and variously encountered papers on cell signaling research (the field th

I came to the biological sciences trained as a chemist. As I began working on my Ph.D. and variously encountered papers on cell signaling research (the field that much "cancer research" would fall into) it was blindingly apparent to me, perhaps because of my chemistry background, what was going on... Cell Signaling (and much of Biology) research is in its Alchemy phase. Now, alchemy often gets a bad rap as being completely worthless, snake-oil type stuff, but this was not the case at all. Rather, individuals were working on an area about which almost nothing was known, and (more importantly) for which the key central organizing laws were not yet revealed (for Alchemy: the atomic theory of matter and chemical bonding, for Cell Signaling: how individual proteins and small molecules interact). Furthermore, the goals were lofty and almost certainly unattainable (for Alchemy: turn base metals into gold, for Cell Signaling: cure cancer), driving people to do "rush" work. It's not that the results are completely invalid, or that the experiments are useless. It's just that everyone feels like they're so close to a solution (Alchemists were way off with Phlogiston, and I'll bet Cell Signaling researchers are similarly clueless as to what really matters) that no one takes the time to step back and synthesize the results in an attempt to understand the forest from the trees. Cell Signaling will, eventually, have its Joseph Priestley, its Dmitri Mendeleev. From experience, the fact that 89% of these "cancer research" papers are not reproducible almost definitely has less to do with fraudulent data and so much more to do with crazy complex experimental setups that end up probing a half-dozen experimental variables all at once (without the researchers even grasping that this is going on). Yeah, publish or perish sucks, but what sucks more is the death of basic science research. The Alchemists eventually became Chemists because they refocused on core principles (atomic theory, bonding) and forgot about the lofty goals (turn lead into gold)... ...but try telling any politician that they should decrease cancer research funding and refocus on genetics, structural biology, and evolution research. I'd love to know how they respond.

jpdoctor 13y ago
> It's not that the results are completely invalid Especially if only 11% of them are reproducible, they're only 89% invalid. I'd go so far to say: If the reproducibility rate is 11%, you're not doing science, you're just pursuing funding.
- epistasis 13y ago
  You're assuming that only "reproducible" results are valid, which is incorrect. Results that are reproducible within a lab, but not when attempted by other people in different settings indicate that there's an unaccounted for variable, and that falls into the 89%. There's still something valuable there, it's just that we don't yet know all the variables. We could just give up, or we could try to continue study of something incredibly complex. Given that we are gaining some ground, it's clearly not a useless endeavor, it's just an extremely difficult one.
  - spikels 13y ago
    If its not reproducible it is not science.
    - bargl 13y ago
      https://news.ycombinator.com/item?id=5680310 https://news.ycombinator.com/item?id=5680310
    - epistasis 13y ago
      Define reproducible, in the terms of the precise actions that people take. Is it reproducible if the same scientist repeats the experiment in the same lab and gets the same results? Because that's the current bar of reproducibility, and that 89% that is not "reproducible" certainly passed that bar. What was being tested was a different lab, with different materials, trying to get the "same" results, for some definition of same. If you give 100 programmers an algorithms book, and tell them to produce code for a binary search, and only 25% of the programmers are able to make something that works, does that mean that binary search is only 25% reproducible? If five different companies benchmark five different web frameworks for their application, and come to 2-4 different answers about which one is the 'best,' does that mean that the benchmarks are not reproducible? Of course not. What's being highlighted here in this study is the extreme diversity of biological models. And one doesn't necessarily expect exact reproducibility in other people's hands, because we simply don't have technology to characterize every single aspect of a biological model, and it's impossible sometimes to even recreate the exact same biological context. Is something "reproducible" if it means that it replicates in 5% of other cell lines, 25% of other cell lines?
zeteo 13y ago
>Cell Signaling will, eventually, have its Joseph Priestley, its Dmitri Mendeleev. That's very optimistic, but not necessarily true (even ignoring the question of when). Priestley succeeded so well because he developed simple, reproducible experiments (expose mercury oxide to sunlight to produce oxygen etc.) that he was happy to share for no financial gain (even traveling to others' labs to help them reproduce his work [1]). The current environment of paywalls and competitive grants seems hardly conducive to the rise of similar figures. [1] http://en.wikipedia.org/wiki/Experiments_and_Observations_on_Different_Kinds_of_Air#Discovery_of_oxygen http://en.wikipedia.org/wiki/Experiments_and_Observations_on...
- jacoblyles 13y ago
  I don't understand why science should be a closed guild in the first place.
  - enraged_camel 13y ago
    Because these days it is institutionalized, and in our society that means it is funded by capitalistic entities, i.e. corporations. In other words, the profit motive dictates that making money to enrich a few individuals is more noble than sharing knowledge for the good of society.
    - anologwintermut 13y ago
      Science is funded entirely by corporations? Tell me, where can I buy stock in the NSF, DARPA, NIH? Or where can I find the articles of incorporation for the MacArthur Foundation or any of the other non-profits that also fund research. Or which corporation is paying my PhD stipend --- I really ought to short their stock since they were dumb enough to hire me.
      - yaok 13y ago
        Those are government entities, and the government's parent company is the corporatocracy that owns all the politicians and media. Science is a business in the service of global capitalism. Truth hurts. I will give you the MacArthur foundation and certain other non-profits. But most non-profits are also operated by the corporatocratic elite, like everything else on this planet.
Florin_Andrei 13y ago
So, basically, it's a combination between: A) A really hard problem to solve; and B) A heck of a lot of pressure to get it solved.
joe_the_user 13y ago
This is a pretty dubious justification. Whether or not Alchemy lead to later easily reproducible results, the alchemists couldn't have engaged in an effort to present themselves as what we now know as scientists. Oppositely, it seems logical to expect that a parade of fake and often impressive results would act as a damper to real, modest gains getting attention and funding.
InclinedPlane 13y ago
I keep coming back to this essay called "Can a Biologist Fix a Radio?": http://protein.bio.msu.ru/biokhimiya/contents/v69/pdf/bcm_1403.pdf http://protein.bio.msu.ru/biokhimiya/contents/v69/pdf/bcm_14... The thing that keeps coming to mind is that it's easy to hand-wave pseudo-understanding of systems using enough jargon and vague diagrams. And if you use enough jargon and cloak things behind complex methodologies (the mechanisms to even begin investigating how cellular machinery works are enormously complex) it's easy to hide a lack of understanding. And I can't help think a similar sort of problem exists in software. The tools and terminology we have for explaining and understanding software systems leaves a lot to be desired, and one of the ways that is revealed is in the diagrams we use for describing systems. At a fairly fine grain we can use UML diagrams, but these tell us almost nothing and in complex systems simply become a giant hairball. Typically we end up describing a system as a series of layers, or as an interconnected set of services, but I can't shake the notion that such diagrams are too much like Figure 3a in the essay above than like 3b, they are the crudest sketch of a system, they don't touch on the nature or interrelation of the components.

jpdoctor 13y ago

> It's not that the results are completely invalid Especially if only 11% of them are reproducible, they're only 89% invalid. I'd go so far to say: If the reproducibility rate is 11%, you're not doing science, you're just pursuing funding.

epistasis 13y ago
You're assuming that only "reproducible" results are valid, which is incorrect. Results that are reproducible within a lab, but not when attempted by other people in different settings indicate that there's an unaccounted for variable, and that falls into the 89%. There's still something valuable there, it's just that we don't yet know all the variables. We could just give up, or we could try to continue study of something incredibly complex. Given that we are gaining some ground, it's clearly not a useless endeavor, it's just an extremely difficult one.
- spikels 13y ago
  If its not reproducible it is not science.
  - bargl 13y ago
    https://news.ycombinator.com/item?id=5680310 https://news.ycombinator.com/item?id=5680310
  - epistasis 13y ago
    Define reproducible, in the terms of the precise actions that people take. Is it reproducible if the same scientist repeats the experiment in the same lab and gets the same results? Because that's the current bar of reproducibility, and that 89% that is not "reproducible" certainly passed that bar. What was being tested was a different lab, with different materials, trying to get the "same" results, for some definition of same. If you give 100 programmers an algorithms book, and tell them to produce code for a binary search, and only 25% of the programmers are able to make something that works, does that mean that binary search is only 25% reproducible? If five different companies benchmark five different web frameworks for their application, and come to 2-4 different answers about which one is the 'best,' does that mean that the benchmarks are not reproducible? Of course not. What's being highlighted here in this study is the extreme diversity of biological models. And one doesn't necessarily expect exact reproducibility in other people's hands, because we simply don't have technology to characterize every single aspect of a biological model, and it's impossible sometimes to even recreate the exact same biological context. Is something "reproducible" if it means that it replicates in 5% of other cell lines, 25% of other cell lines?

zeteo 13y ago

>Cell Signaling will, eventually, have its Joseph Priestley, its Dmitri Mendeleev. That's very optimistic, but not necessarily true (even ignoring the question of when). Priestley succeeded so well because he developed simple, reproducible experiments (expose mercury oxide to sunlight to produce oxygen etc.) that he was happy to share for no financial gain (even traveling to others' labs to help them reproduce his work [1]). The current environment of paywalls and competitive grants seems hardly conducive to the rise of similar figures. [1] http://en.wikipedia.org/wiki/Experiments_and_Observations_on_Different_Kinds_of_Air#Discovery_of_oxygen http://en.wikipedia.org/wiki/Experiments_and_Observations_on...

jacoblyles 13y ago
I don't understand why science should be a closed guild in the first place.
- enraged_camel 13y ago
  Because these days it is institutionalized, and in our society that means it is funded by capitalistic entities, i.e. corporations. In other words, the profit motive dictates that making money to enrich a few individuals is more noble than sharing knowledge for the good of society.
  - anologwintermut 13y ago
    Science is funded entirely by corporations? Tell me, where can I buy stock in the NSF, DARPA, NIH? Or where can I find the articles of incorporation for the MacArthur Foundation or any of the other non-profits that also fund research. Or which corporation is paying my PhD stipend --- I really ought to short their stock since they were dumb enough to hire me.
    - yaok 13y ago
      Those are government entities, and the government's parent company is the corporatocracy that owns all the politicians and media. Science is a business in the service of global capitalism. Truth hurts. I will give you the MacArthur foundation and certain other non-profits. But most non-profits are also operated by the corporatocratic elite, like everything else on this planet.

Florin_Andrei 13y ago

So, basically, it's a combination between: A) A really hard problem to solve; and B) A heck of a lot of pressure to get it solved.

joe_the_user 13y ago

This is a pretty dubious justification. Whether or not Alchemy lead to later easily reproducible results, the alchemists couldn't have engaged in an effort to present themselves as what we now know as scientists. Oppositely, it seems logical to expect that a parade of fake and often impressive results would act as a damper to real, modest gains getting attention and funding.

InclinedPlane 13y ago

I keep coming back to this essay called "Can a Biologist Fix a Radio?": http://protein.bio.msu.ru/biokhimiya/contents/v69/pdf/bcm_1403.pdf http://protein.bio.msu.ru/biokhimiya/contents/v69/pdf/bcm_14... The thing that keeps coming to mind is that it's easy to hand-wave pseudo-understanding of systems using enough jargon and vague diagrams. And if you use enough jargon and cloak things behind complex methodologies (the mechanisms to even begin investigating how cellular machinery works are enormously complex) it's easy to hide a lack of understanding. And I can't help think a similar sort of problem exists in software. The tools and terminology we have for explaining and understanding software systems leaves a lot to be desired, and one of the ways that is revealed is in the diagrams we use for describing systems. At a fairly fine grain we can use UML diagrams, but these tell us almost nothing and in complex systems simply become a giant hairball. Typically we end up describing a system as a series of layers, or as an interconnected set of services, but I can't shake the notion that such diagrams are too much like Figure 3a in the essay above than like 3b, they are the crudest sketch of a system, they don't touch on the nature or interrelation of the components.