5 ms·

In a world were 47 of 53 landmark cancer research papers were found to not be replicable... where one scientist admitted (after the meta-researchers tried 50 ti

In a world were 47 of 53 landmark cancer research papers were found to not be replicable... where one scientist admitted (after the meta-researchers tried 50 times to replicate the result and failed) that they had run the experiment 6 times and got the result once and reported that because it made a better "story"... where the act of suppressing refutations of other's bogus research is "culturally commonplace"... Something is rotten in the core of science. These days I give the research the same weight I give the anecdote... and that is no weight at all. Science has been supplanted by money and politics... At least anecdotes admit they're anecdotes! http://news.yahoo.com/cancer-science-many-discoveries-dont-hold-174216262.html http://news.yahoo.com/cancer-science-many-discoveries-dont-h...

Peaker 14y ago
Surely you could find some scientists or scientific institutions you can trust? Or maybe you can trust those experiments that do get replicated successfully?
- JohnHaugeland 14y ago
  Given that the context is "downvote anything that Reddit or HN disagree with," it seems that the point is "don't follow flocks while voting; make your own decisions or don't vote."
bad_user 14y ago
The problem with much of modern medicine is that much of it is based on flawed and biased statistical studies. Whether this is done because medical personnel don't have training in statistics, or because such studies generate funding, I don't know, but something is definitely rotten. Let's take anything involving nutrition. Some challenges are: (1) people lie, (2) such studies can't be double-blind so placebo kicks in, (3) the statistical significance of short-term studies is zero, (4) you can't control all the variables, unless you lock those people in a cage and (5) most conclusions of such studies have the potential to confuse the cause and the effect. But not all of science is like that. Just medicine.
- phren0logy 14y ago
  When you cannot control all the variables, it's important to have a large enough sample that the randomness in each direction for the different variables essentially cancels itself out. Also what does "the statistical significance of short-term studies is zero" mean? I don't think it means what you think it means. I would argue that short-term studies (for nutrition anyway) have little clinical significance, despite their statistical significance. I'm in medicine, and I read papers all the time detecting a statistical difference between control and experimental groups, but the difference is so tiny that it's meaningless. This is the balance you have to strike with large sample sizes. With a large enough sample, small differences are likely to be statistically significant but the key is determining if the difference is worthwhile. I blame bad science reporting for a lot of the anger you are feeling. Reporters don't seem to understand what they are reporting, and often the scientists themselves are (accidentally or on purpose) making it worse.
  - scalable 14y ago
    Not even a large sample will help against systematic errors.
    - phren0logy 14y ago
      Good point; upvoted. However I was just trying to address the idea that there are ways of minimizing problems in study design. No study is ever perfect, but many of them are sufficient.
  - beagle3 14y ago
    > When you cannot control all the variables, it's important to have a large enough sample that the randomness in each direction for the different variables essentially cancels itself out. That's nice in theory, but does not happen for most published research. > I'm in medicine, and I read papers all the time detecting a statistical difference between control and experimental groups, but the difference is so tiny that it's meaningless. I'm trained in statistics. My ex was an MD. I used to read the NEJM for fun for a couple of years. Most of the results published are barely statistically significant for the small group they tested ("our sample included 40 caucasian females between the ages of 37 and 48, and we have a p value of 0.03" with no mention of the context which might make that p value meaningless - but let's assume they got that part right). And then, a couple of years later, some other study takes that result as absolute truth, but assumes it applies to any woman aged >30. And a couple of years later, it is assumed to be universal and speculated to apply to males as well. Is your experience different? > I blame bad science reporting for a lot of the anger you are feeling. I blame tenure publishing requirements. While bad reporting certainly deserves its share of contempt, people these days do everything in order to meet the publishing requirements for tenure. Most stay away from outright fabrication, but otherwise every manipulation of the data that would make it fit for a higher caliber publication is being done as long as it is not outright fraudulent -- including dropping the background context so nicely exemplified by this xkcd comic http://xkcd.com/882/ http://xkcd.com/882/ . It often is the researchers doing the bad reporting with no outside help.
phren0logy 14y ago
This is a terrible perspective. I would argue that if even those research papers could not be replicated, an anecdote is all but worthless.
- JoeAltmaier 14y ago
  An anecdote, if true, is a contrary example which can be useful. It can even disprove an absolute claim - it only takes one counterexample. In mathematics this is done all the time. Statistics are themselves misleading - there are whole books on the subject (oh no! an anecdote! better close your mind now). They are highly contextual, but the popular press excels are stripping that context and proclaiming absurd extremes. Anecdotes are excellent context, putting statistics into perspective.
  - derleth 14y ago
    > oh no! an anecdote! better close your mind now Another idiotic strawman argument.
    - JoeAltmaier 14y ago
      Forgot whimsy doesn't play on HN - too many literal thinkers. Sorry, I'll refrain.
      - scalable 14y ago
        Jokes such as those are really bad for a good discussion. They serve to prove to people on your "side" (an idiotic notion, but I digress), how stupid the people on the other side are. I would downvote if I could.
        JoeAltmaier 14y ago
        Sorry again; I commented while in an excited emotional state. Though it embarrasses me how many times that give better karma than a reasoned paragraph putting forth a cogent argument.
lclarkmichalek 14y ago
Well the one nice thing about scientific research is that it's open (cue conspiracy theories), and that you can usually read the paper/draft. No one is asking you to "just believe" research; that would be stupid, and it's not how the scientific process works. The fact that you are dismissing all research outright is ridiculous, and willfully ignorant. Your sources are shabby, and you talk of "science" and "scientists" as some great collective, which is frankly impossible to do. The idea that "Something is rotten in the core of science" is a wild extrapolation; there are problems with certain fields' practices (and may always be), medicine is probably a good example. However, saying that you give research no weight is stupid, and you discredit yourself further when you say that "Science has been supplanted by money and politics"; you sound like a conspiracy theorist, and I cannot believe you actually know many academics who will back you up on that claim. For whatever the system's flaws, there is _no_ reason to say that all research cannot be trusted.
- beagle3 14y ago
  > No one is asking you to "just believe" research; Did you read the study the parent post is talking about? A well funded laboratory, that was trying to not "just believe" research (as everyone else apparently does) was trying to replicate these results. If the science was good, all it should have taken is time and money (both of which they had enough of). And yet, 47 out of 53 celebrated results published in peer reviewed papers of the highest caliber could not be replicated . Let that sink for a minute before you reply. > there is _no_ reason to say that all research cannot be trusted. Ok. Your reason to state that research can be trusted is that it is eventually replicated (thus confirmed), or thrown out (thus shown false), is that right? (You didn't state that as your reason, so perhaps you have other ideas -- but that's a common one, so I'll reply to it). Assuming that's the case -- do you have any idea what percentage of results are replicated? And how much time after official publication? Because if it takes e.g. 30 years until a bad publication is discredited, and (as the data point given by the parent shows) there are areas in which 90% of the data apparently can be discredited when you try to replicate it -- then, there actually might be reason to distrust research in general, because at any given point in time, more than 90% of non-discredited published results are wrong. See also http://saveyourself.ca/articles/ioannidis.php http://saveyourself.ca/articles/ioannidis.php (and the paper it references). This situation is not science fiction. 90% un-replicatable publications is probably limited to very few subjects. But 50% overall in medicine and biology is totally believable. Which is not to say science (the abstract idea / discipline / method) is wrong - it's right. It's just that the things we human practice and often call "science" is very, very far from the ideal of science. Ignore that at your own peril.
ars 14y ago
> Something is rotten in the core of science. It's not science that is the problem. It's that biology considers a 95% confidence sufficient. Considering how many studies are done each year, this virtually guarantees incorrect results. The reason they do that is that it's impossible to get better results, they just can not do enough trials. So they are stuck.
- stephencanon 14y ago
  95% confidence would be fine (or very nearly fine) if a requirement for funding/publication would be that all data, including negative results, be published as well.
  - beagle3 14y ago
    That would only be true if a) all data, everywhere in the world, including negative results, was published regardless of funding/publication. b) someone actually looked at that data, normalized it, and used it to assess the real significance of every result, in a sane manner (e.g. by using a bayesian inference with some reasonably behaving universal prior). Neither a, b will ever happen, and both are essential. (note: publication of all data is not a sufficient requirement: if 20 independent labs each do the same random experiment, one of them is expected to have a 95% confidence, and when they publish all their data, it consists of that one experiment that seems legit. This _will_ and _already does_ happen by chance)
LinXitoW 14y ago
If you stand for nothing, you'll fall for everything...or something like that. Science and especially statistics have never (and sadly never will be) perfect or free of human bias, but not believing anything outright leaves you open to manipulators that make you "feel" whats right, instead of having you think about whats right. We might think we're super rational, but it just ain't so. Unrelated example: Female students told before a standardized math test that its genetic that women are worse at math also did significantly worse than males and control females. They were (probably) all rational people, yet this seemingly unimportant event changed their rational performance. If science and anecdotes are equally bull to you, how do you make up your mind about things? Magic?
ktizo 14y ago
So from your example, ~10% contained really good information, and this was found using a meta study, conducted by scientists. And you are using that as an example of how you should give scientific research no weight at all? Why do you give weight to the scientific meta study if you think it means that you should not give weight to scientific studies in general? This position is pretty ridiculous on the face of it.
polyfractal 14y ago
This is utter hogwash. I'm as critical as anyone (probably more so, check my comment history) of academic biology because of my background in it. There are certainly things wrong it. And due to the nature of biology, replicating results is really hard. It's a fact of life when you deal with systems that are not perfect, not identical and very opaque. But to say that "Science has been supplanted by money and politics" is stretching the problems of biology into a mountain of conspiracy. Furthermore, I'm reading your "source" and it reads loudly as "I'm an underfunded big-pharma research who has neither the time nor the resources to properly replicate studies". Did you know that most big pharma labs do not have access to the academic literature? They mostly read abstracts because there is little budget to actually purchase the required papers. How much do you trust labs that are A) only trying to recreate data so they can make a drug out of it and B) aren't even reading the original data? While academic labs can have grad students toil away on hard experiemnts for literally years before they perfect them...how long do you think Pfizer or Merck or Glaxco-Smith is going to let their paid researchers fiddle away on a project that is probably low priority anyway? Because, of course, the high-priority projects are the reformulations of penis-enlarging drugs or cholesterol medication...you know, the ones that actually make money. If you are looking for snake oil and shady research, I dare you to read any research paper that comes out of big pharma labs. We would routinely read them just for laughs because they are (often) downright terrible.
- refurb 14y ago
  While I agree that "Science has been supplanted by money and politics" is going way overboard. I'm going to take the other side of the argument because I think you are way off base. I worked in both an academic lab and a "big pharma" lab(4 years and 6 years respectively). To say "most big pharma labs" do not have access to the literature is laughable. We had better access than most academic institutions. If we needed a paper we didn't have access to, it took a few hours to get it. The company was more than willing to pay the $50 to get a copy of whatever paper, since we would often blow $50 running one experiment. Many of the smaller biotech might have poor access to journals, but even then, if you could justify the cost, you could get it. Second of all, yes I trust labs that are trying to recreate data to make a drug out of it. You have to remember that these attempts to recreate data were a very important data point on a potential multi-million (billion?) dollar investment in a new target, these are NOT low priority projects. They WANT the data to be true. They have zero incentive for the data to not be reproducible. Having worked in both academic and commercial labs, I would say the incentive to "tweak" results in much great in academic labs for the following reasons: 1) Often results are never double checked in an academic lab unless the work is use in a later project. Contrast this with a pharma lab where if the data is positive, you'll have to prove it again and again. 2) Academics (both profs and students) live and die by papers, not so in academic (in fact, in the company I worked in, they preferred if you didn't publish) 3) Work in academic is often performed by relatively inexperienced ungrad and grad students, while big pharma scientists often have years of experience.
guelo 14y ago
Stop using anecdotes to support extremely broad statements about all of science.

Peaker 14y ago

Surely you could find some scientists or scientific institutions you can trust? Or maybe you can trust those experiments that do get replicated successfully?

JohnHaugeland 14y ago
Given that the context is "downvote anything that Reddit or HN disagree with," it seems that the point is "don't follow flocks while voting; make your own decisions or don't vote."

bad_user 14y ago

The problem with much of modern medicine is that much of it is based on flawed and biased statistical studies. Whether this is done because medical personnel don't have training in statistics, or because such studies generate funding, I don't know, but something is definitely rotten. Let's take anything involving nutrition. Some challenges are: (1) people lie, (2) such studies can't be double-blind so placebo kicks in, (3) the statistical significance of short-term studies is zero, (4) you can't control all the variables, unless you lock those people in a cage and (5) most conclusions of such studies have the potential to confuse the cause and the effect. But not all of science is like that. Just medicine.

phren0logy 14y ago
When you cannot control all the variables, it's important to have a large enough sample that the randomness in each direction for the different variables essentially cancels itself out. Also what does "the statistical significance of short-term studies is zero" mean? I don't think it means what you think it means. I would argue that short-term studies (for nutrition anyway) have little clinical significance, despite their statistical significance. I'm in medicine, and I read papers all the time detecting a statistical difference between control and experimental groups, but the difference is so tiny that it's meaningless. This is the balance you have to strike with large sample sizes. With a large enough sample, small differences are likely to be statistically significant but the key is determining if the difference is worthwhile. I blame bad science reporting for a lot of the anger you are feeling. Reporters don't seem to understand what they are reporting, and often the scientists themselves are (accidentally or on purpose) making it worse.
- scalable 14y ago
  Not even a large sample will help against systematic errors.
  - phren0logy 14y ago
    Good point; upvoted. However I was just trying to address the idea that there are ways of minimizing problems in study design. No study is ever perfect, but many of them are sufficient.
- beagle3 14y ago
  > When you cannot control all the variables, it's important to have a large enough sample that the randomness in each direction for the different variables essentially cancels itself out. That's nice in theory, but does not happen for most published research. > I'm in medicine, and I read papers all the time detecting a statistical difference between control and experimental groups, but the difference is so tiny that it's meaningless. I'm trained in statistics. My ex was an MD. I used to read the NEJM for fun for a couple of years. Most of the results published are barely statistically significant for the small group they tested ("our sample included 40 caucasian females between the ages of 37 and 48, and we have a p value of 0.03" with no mention of the context which might make that p value meaningless - but let's assume they got that part right). And then, a couple of years later, some other study takes that result as absolute truth, but assumes it applies to any woman aged >30. And a couple of years later, it is assumed to be universal and speculated to apply to males as well. Is your experience different? > I blame bad science reporting for a lot of the anger you are feeling. I blame tenure publishing requirements. While bad reporting certainly deserves its share of contempt, people these days do everything in order to meet the publishing requirements for tenure. Most stay away from outright fabrication, but otherwise every manipulation of the data that would make it fit for a higher caliber publication is being done as long as it is not outright fraudulent -- including dropping the background context so nicely exemplified by this xkcd comic http://xkcd.com/882/ http://xkcd.com/882/ . It often is the researchers doing the bad reporting with no outside help.

phren0logy 14y ago

This is a terrible perspective. I would argue that if even those research papers could not be replicated, an anecdote is all but worthless.

JoeAltmaier 14y ago
An anecdote, if true, is a contrary example which can be useful. It can even disprove an absolute claim - it only takes one counterexample. In mathematics this is done all the time. Statistics are themselves misleading - there are whole books on the subject (oh no! an anecdote! better close your mind now). They are highly contextual, but the popular press excels are stripping that context and proclaiming absurd extremes. Anecdotes are excellent context, putting statistics into perspective.
- derleth 14y ago
  > oh no! an anecdote! better close your mind now Another idiotic strawman argument.
  - JoeAltmaier 14y ago
    Forgot whimsy doesn't play on HN - too many literal thinkers. Sorry, I'll refrain.
    - scalable 14y ago
      Jokes such as those are really bad for a good discussion. They serve to prove to people on your "side" (an idiotic notion, but I digress), how stupid the people on the other side are. I would downvote if I could.
      - JoeAltmaier 14y ago
        Sorry again; I commented while in an excited emotional state. Though it embarrasses me how many times that give better karma than a reasoned paragraph putting forth a cogent argument.

lclarkmichalek 14y ago

Well the one nice thing about scientific research is that it's open (cue conspiracy theories), and that you can usually read the paper/draft. No one is asking you to "just believe" research; that would be stupid, and it's not how the scientific process works. The fact that you are dismissing all research outright is ridiculous, and willfully ignorant. Your sources are shabby, and you talk of "science" and "scientists" as some great collective, which is frankly impossible to do. The idea that "Something is rotten in the core of science" is a wild extrapolation; there are problems with certain fields' practices (and may always be), medicine is probably a good example. However, saying that you give research no weight is stupid, and you discredit yourself further when you say that "Science has been supplanted by money and politics"; you sound like a conspiracy theorist, and I cannot believe you actually know many academics who will back you up on that claim. For whatever the system's flaws, there is _no_ reason to say that all research cannot be trusted.

beagle3 14y ago
> No one is asking you to "just believe" research; Did you read the study the parent post is talking about? A well funded laboratory, that was trying to not "just believe" research (as everyone else apparently does) was trying to replicate these results. If the science was good, all it should have taken is time and money (both of which they had enough of). And yet, 47 out of 53 celebrated results published in peer reviewed papers of the highest caliber could not be replicated . Let that sink for a minute before you reply. > there is _no_ reason to say that all research cannot be trusted. Ok. Your reason to state that research can be trusted is that it is eventually replicated (thus confirmed), or thrown out (thus shown false), is that right? (You didn't state that as your reason, so perhaps you have other ideas -- but that's a common one, so I'll reply to it). Assuming that's the case -- do you have any idea what percentage of results are replicated? And how much time after official publication? Because if it takes e.g. 30 years until a bad publication is discredited, and (as the data point given by the parent shows) there are areas in which 90% of the data apparently can be discredited when you try to replicate it -- then, there actually might be reason to distrust research in general, because at any given point in time, more than 90% of non-discredited published results are wrong. See also http://saveyourself.ca/articles/ioannidis.php http://saveyourself.ca/articles/ioannidis.php (and the paper it references). This situation is not science fiction. 90% un-replicatable publications is probably limited to very few subjects. But 50% overall in medicine and biology is totally believable. Which is not to say science (the abstract idea / discipline / method) is wrong - it's right. It's just that the things we human practice and often call "science" is very, very far from the ideal of science. Ignore that at your own peril.

ars 14y ago

> Something is rotten in the core of science. It's not science that is the problem. It's that biology considers a 95% confidence sufficient. Considering how many studies are done each year, this virtually guarantees incorrect results. The reason they do that is that it's impossible to get better results, they just can not do enough trials. So they are stuck.

stephencanon 14y ago
95% confidence would be fine (or very nearly fine) if a requirement for funding/publication would be that all data, including negative results, be published as well.
- beagle3 14y ago
  That would only be true if a) all data, everywhere in the world, including negative results, was published regardless of funding/publication. b) someone actually looked at that data, normalized it, and used it to assess the real significance of every result, in a sane manner (e.g. by using a bayesian inference with some reasonably behaving universal prior). Neither a, b will ever happen, and both are essential. (note: publication of all data is not a sufficient requirement: if 20 independent labs each do the same random experiment, one of them is expected to have a 95% confidence, and when they publish all their data, it consists of that one experiment that seems legit. This _will_ and _already does_ happen by chance)

LinXitoW 14y ago

If you stand for nothing, you'll fall for everything...or something like that. Science and especially statistics have never (and sadly never will be) perfect or free of human bias, but not believing anything outright leaves you open to manipulators that make you "feel" whats right, instead of having you think about whats right. We might think we're super rational, but it just ain't so. Unrelated example: Female students told before a standardized math test that its genetic that women are worse at math also did significantly worse than males and control females. They were (probably) all rational people, yet this seemingly unimportant event changed their rational performance. If science and anecdotes are equally bull to you, how do you make up your mind about things? Magic?

ktizo 14y ago

So from your example, ~10% contained really good information, and this was found using a meta study, conducted by scientists. And you are using that as an example of how you should give scientific research no weight at all? Why do you give weight to the scientific meta study if you think it means that you should not give weight to scientific studies in general? This position is pretty ridiculous on the face of it.

polyfractal 14y ago

This is utter hogwash. I'm as critical as anyone (probably more so, check my comment history) of academic biology because of my background in it. There are certainly things wrong it. And due to the nature of biology, replicating results is really hard. It's a fact of life when you deal with systems that are not perfect, not identical and very opaque. But to say that "Science has been supplanted by money and politics" is stretching the problems of biology into a mountain of conspiracy. Furthermore, I'm reading your "source" and it reads loudly as "I'm an underfunded big-pharma research who has neither the time nor the resources to properly replicate studies". Did you know that most big pharma labs do not have access to the academic literature? They mostly read abstracts because there is little budget to actually purchase the required papers. How much do you trust labs that are A) only trying to recreate data so they can make a drug out of it and B) aren't even reading the original data? While academic labs can have grad students toil away on hard experiemnts for literally years before they perfect them...how long do you think Pfizer or Merck or Glaxco-Smith is going to let their paid researchers fiddle away on a project that is probably low priority anyway? Because, of course, the high-priority projects are the reformulations of penis-enlarging drugs or cholesterol medication...you know, the ones that actually make money. If you are looking for snake oil and shady research, I dare you to read any research paper that comes out of big pharma labs. We would routinely read them just for laughs because they are (often) downright terrible.

refurb 14y ago
While I agree that "Science has been supplanted by money and politics" is going way overboard. I'm going to take the other side of the argument because I think you are way off base. I worked in both an academic lab and a "big pharma" lab(4 years and 6 years respectively). To say "most big pharma labs" do not have access to the literature is laughable. We had better access than most academic institutions. If we needed a paper we didn't have access to, it took a few hours to get it. The company was more than willing to pay the $50 to get a copy of whatever paper, since we would often blow $50 running one experiment. Many of the smaller biotech might have poor access to journals, but even then, if you could justify the cost, you could get it. Second of all, yes I trust labs that are trying to recreate data to make a drug out of it. You have to remember that these attempts to recreate data were a very important data point on a potential multi-million (billion?) dollar investment in a new target, these are NOT low priority projects. They WANT the data to be true. They have zero incentive for the data to not be reproducible. Having worked in both academic and commercial labs, I would say the incentive to "tweak" results in much great in academic labs for the following reasons: 1) Often results are never double checked in an academic lab unless the work is use in a later project. Contrast this with a pharma lab where if the data is positive, you'll have to prove it again and again. 2) Academics (both profs and students) live and die by papers, not so in academic (in fact, in the company I worked in, they preferred if you didn't publish) 3) Work in academic is often performed by relatively inexperienced ungrad and grad students, while big pharma scientists often have years of experience.

guelo 14y ago

Stop using anecdotes to support extremely broad statements about all of science.