7 ms·

Rethinking the scientific method

59 points by neilc 16y ago

powera 16y ago
Considering that most of these studies are by people with an interest in getting a certain result, it's likely experimental errors/not publishing some results will bias the published numbers in a certain direction. The scientific method has never been proof against people that lie by omission.
- gruseom 16y ago
  He's writing about results that were validated through replication. That is the scientific method. If what he's saying is accurate, the problem isn't a few bad apples (that's the proverbial reflexive defense of the status quo anyway). But now all sorts of well-established, multiply confirmed findings have started to look increasingly uncertain. I have no idea if it's accurate, but Lehrer's a credible journalist, less prone to the sensational than most. If he's writing about this, it's probably because serious scientists are concerned about it. (On another note, when I pasted the above, the following text showed up appended to my selection: Read more http://www.newyorker.com/reporting/2010/12/13/101213fa_fact_lehrer#ixzz17PM5acPA http://www.newyorker.com/reporting/2010/12/13/101213fa_fact_... How can such a venerable publication as the New Yorker resort to such tackiness?)
  - burgerbrain 16y ago
    He mentions, among other things,studies in ESP as falling prey to this mysterious "well it worked the first time" syndrome. Now maybe the other examples are more credible, but mixing bullshit like ESP in doesn't help your case at all.
    - michael_dorfman 16y ago
      Actually, it does. The likelihood that ESP actually works is significantly less than the likelihood of "the decline effect", which is explained by a number of perfectly rational factors (touched upon in the article.) If you want to claim that "ESP is bullshit" (as you and I both do, I imagine), you somehow need to account for those (rare) studies which oddly enough seem to support an ESP effect at a statistically significant level. "The decline effect" goes a ways toward providing this explanation.
      - drndown2007 16y ago
        This article is one example (pre-cognition as opposed to ESP, but in the same vein): http://hplusmagazine.com/editors-blog/precognition-real-cornell-university-lab-releases-powerful-new-evidence-human-mind-can- http://hplusmagazine.com/editors-blog/precognition-real-corn... I think it comes down to we don't know nearly as much about our little existence as we like to think.
      - khafra 16y ago
        In other words, parapsychology is the control group for science: http://lesswrong.com/lw/1ib/parapsychology_the_control_group_for_science/ http://lesswrong.com/lw/1ib/parapsychology_the_control_group...
  - St-Clock 16y ago
    "But now all sorts of well-established, multiply confirmed findings have started to look increasingly uncertain." Good! That's what science is all about. Making theories that can later be "disproved" or "confirmed". It's when you cannot disprove something that it is no longer science.
    - drndown2007 16y ago
      This means nothing can ever be confirmed though -- you never know if you'll learn something later to disprove what you think you know now. Which in turn means a certain level of humility is required. To me, that's where science falls short -- anything that doesn't fit what is currently 'known' and 'proven' is not taken seriously.
  - bugsy 16y ago
    The replication they are talking about in these particular pharma studies are of the nature that a pharma corp orders up twelve identical 8 person studies. Three studies come back showing their new pill helped a tiny bit, six show no the pill did nothing, and three show the patients got worse. They then cherry pick the three that showed it helped, and possibly toss in one of the ones that showed the pill did nothing just to cover up what they are doing. They then publish a paper showing that 3 out of 4 studies validated that the pills work. The other 8 studies are set on fire and never mentioned. And there is your multiple studies. This is not some crazed conspiracy theory either, this (doing multiple very small sample size studies rather than one slightly larger study and discarding and never mentioning some of the studies with the least favorable results) is actually how it is now known to be done, and this methodology is the reason why some journals have started to say that all of the studies have to be registered in advance if they want to publish their results so that the companies can't selectively discard results like they have been doing.
    - drndown2007 16y ago
      A great article about how shaking our medical science is: http://www.theatlantic.com/magazine/print/2010/11/lies-damned-lies-and-medical-science/8269 http://www.theatlantic.com/magazine/print/2010/11/lies-damne... And yet, surgeons and medicines have definitely helped me....
  - brudgers 16y ago
    >"He's writing about results that were validated through replication. That is the scientific method." Experiments confirming phlogiston were validated through replication. The theory of phlogiston was however, incorrect. That's the problem with the scientific method.
    - rskar 16y ago
      The Theory of Phlogiston was discredited via the scientific method. It was in the act of replication that it got challenged. Phlogiston is an example of a theory that eventually got rejected as new facts were discovered and experimental design was critiqued, despite having about 100 years of apparent success. Phlogiston theory was, however, useful to chemists. When it met its limits, and a better theory came along, they of course dumped it. Apparently, the issue isn't scientific method per se, but of selfish interests who intentionally compromise experimental design and scientific debate for their own fame, glory, and gold. The good news is that the these half-truths and lies will get in the way of someone else's research and agenda, and that same someone else will likely make a big stink of it.
vacri 16y ago
A rather sensationalist headline given that the scientific method already recognises the problems of human researchers and not only offers some mechanisms to counter those problems, but is eager for more.
- rickmode 16y ago
  I agree. The article's headlines has a distinct anti-intellectual flair, and yet the article itself points out human flaws in research rather than a fundamental flaw in the scientific method.
swombat 16y ago
This is not so much a problem with the scientific method as a problem with self-interested applications of statistics. F still equals m*a hundreds of years later, and that's not about to change.
- gruseom 16y ago
  Yet I wonder if "the" scientific method has in fact been overapplied way past its sweet spot of mechanics and so on. It's by no means obvious that it works as well in, say, medicine, let alone psychology, let alone sociology, let alone economics. The fact that it's the cult of our age perhaps blinds us from asking the interesting questions about its limits.
  - swombat 16y ago
    Oh yes, there are definitely some problems with over-zealous applications of the scientific method (or rather, over-zealous discarding of other methods of reasoning) - but this article doesn't really cover them. Not everything of value can be measured in a double-blind experiment.
  - rmc 16y ago
    What's "the scientific method" you speak of? There main idea behind scientific reasoning is "Look for evidence to test things, don't be afraid to challenge the big boys"
    - michael_dorfman 16y ago
      There's quite a bit more to the scientific method than that-- the notions of controlled observation and reproducible results surely matter, as does the principle of parsimonious explanation.
      - arethuza 16y ago
        And falsifiability....
  - 16y ago
defen 16y ago
Is there any non-paywall way to view the article? This looks like it might be interesting but I can only see the first couple paragraphs. The title makes me think of three things: Paul Feyerabend, Bayesian inference, and the increasing effectiveness of placebos. I'd be curious to see if any of them are in the article.
- michael_dorfman 16y ago
  I don't know of any non-paywall way to view the article; I'm a (satisfied) New Yorker subscriber, so I've already paid. It's a great article, by the way. The answer to your latter (implied) question is "no"-- the article does not explicitly mention any of those three things.
- khafra 16y ago
  http://news.ycombinator.com/item?id=1978638 http://news.ycombinator.com/item?id=1978638 was posted after you commented.
bugsy 16y ago
The scientific method is fine. Note that this is about certain psychiatric pharmaceuticals being found not to have the efficacy that was earlier claimed, in studies done by pharmaceutical companies who doctored results and played statistics games and selectively hid results that didn't support profitability. It's fascinating to see the level of hubris in the corporate pharmaceutical industry, that after cornered and confronted with the fact that for decades their self-serving fabricated so called research is fraudulent, they would now have a conference to announce that the scientific method has failed.
- michael_dorfman 16y ago
  Did you read the article? It is not primarily about the pharmaceutical industry, or about doctoring results, although it touches on both of those topics. The argument is far more subtle, and more far-reaching.
  - drndown2007 16y ago
    Agreed. The one that has me wondering how far off we might be is the idea that radioactive decay might not be constant: http://io9.com/5619954/the-sun-is-changing-the-rate-of-radioactive-decay-and-breaking-the-rules-of-chemistry http://io9.com/5619954/the-sun-is-changing-the-rate-of-radio...
    - bugsy 16y ago
      That too demonstrates the scientific method in action. Experiments showed that the prevailing paradigm is likely flawed. Now we know more.
crayz 16y ago
Here's a PDF of the full article: http://crayz.org/science.pdf http://crayz.org/science.pdf
- JoachimSchipper 16y ago
  It's interesting and all, but I think the New Yorker has a right to restrict access to paying subscribers. EDIT: well, non-paying subscribers apparently. That makes me feel better.
  - crayz 16y ago
    I downloaded it by signing up for a free account, and it's littered with their ads

59 points by neilc 16y ago

Considering that most of these studies are by people with an interest in getting a certain result, it's likely experimental errors/not publishing some results will bias the published numbers in a certain direction. The scientific method has never been proof against people that lie by omission.

gruseom 16y ago
He's writing about results that were validated through replication. That is the scientific method. If what he's saying is accurate, the problem isn't a few bad apples (that's the proverbial reflexive defense of the status quo anyway). But now all sorts of well-established, multiply confirmed findings have started to look increasingly uncertain. I have no idea if it's accurate, but Lehrer's a credible journalist, less prone to the sensational than most. If he's writing about this, it's probably because serious scientists are concerned about it. (On another note, when I pasted the above, the following text showed up appended to my selection: Read more http://www.newyorker.com/reporting/2010/12/13/101213fa_fact_lehrer#ixzz17PM5acPA http://www.newyorker.com/reporting/2010/12/13/101213fa_fact_... How can such a venerable publication as the New Yorker resort to such tackiness?)
- burgerbrain 16y ago
  He mentions, among other things,studies in ESP as falling prey to this mysterious "well it worked the first time" syndrome. Now maybe the other examples are more credible, but mixing bullshit like ESP in doesn't help your case at all.
  - michael_dorfman 16y ago
    Actually, it does. The likelihood that ESP actually works is significantly less than the likelihood of "the decline effect", which is explained by a number of perfectly rational factors (touched upon in the article.) If you want to claim that "ESP is bullshit" (as you and I both do, I imagine), you somehow need to account for those (rare) studies which oddly enough seem to support an ESP effect at a statistically significant level. "The decline effect" goes a ways toward providing this explanation.
    - drndown2007 16y ago
      This article is one example (pre-cognition as opposed to ESP, but in the same vein): http://hplusmagazine.com/editors-blog/precognition-real-cornell-university-lab-releases-powerful-new-evidence-human-mind-can- http://hplusmagazine.com/editors-blog/precognition-real-corn... I think it comes down to we don't know nearly as much about our little existence as we like to think.
    - khafra 16y ago
      In other words, parapsychology is the control group for science: http://lesswrong.com/lw/1ib/parapsychology_the_control_group_for_science/ http://lesswrong.com/lw/1ib/parapsychology_the_control_group...
- St-Clock 16y ago
  "But now all sorts of well-established, multiply confirmed findings have started to look increasingly uncertain." Good! That's what science is all about. Making theories that can later be "disproved" or "confirmed". It's when you cannot disprove something that it is no longer science.
  - drndown2007 16y ago
    This means nothing can ever be confirmed though -- you never know if you'll learn something later to disprove what you think you know now. Which in turn means a certain level of humility is required. To me, that's where science falls short -- anything that doesn't fit what is currently 'known' and 'proven' is not taken seriously.
- bugsy 16y ago
  The replication they are talking about in these particular pharma studies are of the nature that a pharma corp orders up twelve identical 8 person studies. Three studies come back showing their new pill helped a tiny bit, six show no the pill did nothing, and three show the patients got worse. They then cherry pick the three that showed it helped, and possibly toss in one of the ones that showed the pill did nothing just to cover up what they are doing. They then publish a paper showing that 3 out of 4 studies validated that the pills work. The other 8 studies are set on fire and never mentioned. And there is your multiple studies. This is not some crazed conspiracy theory either, this (doing multiple very small sample size studies rather than one slightly larger study and discarding and never mentioning some of the studies with the least favorable results) is actually how it is now known to be done, and this methodology is the reason why some journals have started to say that all of the studies have to be registered in advance if they want to publish their results so that the companies can't selectively discard results like they have been doing.
  - drndown2007 16y ago
    A great article about how shaking our medical science is: http://www.theatlantic.com/magazine/print/2010/11/lies-damned-lies-and-medical-science/8269 http://www.theatlantic.com/magazine/print/2010/11/lies-damne... And yet, surgeons and medicines have definitely helped me....
- brudgers 16y ago
  >"He's writing about results that were validated through replication. That is the scientific method." Experiments confirming phlogiston were validated through replication. The theory of phlogiston was however, incorrect. That's the problem with the scientific method.
  - rskar 16y ago
    The Theory of Phlogiston was discredited via the scientific method. It was in the act of replication that it got challenged. Phlogiston is an example of a theory that eventually got rejected as new facts were discovered and experimental design was critiqued, despite having about 100 years of apparent success. Phlogiston theory was, however, useful to chemists. When it met its limits, and a better theory came along, they of course dumped it. Apparently, the issue isn't scientific method per se, but of selfish interests who intentionally compromise experimental design and scientific debate for their own fame, glory, and gold. The good news is that the these half-truths and lies will get in the way of someone else's research and agenda, and that same someone else will likely make a big stink of it.

vacri 16y ago

A rather sensationalist headline given that the scientific method already recognises the problems of human researchers and not only offers some mechanisms to counter those problems, but is eager for more.

rickmode 16y ago
I agree. The article's headlines has a distinct anti-intellectual flair, and yet the article itself points out human flaws in research rather than a fundamental flaw in the scientific method.

swombat 16y ago

This is not so much a problem with the scientific method as a problem with self-interested applications of statistics. F still equals m*a hundreds of years later, and that's not about to change.

gruseom 16y ago
Yet I wonder if "the" scientific method has in fact been overapplied way past its sweet spot of mechanics and so on. It's by no means obvious that it works as well in, say, medicine, let alone psychology, let alone sociology, let alone economics. The fact that it's the cult of our age perhaps blinds us from asking the interesting questions about its limits.
- swombat 16y ago
  Oh yes, there are definitely some problems with over-zealous applications of the scientific method (or rather, over-zealous discarding of other methods of reasoning) - but this article doesn't really cover them. Not everything of value can be measured in a double-blind experiment.
- rmc 16y ago
  What's "the scientific method" you speak of? There main idea behind scientific reasoning is "Look for evidence to test things, don't be afraid to challenge the big boys"
  - michael_dorfman 16y ago
    There's quite a bit more to the scientific method than that-- the notions of controlled observation and reproducible results surely matter, as does the principle of parsimonious explanation.
    - arethuza 16y ago
      And falsifiability....
- 16y ago

defen 16y ago

Is there any non-paywall way to view the article? This looks like it might be interesting but I can only see the first couple paragraphs. The title makes me think of three things: Paul Feyerabend, Bayesian inference, and the increasing effectiveness of placebos. I'd be curious to see if any of them are in the article.

michael_dorfman 16y ago
I don't know of any non-paywall way to view the article; I'm a (satisfied) New Yorker subscriber, so I've already paid. It's a great article, by the way. The answer to your latter (implied) question is "no"-- the article does not explicitly mention any of those three things.
khafra 16y ago
http://news.ycombinator.com/item?id=1978638 http://news.ycombinator.com/item?id=1978638 was posted after you commented.

bugsy 16y ago

The scientific method is fine. Note that this is about certain psychiatric pharmaceuticals being found not to have the efficacy that was earlier claimed, in studies done by pharmaceutical companies who doctored results and played statistics games and selectively hid results that didn't support profitability. It's fascinating to see the level of hubris in the corporate pharmaceutical industry, that after cornered and confronted with the fact that for decades their self-serving fabricated so called research is fraudulent, they would now have a conference to announce that the scientific method has failed.

michael_dorfman 16y ago
Did you read the article? It is not primarily about the pharmaceutical industry, or about doctoring results, although it touches on both of those topics. The argument is far more subtle, and more far-reaching.
- drndown2007 16y ago
  Agreed. The one that has me wondering how far off we might be is the idea that radioactive decay might not be constant: http://io9.com/5619954/the-sun-is-changing-the-rate-of-radioactive-decay-and-breaking-the-rules-of-chemistry http://io9.com/5619954/the-sun-is-changing-the-rate-of-radio...
  - bugsy 16y ago
    That too demonstrates the scientific method in action. Experiments showed that the prevailing paradigm is likely flawed. Now we know more.

crayz 16y ago

Here's a PDF of the full article: http://crayz.org/science.pdf http://crayz.org/science.pdf

JoachimSchipper 16y ago
It's interesting and all, but I think the New Yorker has a right to restrict access to paying subscribers. EDIT: well, non-paying subscribers apparently. That makes me feel better.
- crayz 16y ago
  I downloaded it by signing up for a free account, and it's littered with their ads