3 ms·

This is a meta-piece about another article. The actual article: http://www.nature.com/news/nobel-laureate-challenges-psychologists-to-clean-up-their-act-1.1153

by dfc 14y ago

This is a meta-piece about another article. The actual article: http://www.nature.com/news/nobel-laureate-challenges-psychologists-to-clean-up-their-act-1.11535 http://www.nature.com/news/nobel-laureate-challenges-psychol... The actual article is worth a read, I was a little intimidated after reading "here’s some overmatter that didn’t make it into the piece because of length" but the actual article is only 712 words. The over-matter is 500 words. More background about the problem: "Replication studies: Bad copy" http://www.nature.com/news/replication-studies-bad-copy-1.10634 http://www.nature.com/news/replication-studies-bad-copy-1.10...

bambax 14y ago
"Replication studies: Bad copy" is very interesting. Sample quote: “I've seen students spending their entire PhD period trying to replicate a phenomenon, failing, and quitting academia because they had nothing to show for their time.” We say that a study "succeeds" when it illustrates a novel behavior and "fails" when the expected behavior doesn't happen. This is a very slippery use of words. A finding that can't be replicated isn't a finding. It's a rumor, an urban myth, but not a scientific step forward. A study that "fails to replicate" a previous finding isn't a "failure" -- it succeeded at debunking a false belief! And it should be granted wide access to publishing. (Same thing happens in medical studies BTW, where it's arguably much more dangerous).
- chii 14y ago
  > A study that "fails to replicate" a previous finding isn't a "failure". Yep. the problem is that to the "laymen", it seems a bunch of people wasted time trying to re-do what someone else has "already" done, and failed. Whether the failure was due to incompetance, chance, or an actual failure to replicate (meaning it is a significant), to the mind of the laymen, they are one and the same. this then end up being difficult for people who might fund such studies to fund these sorts of studies (despite it being quite useful), and leads to people seeking "newer" things to research. its a bit of a chilling effect really.
  - StavrosK 14y ago
    Who are these laymen, and why are they reviewing papers?
    - chii 14y ago
      they are not reviewing papers, nor are they the fund contributors. They are the general population, whose sentiments affect the way society sees things overall. For example, a lot of people are now rightfully scared of nuclear tech (due to recent disasters). And yet, most people dont understand that neclear energy is quite safe (compared to coal, say). If a political leader decides to invest in nuclear energy, they instantly become unpopular, and thus lies the problem of funding via such channels.
      - StavrosK 14y ago
        Sure, but they aren't the ones deciding what gets published or not. The study that tried to replicate another study but failed has already been done, and it's not up to the laymen to decide whether or not to publish the results. Nor can we predict which studies will fail to duplicate results and not fund them, otherwise we wouldn't have to make them. Results of failure are still results, and they represent valuable knowledge. Too bad the people who got them consider them a failure.
        lars512 14y ago
        If there's an obvious flaw in the existing study, you can make something out of a new study which doesn't contain that flaw. But it's very hard to convince yourself (and others) that replicating/failing to replicate someone's work is the same level of contribution as the original work. What would be great: a centralised place for people to share their attempts to replicate a work. If several groups have failed, and nobody has succeeded, then there's certainly room for a rigorous, joint debunking of the experiment. If, as is often the case, there's some subtlety to get the experiment right, then the discussion around it between groups actually enhances the original work.

by dfc 14y ago

bambax 14y ago

"Replication studies: Bad copy" is very interesting. Sample quote: “I've seen students spending their entire PhD period trying to replicate a phenomenon, failing, and quitting academia because they had nothing to show for their time.” We say that a study "succeeds" when it illustrates a novel behavior and "fails" when the expected behavior doesn't happen. This is a very slippery use of words. A finding that can't be replicated isn't a finding. It's a rumor, an urban myth, but not a scientific step forward. A study that "fails to replicate" a previous finding isn't a "failure" -- it succeeded at debunking a false belief! And it should be granted wide access to publishing. (Same thing happens in medical studies BTW, where it's arguably much more dangerous).

chii 14y ago
> A study that "fails to replicate" a previous finding isn't a "failure". Yep. the problem is that to the "laymen", it seems a bunch of people wasted time trying to re-do what someone else has "already" done, and failed. Whether the failure was due to incompetance, chance, or an actual failure to replicate (meaning it is a significant), to the mind of the laymen, they are one and the same. this then end up being difficult for people who might fund such studies to fund these sorts of studies (despite it being quite useful), and leads to people seeking "newer" things to research. its a bit of a chilling effect really.
- StavrosK 14y ago
  Who are these laymen, and why are they reviewing papers?
  - chii 14y ago
    they are not reviewing papers, nor are they the fund contributors. They are the general population, whose sentiments affect the way society sees things overall. For example, a lot of people are now rightfully scared of nuclear tech (due to recent disasters). And yet, most people dont understand that neclear energy is quite safe (compared to coal, say). If a political leader decides to invest in nuclear energy, they instantly become unpopular, and thus lies the problem of funding via such channels.
    - StavrosK 14y ago
      Sure, but they aren't the ones deciding what gets published or not. The study that tried to replicate another study but failed has already been done, and it's not up to the laymen to decide whether or not to publish the results. Nor can we predict which studies will fail to duplicate results and not fund them, otherwise we wouldn't have to make them. Results of failure are still results, and they represent valuable knowledge. Too bad the people who got them consider them a failure.
      - lars512 14y ago
        If there's an obvious flaw in the existing study, you can make something out of a new study which doesn't contain that flaw. But it's very hard to convince yourself (and others) that replicating/failing to replicate someone's work is the same level of contribution as the original work. What would be great: a centralised place for people to share their attempts to replicate a work. If several groups have failed, and nobody has succeeded, then there's certainly room for a rigorous, joint debunking of the experiment. If, as is often the case, there's some subtlety to get the experiment right, then the discussion around it between groups actually enhances the original work.