5 ms·

Replication might suggest they obtained the same results. A better title for laypeople would be "We _refuted_ a Science paper that can't be replicated and Scie

by gota 7y ago

Replication might suggest they obtained the same results. A better title for laypeople would be "We _refuted_ a Science paper that can't be replicated and Science won't publish it"

scoofy 7y ago
Except that's not how science works. Publishing a paper with contradictory result is not a refutation. It's a data point.
- gota 7y ago
  I'm not educated in how it works for this kind of (sociological? Psychological?) experiment but finding repeated instances - as per the trials reported in the article - seems like more than just a data point. They have many more negative data points than the original paper had positive ones In pure formal logic they have at least proved it is not true for all cases, so it's a refutation in that sense at least But in any case, "refutation" is closer in popular lingo to what happened than "replication"
  - scoofy 7y ago
    This gets into deep philosophical questions. Logic can be deductive or inductive. Your use of formal logic implies deductive logic, but empirical science is inductive. With induction, one can only have negative knowledge (this is know as the problem of induction or the black swan problem), but it gets more complicated than that. Most studies are trying to look at probabilistic outcomes and are trying to get significant p-values. There is a myriad of problems with this beyond simply the existing probability that their sample was bias. E.g., their methodology could be improper, or even just less than ideal. There could even be unknown problems with the underlying assumptions and the model being used by researches. To suggest that an opposite result refutes a previously obtained result deeply mis-characterizes how science and human understanding works, and the problems inherent in this thought process deeply overvalues the knowledge we have and undervalues the more-than-likely seriously uncertain data we work with every day. If you're interested in these problems, i'd suggest Karl Popper's work on philosophy of science.
    - gota 7y ago
      Thanks for your comment, I see it can help frame the conversation, but I meant to say I'm not versed in the terminology for social sciences. I'm about to finish my Doctorate heavily involving formal logic and logic programming, and have read a sufficient bit about the philosophy of science This is an interesting discussion but I'm unfortunately out of time now. I'd only point that most of science is abductive, rather than inductive, and that the whole concept of refutation will take us to discuss what is scientific _advance_
      - your-nanny 7y ago
        best just think in terms of Bayesian update.
        gota 7y ago
        Indeed, abduction is a way to base your priors in Bayesian nets!
    - StavrosK 7y ago
      In the same vein, a positive result doesn't prove anything, it's merely a data point in the opposite direction.
    - heyoni 7y ago
      Any publications in particular you suggest?
- wallace_f 7y ago
  Science is supposed to be about the scientific method--a testable hypothesis. This strict definition has been forsaken for the benefit of all who would like to publish, and take credit for, science.
  - heyoni 7y ago
    This is bound to happen in a social sciences experiment where the environment can’t be completely controlled.
    - DoreenMichele 7y ago
      Social science often isn't science at all. It's often really bad. Social experiments by their very nature tend to pollute the data. This is part of why I don't believe UBI would actually work: Historical real world examples of the idea consistently fail. I think the UBI experiments are all nonsense.
      - Kim_Bruning 7y ago
        This is clearly a side branch, but I'll bite. Can you point to historical real world examples?
        DoreenMichele 7y ago
        Communism promised to provide equally well to all people. It's considered to be very much a failed experiment. Early settlers in the US promised to provide for everyone. When freeloaders eventually ruined that deal, they changed it to "If you don't work, you don't eat." Every time this has been tried out in the world, it works for a small group of committed, like-minded people. When it is opened up to "anyone/everyone," they inevitably have to come up with rules to account for free riders who want to take advantage. Another real world example: When Spanish Conquistadors sent large quantities of silver back to Spain from the Americas, it didn't make everyone in Spain rich. It merely fueled inflation. That's what happens when you add money to the system without adding goods and services: Prices go up. The current UBI experiments are so limited in scope that the conclusion is akin to saying "People with kindly rich uncles are better off than people without such." But when it stops being a small group of privileged experiment subjects and becomes everyone, the conclusions drawn from the experiments won't apply anymore.
- philwelch 7y ago
  A failed replication isn’t just a contradictory result. That would be if you tried to test the same hypothesis a different way. A true replication attempt is when you try to repeat the same experiment and fail. Likewise, if the original paper is a data point, a failed replication gives you two data points that cancel out.
  - deleted 7y ago
    [deleted]
- EGreg 7y ago
  Depends - in physics it’s most definitely a refutation.
- cm2187 7y ago
  Except that this is social sciences, not science. Unfalsifiable theories and unreplicable "experiments" are common place but unfortunately the word "science" in the title makes it look like it is something else than a mix of journalism and philosophy.
- deleted 7y ago
  [deleted]
NoodleIncident 7y ago
It sounds like they gave up after Science wouldn't send it out for peer review themselves; whether it's a replication or a refutation, it is not yet even peer reviewed, much less published. I don't think they provide any compelling reasons why the same journal is obligated to publish replications of any arbitrary study they've published in the past. It wouldn't exactly scale.
- rjf72 7y ago
  By replication, let's refer to refutations. I'd unfortunately agree that positive replications would provide relatively little value. But replications that seem to refute previously published studies are critical for at least two reasons I immediately see, and undoubtedly many more: 1) Replication efforts need to be strongly incentivized. Science in modern times has been taking repeated hammer blows to its credibility. The systems we have in place seem to result in more and more simply 'fake' science being published, including in the top journals. This is not an easy problem to solve since replication efforts can be very costly and things like pre-publishing still ends up relying on good behavior, which is something we can no longer necessarily take for granted. The one small tool we do have against bad science is replication efforts. Every effort should be made to incentivize these. The willingness to publish significant refutations to influential studies is well below the bare minimum we should expect from journals on this front. 2) What is the purpose of a journal, outside of making money? It's to inform people by providing a filtered repository of the information the reaches a high level on the scale between reliability and relevance. When a journal publishes a study that may have been false, they end up doing the exact opposite of this - they are misinforming society. And this is especially true for the study which seems to have been refuted. It was not a footnote - it has been widely referenced and had a significant impact on public discourse. This all leads to a strong obligation to publish. 3) (another point derived from #1 while writing) - Publication of refutations aligns journalistic interests with public interests. In our current system journals are primarily motivated to publish the most 'meaningful' results which is ultimately a euphemism for the most shocking results. So they have an incentive to want to give these studies the benefit of the doubt. Yet these are the very studies which should be held to the highest degree of scrutiny and criticism. A journal that is willing to publish refutations of the studies it publishes returns this alignment since if Science found itself full of little more than refutation of 'meaningful' studies it published in the past, it would quickly lose its prestige. Yet at the same time if it was willing to publish refutations, yet they were few and far between, one could hold what was published within the journal to a much higher standard of reliability - which would be good for both the journal and the authors who were able to meet what would undoubtedly be their now increased standards.
pera 7y ago
Replication in this case refers specifically to the replication of the original experiment itself, not its results: https://en.wikipedia.org/wiki/Replication_(statistics) https://en.wikipedia.org/wiki/Replication_(statistics)

by gota 7y ago

Replication might suggest they obtained the same results. A better title for laypeople would be "We _refuted_ a Science paper that can't be replicated and Science won't publish it"

scoofy 7y ago

Except that's not how science works. Publishing a paper with contradictory result is not a refutation. It's a data point.

gota 7y ago
I'm not educated in how it works for this kind of (sociological? Psychological?) experiment but finding repeated instances - as per the trials reported in the article - seems like more than just a data point. They have many more negative data points than the original paper had positive ones In pure formal logic they have at least proved it is not true for all cases, so it's a refutation in that sense at least But in any case, "refutation" is closer in popular lingo to what happened than "replication"
- scoofy 7y ago
  This gets into deep philosophical questions. Logic can be deductive or inductive. Your use of formal logic implies deductive logic, but empirical science is inductive. With induction, one can only have negative knowledge (this is know as the problem of induction or the black swan problem), but it gets more complicated than that. Most studies are trying to look at probabilistic outcomes and are trying to get significant p-values. There is a myriad of problems with this beyond simply the existing probability that their sample was bias. E.g., their methodology could be improper, or even just less than ideal. There could even be unknown problems with the underlying assumptions and the model being used by researches. To suggest that an opposite result refutes a previously obtained result deeply mis-characterizes how science and human understanding works, and the problems inherent in this thought process deeply overvalues the knowledge we have and undervalues the more-than-likely seriously uncertain data we work with every day. If you're interested in these problems, i'd suggest Karl Popper's work on philosophy of science.
  - gota 7y ago
    Thanks for your comment, I see it can help frame the conversation, but I meant to say I'm not versed in the terminology for social sciences. I'm about to finish my Doctorate heavily involving formal logic and logic programming, and have read a sufficient bit about the philosophy of science This is an interesting discussion but I'm unfortunately out of time now. I'd only point that most of science is abductive, rather than inductive, and that the whole concept of refutation will take us to discuss what is scientific _advance_
    - your-nanny 7y ago
      best just think in terms of Bayesian update.
      - gota 7y ago
        Indeed, abduction is a way to base your priors in Bayesian nets!
  - StavrosK 7y ago
    In the same vein, a positive result doesn't prove anything, it's merely a data point in the opposite direction.
  - heyoni 7y ago
    Any publications in particular you suggest?
wallace_f 7y ago
Science is supposed to be about the scientific method--a testable hypothesis. This strict definition has been forsaken for the benefit of all who would like to publish, and take credit for, science.
- heyoni 7y ago
  This is bound to happen in a social sciences experiment where the environment can’t be completely controlled.
  - DoreenMichele 7y ago
    Social science often isn't science at all. It's often really bad. Social experiments by their very nature tend to pollute the data. This is part of why I don't believe UBI would actually work: Historical real world examples of the idea consistently fail. I think the UBI experiments are all nonsense.
    - Kim_Bruning 7y ago
      This is clearly a side branch, but I'll bite. Can you point to historical real world examples?
      - DoreenMichele 7y ago
        Communism promised to provide equally well to all people. It's considered to be very much a failed experiment. Early settlers in the US promised to provide for everyone. When freeloaders eventually ruined that deal, they changed it to "If you don't work, you don't eat." Every time this has been tried out in the world, it works for a small group of committed, like-minded people. When it is opened up to "anyone/everyone," they inevitably have to come up with rules to account for free riders who want to take advantage. Another real world example: When Spanish Conquistadors sent large quantities of silver back to Spain from the Americas, it didn't make everyone in Spain rich. It merely fueled inflation. That's what happens when you add money to the system without adding goods and services: Prices go up. The current UBI experiments are so limited in scope that the conclusion is akin to saying "People with kindly rich uncles are better off than people without such." But when it stops being a small group of privileged experiment subjects and becomes everyone, the conclusions drawn from the experiments won't apply anymore.
philwelch 7y ago
A failed replication isn’t just a contradictory result. That would be if you tried to test the same hypothesis a different way. A true replication attempt is when you try to repeat the same experiment and fail. Likewise, if the original paper is a data point, a failed replication gives you two data points that cancel out.
- deleted 7y ago
  [deleted]
EGreg 7y ago
Depends - in physics it’s most definitely a refutation.
cm2187 7y ago
Except that this is social sciences, not science. Unfalsifiable theories and unreplicable "experiments" are common place but unfortunately the word "science" in the title makes it look like it is something else than a mix of journalism and philosophy.
deleted 7y ago
[deleted]

NoodleIncident 7y ago

It sounds like they gave up after Science wouldn't send it out for peer review themselves; whether it's a replication or a refutation, it is not yet even peer reviewed, much less published. I don't think they provide any compelling reasons why the same journal is obligated to publish replications of any arbitrary study they've published in the past. It wouldn't exactly scale.

rjf72 7y ago
By replication, let's refer to refutations. I'd unfortunately agree that positive replications would provide relatively little value. But replications that seem to refute previously published studies are critical for at least two reasons I immediately see, and undoubtedly many more: 1) Replication efforts need to be strongly incentivized. Science in modern times has been taking repeated hammer blows to its credibility. The systems we have in place seem to result in more and more simply 'fake' science being published, including in the top journals. This is not an easy problem to solve since replication efforts can be very costly and things like pre-publishing still ends up relying on good behavior, which is something we can no longer necessarily take for granted. The one small tool we do have against bad science is replication efforts. Every effort should be made to incentivize these. The willingness to publish significant refutations to influential studies is well below the bare minimum we should expect from journals on this front. 2) What is the purpose of a journal, outside of making money? It's to inform people by providing a filtered repository of the information the reaches a high level on the scale between reliability and relevance. When a journal publishes a study that may have been false, they end up doing the exact opposite of this - they are misinforming society. And this is especially true for the study which seems to have been refuted. It was not a footnote - it has been widely referenced and had a significant impact on public discourse. This all leads to a strong obligation to publish. 3) (another point derived from #1 while writing) - Publication of refutations aligns journalistic interests with public interests. In our current system journals are primarily motivated to publish the most 'meaningful' results which is ultimately a euphemism for the most shocking results. So they have an incentive to want to give these studies the benefit of the doubt. Yet these are the very studies which should be held to the highest degree of scrutiny and criticism. A journal that is willing to publish refutations of the studies it publishes returns this alignment since if Science found itself full of little more than refutation of 'meaningful' studies it published in the past, it would quickly lose its prestige. Yet at the same time if it was willing to publish refutations, yet they were few and far between, one could hold what was published within the journal to a much higher standard of reliability - which would be good for both the journal and the authors who were able to meet what would undoubtedly be their now increased standards.

pera 7y ago

Replication in this case refers specifically to the replication of the original experiment itself, not its results: https://en.wikipedia.org/wiki/Replication_(statistics) https://en.wikipedia.org/wiki/Replication_(statistics)