3 ms·

it's the same as everything. there should be more and easier money for the less rewarding task of verification/replication. some people actually enjoy this sort

by a-dub 5y ago

it's the same as everything. there should be more and easier money for the less rewarding task of verification/replication. some people actually enjoy this sort of work just as much as some people enjoy being on the bleeding edge... but there are probably less of them. where it would get complicated is also the same as everything. when the verification effort neither supports nor refutes the original one. many would argue that it means it wasn't done right, but lots of things aren't done right in life. then there can be the triple replication revolution! so it goes...

qchris 5y ago
This isn't intended as snarky, but I don't understand what "it's the same as everything" is supposed to mean. What is "it"? What is "everything"? Why are they the same? I'd also argue that your reduction of this problem sort of misses the point. One of the big problems with the way that studies are done is not that replication efforts aren't conclusive (it's very difficult to prove something doesn't exist), it's that a) non-replicable studies are generally considered as valuable as replicable ones, and b) as a result, it's extremely difficult to replicate many studies to begin with, because there's no incentive to take the time to make it possible. Even if the end result of a replication paper is "we couldn't produce the same results", the people working on it can say "this author's experiments were exceedingly difficult to even try to reproduce," or conversely "we didn't get the same results, but their data collection methods and analysis code were well-documented and accessible." That has a lot of value! If you tried doing triple replication for every paper, I agree that maybe wouldn't be the best use of resources. But the current state of affairs is so bad that a well-organized drive to create single-attempt replication on a fraction of publicly-funded projects has the potential to be a significant driver of change.
- a-dub 5y ago
  "the same as everything" is an observation that often times verification/correctness/accuracy efforts are tossed aside in favor of new development and this is a truism across many fields. in science you see this as funding being committed to shiny new nature and science cover stories, with replication being left as an afterthought. in software you see this as heavy commitments to new features that drive revenue, with security/compliance/architecture and qa remaining underfunded and less respected. (until, of course, the problems that result from underfunding them make themselves apparent).
Beldin 5y ago
Funding replication is a great idea, but cannot solve this. It would require roughly add much funding add now goes to science merely to replicate results produced now. That still leaves a rather hefty backlog. Moreover, the pace with which scientific output doubles is increasing. From top of my head out would be below a decade nowadays. Even if 90% of those would not need replication (new algorithms that work), then merely keeping pace would basically require 1 in 10 institutions to devote itself fully to replication studies. Even then we'd need more capacity to look at previous results - that 10% is fully needed to investigate new results. Note that this is optimistic: I'd expect the percentage of publications where a reproducibility study makes sense to be above 50%.

by a-dub 5y ago

qchris 5y ago

This isn't intended as snarky, but I don't understand what "it's the same as everything" is supposed to mean. What is "it"? What is "everything"? Why are they the same? I'd also argue that your reduction of this problem sort of misses the point. One of the big problems with the way that studies are done is not that replication efforts aren't conclusive (it's very difficult to prove something doesn't exist), it's that a) non-replicable studies are generally considered as valuable as replicable ones, and b) as a result, it's extremely difficult to replicate many studies to begin with, because there's no incentive to take the time to make it possible. Even if the end result of a replication paper is "we couldn't produce the same results", the people working on it can say "this author's experiments were exceedingly difficult to even try to reproduce," or conversely "we didn't get the same results, but their data collection methods and analysis code were well-documented and accessible." That has a lot of value! If you tried doing triple replication for every paper, I agree that maybe wouldn't be the best use of resources. But the current state of affairs is so bad that a well-organized drive to create single-attempt replication on a fraction of publicly-funded projects has the potential to be a significant driver of change.

a-dub 5y ago
"the same as everything" is an observation that often times verification/correctness/accuracy efforts are tossed aside in favor of new development and this is a truism across many fields. in science you see this as funding being committed to shiny new nature and science cover stories, with replication being left as an afterthought. in software you see this as heavy commitments to new features that drive revenue, with security/compliance/architecture and qa remaining underfunded and less respected. (until, of course, the problems that result from underfunding them make themselves apparent).

Beldin 5y ago

Funding replication is a great idea, but cannot solve this. It would require roughly add much funding add now goes to science merely to replicate results produced now. That still leaves a rather hefty backlog. Moreover, the pace with which scientific output doubles is increasing. From top of my head out would be below a decade nowadays. Even if 90% of those would not need replication (new algorithms that work), then merely keeping pace would basically require 1 in 10 institutions to devote itself fully to replication studies. Even then we'd need more capacity to look at previous results - that 10% is fully needed to investigate new results. Note that this is optimistic: I'd expect the percentage of publications where a reproducibility study makes sense to be above 50%.