3 ms·

Replication is not supposed to be a "challenge", that is not the point. The first point is ensure you can communicate the methods effectively, this demonstrates

by nonbel 11y ago

Replication is not supposed to be a "challenge", that is not the point. The first point is ensure you can communicate the methods effectively, this demonstrates the experimental conditions are understood. The other point is to see if the observation is stable to whatever differences arise due to varying time/location. And they should be looking at estimates of effect size, not whether p<0.05. Attempting to replace the role of independent replication with opinion is an awful idea. And that is the goal here, to replace, not to supplement. Of course, it doesn't seem replication attempts are very common in this area to begin with. So this is actually going to be a justification for continuing that pseudoscientific practice and to avoid checking all the previous results.

yummyfajitas 11y ago
In a Bayesian framework, replication is pretty easy to make predictions on. Suppose there is a parameter - say the probability q of a Bernoulli random variable being true. Based on your past experiments you have a posterior p(q). Then given p(q), you can easily compute a probability distribution on S(N), where S(N) is the number of successes you'd get from a bernoulli random variable with N attempts. In code terms, you are just computing posterior.flatMap(q => Bernoulli(N,q)). (Using the inherent monadic structure of probability.) This actually works in general. If you want to predict the outcome of a later experiment (the replication) you just compute posterior.flatMap(parameter => generateResult(parameter)).
- cba9 11y ago
  Or more concretely, since these projects are typically running on a hypothesis-testing paradigm, you can do a hybrid power analysis: compute a fully Bayesian analysis of the original experiment using other experiments and informative priors which take into account the true distribution of effects in a particular subfield (eg a broad distribution with many large effects if it's related to IQ, or a narrow distribution around zero if it's related to things like priming or stereotype thread) to generate a posterior distribution of effect sizes. Then you can simulate the results of future experiments with _n_ datapoints: sample 1 effect size from that distribution, generate _n_ datapoints assuming that effect size, run the hypothesis-testing, and return the _p_-value. The fraction of _p_<=0.05 is your best forecast of whether the future experiment will succeed in reproducing it or not.
btilly 11y ago
No, the main purpose of attempting replication is to challenge of the result. "I think you made a mistake, let's try to replicate and show it." And if you fail to replicate, you publish that and show the world that the result shouldn't be believed. If you successfully replicate, then it has survived the challenge. The more challenges it survives, the more confidence we can have that the result is valid. That said, we do not try to make replication challenging. We try to present experiments in a way that makes replication as easy as possible to perform. Exactly so that people don't have to take what we said on faith. The point of the prediction market here is not to replace independent replication with opinion. It is to ensure that the energy that gets spent on replication is more likely to be spent effectively. Ideally, of course, targeting replication efforts more effectively will increase the value of time spent in attempting replication. This should therefore increase how much effort is spent on replication. Which is exactly the opposite of replacing independent replication with opinion!
- nonbel 11y ago
  > "if you fail to replicate, you publish that and show the world that the result shouldn't be believed." A lone report of some observation should never be believed. It should be verified by others who retrace the steps. >"If you successfully replicate, then it has survived the challenge. The more challenges it survives, the more confidence we can have that the result is valid." If the observation can be independently replicated, it shows that the methods required are understood well enough to communicate and that it is stable in the face of unknown influences. This increases our confidence that we understand what is going on and that the phenomenon is worth theorizing about. It has nothing to do with an observation being true or valid. The observation was made, it happened. It is true. It is valid. (Excepting outright fraud, which is treated equivalently to some severely unstable phenomenon) >"The point of the prediction market here is not to replace independent replication with opinion." They explicitly say that is the goal in the paper. I like this method of eliciting priors, not that the goal is to substitute it for actual replications. >"It is to ensure that the energy that gets spent on replication is more likely to be spent effectively." If a study isn't worth replicating, then it wasn't worth doing and reporting in the first place.
Fomite 11y ago
> And they should be looking at estimates of effect size This. Null hypothesis testing: Just stop it. We've known this is a bad idea for decades

by nonbel 11y ago

yummyfajitas 11y ago

In a Bayesian framework, replication is pretty easy to make predictions on. Suppose there is a parameter - say the probability q of a Bernoulli random variable being true. Based on your past experiments you have a posterior p(q). Then given p(q), you can easily compute a probability distribution on S(N), where S(N) is the number of successes you'd get from a bernoulli random variable with N attempts. In code terms, you are just computing posterior.flatMap(q => Bernoulli(N,q)). (Using the inherent monadic structure of probability.) This actually works in general. If you want to predict the outcome of a later experiment (the replication) you just compute posterior.flatMap(parameter => generateResult(parameter)).

cba9 11y ago
Or more concretely, since these projects are typically running on a hypothesis-testing paradigm, you can do a hybrid power analysis: compute a fully Bayesian analysis of the original experiment using other experiments and informative priors which take into account the true distribution of effects in a particular subfield (eg a broad distribution with many large effects if it's related to IQ, or a narrow distribution around zero if it's related to things like priming or stereotype thread) to generate a posterior distribution of effect sizes. Then you can simulate the results of future experiments with _n_ datapoints: sample 1 effect size from that distribution, generate _n_ datapoints assuming that effect size, run the hypothesis-testing, and return the _p_-value. The fraction of _p_<=0.05 is your best forecast of whether the future experiment will succeed in reproducing it or not.

btilly 11y ago

No, the main purpose of attempting replication is to challenge of the result. "I think you made a mistake, let's try to replicate and show it." And if you fail to replicate, you publish that and show the world that the result shouldn't be believed. If you successfully replicate, then it has survived the challenge. The more challenges it survives, the more confidence we can have that the result is valid. That said, we do not try to make replication challenging. We try to present experiments in a way that makes replication as easy as possible to perform. Exactly so that people don't have to take what we said on faith. The point of the prediction market here is not to replace independent replication with opinion. It is to ensure that the energy that gets spent on replication is more likely to be spent effectively. Ideally, of course, targeting replication efforts more effectively will increase the value of time spent in attempting replication. This should therefore increase how much effort is spent on replication. Which is exactly the opposite of replacing independent replication with opinion!

nonbel 11y ago
> "if you fail to replicate, you publish that and show the world that the result shouldn't be believed." A lone report of some observation should never be believed. It should be verified by others who retrace the steps. >"If you successfully replicate, then it has survived the challenge. The more challenges it survives, the more confidence we can have that the result is valid." If the observation can be independently replicated, it shows that the methods required are understood well enough to communicate and that it is stable in the face of unknown influences. This increases our confidence that we understand what is going on and that the phenomenon is worth theorizing about. It has nothing to do with an observation being true or valid. The observation was made, it happened. It is true. It is valid. (Excepting outright fraud, which is treated equivalently to some severely unstable phenomenon) >"The point of the prediction market here is not to replace independent replication with opinion." They explicitly say that is the goal in the paper. I like this method of eliciting priors, not that the goal is to substitute it for actual replications. >"It is to ensure that the energy that gets spent on replication is more likely to be spent effectively." If a study isn't worth replicating, then it wasn't worth doing and reporting in the first place.

Fomite 11y ago

> And they should be looking at estimates of effect size This. Null hypothesis testing: Just stop it. We've known this is a bad idea for decades