5 ms·
In a Bayesian framework, replication is pretty easy to make predictions on. Suppose there is a parameter - say the probability q of a Bernoulli random variable
by yummyfajitas 11y ago
In a Bayesian framework, replication is pretty easy to make predictions on.
Suppose there is a parameter - say the probability q of a Bernoulli random variable being true. Based on your past experiments you have a posterior p(q). Then given p(q), you can easily compute a probability distribution on S(N), where S(N) is the number of successes you'd get from a bernoulli random variable with N attempts.
In code terms, you are just computing posterior.flatMap(q => Bernoulli(N,q)). (Using the inherent monadic structure of probability.)
This actually works in general. If you want to predict the outcome of a later experiment (the replication) you just compute posterior.flatMap(parameter => generateResult(parameter)).
- cba9 11y agoOr more concretely, since these projects are typically running on a hypothesis-testing paradigm, you can do a hybrid power analysis: compute a fully Bayesian analysis of the original experiment using other experiments and informative priors which take into account the true distribution of effects in a particular subfield (eg a broad distribution with many large effects if it's related to IQ, or a narrow distribution around zero if it's related to things like priming or stereotype thread) to generate a posterior distribution of effect sizes. Then you can simulate the results of future experiments with _n_ datapoints: sample 1 effect size from that distribution, generate _n_ datapoints assuming that effect size, run the hypothesis-testing, and return the _p_-value. The fraction of _p_<=0.05 is your best forecast of whether the future experiment will succeed in reproducing it or not.