3 ms·

Great comment. I can see how the Bayesian setting uniquely equips you for dealing with non-point testing settings in which reasoning about alternatives to do po

by vladf 6y ago

Great comment. I can see how the Bayesian setting uniquely equips you for dealing with non-point testing settings in which reasoning about alternatives to do power, type m, and type s design would be hard. I'd be very curious about two follow-up questions here: 1 - A frequentist approach to the above could still be formulated in a minimax sort of way, but then you have to deal with alternatives which are close to your null. It's not like this problem goes away for Bayesians, it still seems like the final sample sizes you calculate could end up being very sensitive to the prior. Does this happen in practice? 2 - What kind of optimization goals do your users prefer when trading off power, type m, and type s? My guess based on this formulation it's something of the flavor "max power st P(type m or type s or type I) <= alpha", but wanted to check.

mlthoughts2018 6y ago
For 1 - keep in mind that frequentist inference cannot be used to support a statement like “the probability that variant A is better than variant B is X” or “the distribution of improvement from variant B over variant A is Y” - the only question a frequentist analysis can answer is, “Assuming the null hypothesis distribution is true, the unlikeliness on the observed data is Z.” From that point of view, we find frequentist analysis simply is epistemologically unsuited for comparing multiple (or a continuum of) policy options, period. Given this, then we start to ask totally new types of experiment design questions. We no longer ask an unphysical question such as, “assuming effect size X and sample size N, what is the probability of falsely rejecting the null” - the idea of “rejecting the null” doesn’t map to any notion of optimal policy selection, so we just don’t care about such a question when designing an experiment. Instead we ask ourselves, “if the true effect size is X, what is the probability I will make a mistake in estimating that effect size, and how large a mistake?” or “if the true relationship between the target and the covariate is positive, what is the probability I’ll mistakenly think it’s negative?” - basing experiment design on how my physical beliefs can be wrong helps me make decisions. Basing it on tail properties of a null distribution does not. 2. It really depends on the experiment. For causal inference, both type s and type m need to be very low, but type I can be high (I don’t care about rejecting a null). For inference where I only care about final predictive accuracy, this may not matter. For example in an extreme case I could have two perfectly collinear predictors, which means their coefficients can be arbitrary as long as the sum yields the true coefficient on the underlying linear component. If my goal is causal inference of the effect size, this would ruin it, since type m error can be unboundedly bad. But if the goal is overall predictive accuracy, it doesn’t matter at all - I just ignore the coefficients. > “ it still seems like the final sample sizes you calculate could end up being very sensitive to the prior.” I prefer to flip this around. A frequentist model has a prior too whether anyone wants to admit it or not - usually it is some unrealistic flat / uninformative prior or improper prior. The results of a frequentist method are equally sensitive to this implicit (huge) assumption. A Bayesian approach at least makes it explicit, admits the sensitivity, puts the range of prior choices out in the open for skeptical review, lets you carry out sensitivity analysis, and very often relies of real data and domain expertise to posit a much more physically plausible prior.

by vladf 6y ago

mlthoughts2018 6y ago

For 1 - keep in mind that frequentist inference cannot be used to support a statement like “the probability that variant A is better than variant B is X” or “the distribution of improvement from variant B over variant A is Y” - the only question a frequentist analysis can answer is, “Assuming the null hypothesis distribution is true, the unlikeliness on the observed data is Z.” From that point of view, we find frequentist analysis simply is epistemologically unsuited for comparing multiple (or a continuum of) policy options, period. Given this, then we start to ask totally new types of experiment design questions. We no longer ask an unphysical question such as, “assuming effect size X and sample size N, what is the probability of falsely rejecting the null” - the idea of “rejecting the null” doesn’t map to any notion of optimal policy selection, so we just don’t care about such a question when designing an experiment. Instead we ask ourselves, “if the true effect size is X, what is the probability I will make a mistake in estimating that effect size, and how large a mistake?” or “if the true relationship between the target and the covariate is positive, what is the probability I’ll mistakenly think it’s negative?” - basing experiment design on how my physical beliefs can be wrong helps me make decisions. Basing it on tail properties of a null distribution does not. 2. It really depends on the experiment. For causal inference, both type s and type m need to be very low, but type I can be high (I don’t care about rejecting a null). For inference where I only care about final predictive accuracy, this may not matter. For example in an extreme case I could have two perfectly collinear predictors, which means their coefficients can be arbitrary as long as the sum yields the true coefficient on the underlying linear component. If my goal is causal inference of the effect size, this would ruin it, since type m error can be unboundedly bad. But if the goal is overall predictive accuracy, it doesn’t matter at all - I just ignore the coefficients. > “ it still seems like the final sample sizes you calculate could end up being very sensitive to the prior.” I prefer to flip this around. A frequentist model has a prior too whether anyone wants to admit it or not - usually it is some unrealistic flat / uninformative prior or improper prior. The results of a frequentist method are equally sensitive to this implicit (huge) assumption. A Bayesian approach at least makes it explicit, admits the sensitivity, puts the range of prior choices out in the open for skeptical review, lets you carry out sensitivity analysis, and very often relies of real data and domain expertise to posit a much more physically plausible prior.