For 1 - keep in mind that frequentist inference cannot be used to support a statement like “the probability that variant A is better than variant B is X” or “th

For 1 - keep in mind that frequentist inference cannot be used to support a statement like “the probability that variant A is better than variant B is X” or “the distribution of improvement from variant B over variant A is Y” - the only question a frequentist analysis can answer is, “Assuming the null hypothesis distribution is true, the unlikeliness on the observed data is Z.” From that point of view, we find frequentist analysis simply is epistemologically unsuited for comparing multiple (or a continuum of) policy options, period. Given this, then we start to ask totally new types of experiment design questions. We no longer ask an unphysical question such as, “assuming effect size X and sample size N, what is the probability of falsely rejecting the null” - the idea of “rejecting the null” doesn’t map to any notion of optimal policy selection, so we just don’t care about such a question when designing an experiment. Instead we ask ourselves, “if the true effect size is X, what is the probability I will make a mistake in estimating that effect size, and how large a mistake?” or “if the true relationship between the target and the covariate is positive, what is the probability I’ll mistakenly think it’s negative?” - basing experiment design on how my physical beliefs can be wrong helps me make decisions. Basing it on tail properties of a null distribution does not. 2. It really depends on the experiment. For causal inference, both type s and type m need to be very low, but type I can be high (I don’t care about rejecting a null). For inference where I only care about final predictive accuracy, this may not matter. For example in an extreme case I could have two perfectly collinear predictors, which means their coefficients can be arbitrary as long as the sum yields the true coefficient on the underlying linear component. If my goal is causal inference of the effect size, this would ruin it, since type m error can be unboundedly bad. But if the goal is overall predictive accuracy, it doesn’t matter at all - I just ignore the coefficients. > “ it still seems like the final sample sizes you calculate could end up being very sensitive to the prior.” I prefer to flip this around. A frequentist model has a prior too whether anyone wants to admit it or not - usually it is some unrealistic flat / uninformative prior or improper prior. The results of a frequentist method are equally sensitive to this implicit (huge) assumption. A Bayesian approach at least makes it explicit, admits the sensitivity, puts the range of prior choices out in the open for skeptical review, lets you carry out sensitivity analysis, and very often relies of real data and domain expertise to posit a much more physically plausible prior.