To be fair, a 1% bias would probably not be detected in a hundred trials. CI for Bernoulli is +/- z * sqrt(p * (1-p) / n), so for a 95% confidence interval smal

To be fair, a 1% bias would probably not be detected in a hundred trials. CI for Bernoulli is +/- z * sqrt(p * (1-p) / n), so for a 95% confidence interval smaller than a percentage point, we have 1.96 * sqrt(p * (1-p) / n) = 0.01 0.5^2 / n = (0.01/1.96)^2 n = (1.96/0.01 * 0.5)^2 Or about 9604. So you need roughly 10k trials to determine with 95% confidence a bias of 1%.