And that's not how to interpret the 95 and 99 thresholds. Depending on what proportion of "Version B" ideas are truly effective, your false positive rate (what

And that's not how to interpret the 95 and 99 thresholds. Depending on what proportion of "Version B" ideas are truly effective, your false positive rate (what the author is concerned about) will be very far away from 5% or 1%. At the 95 threshold (p-value of 0.05) you could have that >50% of what you identified as an "effective" version are not really effective (false positive). This is a good article to build the intuition around it: https://www.statisticsdonewrong.com/p-value.html https://www.statisticsdonewrong.com/p-value.html In the end it will also depend on how costly it is to have a false negative. The author hints that in this case is very costly.