2 ms·

It's only suspect if they claim 10/10 successes with p value of 0.1, right?

• gwern 12y ago
  No, it's not quite that simple: there's a three-way tradeoff between _n_, power, and _p_ - you can tradeoff one to save on another. So the 10/10 will be suspicious depending on the specific values of _n_, power, and _p_. However, usually psychology will always use a p<0.05, so that variable is fixed in advance, and they almost never use big enough _n_ for funding & convenience so you can safely assume _n_ will be 'too small'; with those two variable set, that implies that usually the power will be very small, like 10-50%/0.1-0.5, and then 10/10 would indeed be very suspicious (0.5^10 = a small number; if you flip a fair coin 10 times, it's not often you'll get 10 heads).