3 ms·
> The wrong question to ask is: Is A better than B, given some confidence threshold? That's a perfectly fine question so long as there are three acceptable ans
by jgalt212 10y ago
> The wrong question to ask is: Is A better than B, given some confidence threshold?
That's a perfectly fine question so long as there are three acceptable answers: Yes, No and Unclear.
- tedsanders 10y agoSuppose A is significantly better than the control. And B is not significantly better than the control. Does that imply A is significantly better than B? No. Statistical differences do not the follow the same logic as normal differences. It feels so pointless to bin probabilities into yes, no, and unclear. We're throwing away information with value when we compress the problem this way. And I think it leads to more misinterpretations.
- imh 10y agoIn A/B testing, either A or B is the control.