5 ms·

I think the entire approach discussed in this pdf is flawed. (Edit: not saying PDF itself is flawed or wrong, just the hypothesis testing approach to A/B testin

I think the entire approach discussed in this pdf is flawed. (Edit: not saying PDF itself is flawed or wrong, just the hypothesis testing approach to A/B testing.) The right question to ask is: What is the difference between A and B, and what is our uncertainty on that estimate? The wrong question to ask is: Is A different/better than B, given some confidence threshold? The reason this is the wrong question is that it's unnecessarily binary. It is a non-linear transformation of information that undervalues confidence away from the arbitrary threshold and overvalues confidence right at the arbitrary threshold. A test with only 10 or 100 samples still gives you information. It gives you weak information, sure, but information nonetheless. If you approach the problem from a continuous perspective (asking how big the difference is), you can straightforwardly use the information. But if you approach the problem from a binary hypothesis-testing perspective (asking is there a difference), you'll be throwing away lots of weak information when it could be proving real (yet uncertain) value. Once you switch away from the binary hypothesis-testing framework, you no longer have to worry about silly issues like stopping too early or false positives or false negatives. You simply have a distribution of probabilities over possible effect sizes.

jgalt212 10y ago
> The wrong question to ask is: Is A better than B, given some confidence threshold? That's a perfectly fine question so long as there are three acceptable answers: Yes, No and Unclear.
- tedsanders 10y ago
  Suppose A is significantly better than the control. And B is not significantly better than the control. Does that imply A is significantly better than B? No. Statistical differences do not the follow the same logic as normal differences. It feels so pointless to bin probabilities into yes, no, and unclear. We're throwing away information with value when we compress the problem this way. And I think it leads to more misinterpretations.
  - imh 10y ago
    In A/B testing, either A or B is the control.
martingoodson 10y ago
I don't know about 'flawed'. Classical hypothesis testing has its place, when used correctly. I guess the point of the article is that it falls apart dramatically when you do certain things wrong. And that these things turn out to be in very common practice. Disclaimer: I wrote that article (a long time ago...)
Obi_Juan_Kenobi 10y ago
That's putting the cart before the horse. Before you can quantify a difference, you have to determine whether one exists in the first place. That is the purpose of binary testing; without it, you're just looking at noise without any means to decide what is real and what is not. As a corollary, if you can meaningfully quantify the difference between A and B, then you should have no trouble establishing that they are different. Obviously business decisions are not generally going to uphold the rigor of good science, but what is the purpose of quantifying things when you're as likely to be wrong as your are right?
- tedsanders 10y ago
  Asking whether A and B are different is not a useful question. There will be a difference 100% of the time. (Though of course sampling may not reliably detect the difference for feasible samples.) The superior question is which is probably better, and by how much. If all you know is that A is 75% likely to be better than B, then go with A. It's useful information, even if it doesn't cross a an arbitrarily preset threshold of 95% or whatever you use. You don't need to wait for certainty to act. In fact, all actions are taken under uncertainty. So it feels incredibly artificial and counterproductive to frame these questions in such a binary, nonlinear way. It's clinging to certainty when certainty does not exist.
  - yummyfajitas 10y ago
    Actually even that isn't the best question. The best question is what's the expected reward from choosing A. Consider the scenario of 10% chance B > A, 90% chance they are equal. To me that sounds like a winning bet. In contrast, 70% chance B is a lot better than A, 30% chance B is a lot worse than A sounds like a scary bet. I'll wait and gather more info. Proper decision theory is based on a loss function which incorporates the magnitude of gains/losses rather than just their existence. https://en.wikipedia.org/wiki/Loss_function https://en.wikipedia.org/wiki/Loss_function
korzun 10y ago
> A test with only 10 or 100 samples still gives you information. It gives you weak information, sure, but information nonetheless. If your market consists of 1000 people.
- tedsanders 10y ago
  Could you elaborate? Even if your market consists of infinite people, it gives you real information.
dxhdr 10y ago
Maybe I'm misunderstanding but isn't the whole point of A/B testing to answer the question "is A better than B?"
Silhouette 10y ago
If you're running an A/B test, by which I mean precisely that you already have two versions available and are simply trying to decide which is most likely to give better results in the future by comparing how successfully they achieve the desired result with all else being equal, then I'm still waiting for someone to explain to me why the correct question is not simply "Which performs better, A or B?" The statistical rigour of all the hypothesis tests and Bayesian methods and so on is laudable, but fundamentally any result and any conclusion you draw from those tests can only ever be as good as the underlying assumptions from which the result was derived. For example, if you choose to perform any statistical test that designates one of A and B the default case and tests for evidence in favour of the other, you have made a determination that the situation is not symmetric. You can choose a level of power or a significance level you want to achieve, but the 5% convention is again arbitrary. The same goes for any prior you choose in a Bayesian method. Fundamentally, if you are truly starting from a neutral situation where you just want to know which of A and B is better, and you truly have no other information to go on other than how they have performed since the start of your measurements with all else being equal, then it remains the case that you still have nothing else to go on no matter what statistical calculations you perform.

jgalt212 10y ago

> The wrong question to ask is: Is A better than B, given some confidence threshold? That's a perfectly fine question so long as there are three acceptable answers: Yes, No and Unclear.

tedsanders 10y ago
Suppose A is significantly better than the control. And B is not significantly better than the control. Does that imply A is significantly better than B? No. Statistical differences do not the follow the same logic as normal differences. It feels so pointless to bin probabilities into yes, no, and unclear. We're throwing away information with value when we compress the problem this way. And I think it leads to more misinterpretations.
- imh 10y ago
  In A/B testing, either A or B is the control.

martingoodson 10y ago

I don't know about 'flawed'. Classical hypothesis testing has its place, when used correctly. I guess the point of the article is that it falls apart dramatically when you do certain things wrong. And that these things turn out to be in very common practice. Disclaimer: I wrote that article (a long time ago...)

Obi_Juan_Kenobi 10y ago

That's putting the cart before the horse. Before you can quantify a difference, you have to determine whether one exists in the first place. That is the purpose of binary testing; without it, you're just looking at noise without any means to decide what is real and what is not. As a corollary, if you can meaningfully quantify the difference between A and B, then you should have no trouble establishing that they are different. Obviously business decisions are not generally going to uphold the rigor of good science, but what is the purpose of quantifying things when you're as likely to be wrong as your are right?

tedsanders 10y ago
Asking whether A and B are different is not a useful question. There will be a difference 100% of the time. (Though of course sampling may not reliably detect the difference for feasible samples.) The superior question is which is probably better, and by how much. If all you know is that A is 75% likely to be better than B, then go with A. It's useful information, even if it doesn't cross a an arbitrarily preset threshold of 95% or whatever you use. You don't need to wait for certainty to act. In fact, all actions are taken under uncertainty. So it feels incredibly artificial and counterproductive to frame these questions in such a binary, nonlinear way. It's clinging to certainty when certainty does not exist.
- yummyfajitas 10y ago
  Actually even that isn't the best question. The best question is what's the expected reward from choosing A. Consider the scenario of 10% chance B > A, 90% chance they are equal. To me that sounds like a winning bet. In contrast, 70% chance B is a lot better than A, 30% chance B is a lot worse than A sounds like a scary bet. I'll wait and gather more info. Proper decision theory is based on a loss function which incorporates the magnitude of gains/losses rather than just their existence. https://en.wikipedia.org/wiki/Loss_function https://en.wikipedia.org/wiki/Loss_function

korzun 10y ago

> A test with only 10 or 100 samples still gives you information. It gives you weak information, sure, but information nonetheless. If your market consists of 1000 people.

tedsanders 10y ago
Could you elaborate? Even if your market consists of infinite people, it gives you real information.

dxhdr 10y ago

Maybe I'm misunderstanding but isn't the whole point of A/B testing to answer the question "is A better than B?"

Silhouette 10y ago

If you're running an A/B test, by which I mean precisely that you already have two versions available and are simply trying to decide which is most likely to give better results in the future by comparing how successfully they achieve the desired result with all else being equal, then I'm still waiting for someone to explain to me why the correct question is not simply "Which performs better, A or B?" The statistical rigour of all the hypothesis tests and Bayesian methods and so on is laudable, but fundamentally any result and any conclusion you draw from those tests can only ever be as good as the underlying assumptions from which the result was derived. For example, if you choose to perform any statistical test that designates one of A and B the default case and tests for evidence in favour of the other, you have made a determination that the situation is not symmetric. You can choose a level of power or a significance level you want to achieve, but the 5% convention is again arbitrary. The same goes for any prior you choose in a Bayesian method. Fundamentally, if you are truly starting from a neutral situation where you just want to know which of A and B is better, and you truly have no other information to go on other than how they have performed since the start of your measurements with all else being equal, then it remains the case that you still have nothing else to go on no matter what statistical calculations you perform.