4 ms·

That's putting the cart before the horse. Before you can quantify a difference, you have to determine whether one exists in the first place. That is the purpo

That's putting the cart before the horse. Before you can quantify a difference, you have to determine whether one exists in the first place. That is the purpose of binary testing; without it, you're just looking at noise without any means to decide what is real and what is not. As a corollary, if you can meaningfully quantify the difference between A and B, then you should have no trouble establishing that they are different. Obviously business decisions are not generally going to uphold the rigor of good science, but what is the purpose of quantifying things when you're as likely to be wrong as your are right?

tedsanders 10y ago
Asking whether A and B are different is not a useful question. There will be a difference 100% of the time. (Though of course sampling may not reliably detect the difference for feasible samples.) The superior question is which is probably better, and by how much. If all you know is that A is 75% likely to be better than B, then go with A. It's useful information, even if it doesn't cross a an arbitrarily preset threshold of 95% or whatever you use. You don't need to wait for certainty to act. In fact, all actions are taken under uncertainty. So it feels incredibly artificial and counterproductive to frame these questions in such a binary, nonlinear way. It's clinging to certainty when certainty does not exist.
- yummyfajitas 10y ago
  Actually even that isn't the best question. The best question is what's the expected reward from choosing A. Consider the scenario of 10% chance B > A, 90% chance they are equal. To me that sounds like a winning bet. In contrast, 70% chance B is a lot better than A, 30% chance B is a lot worse than A sounds like a scary bet. I'll wait and gather more info. Proper decision theory is based on a loss function which incorporates the magnitude of gains/losses rather than just their existence. https://en.wikipedia.org/wiki/Loss_function https://en.wikipedia.org/wiki/Loss_function

tedsanders 10y ago

Asking whether A and B are different is not a useful question. There will be a difference 100% of the time. (Though of course sampling may not reliably detect the difference for feasible samples.) The superior question is which is probably better, and by how much. If all you know is that A is 75% likely to be better than B, then go with A. It's useful information, even if it doesn't cross a an arbitrarily preset threshold of 95% or whatever you use. You don't need to wait for certainty to act. In fact, all actions are taken under uncertainty. So it feels incredibly artificial and counterproductive to frame these questions in such a binary, nonlinear way. It's clinging to certainty when certainty does not exist.

yummyfajitas 10y ago
Actually even that isn't the best question. The best question is what's the expected reward from choosing A. Consider the scenario of 10% chance B > A, 90% chance they are equal. To me that sounds like a winning bet. In contrast, 70% chance B is a lot better than A, 30% chance B is a lot worse than A sounds like a scary bet. I'll wait and gather more info. Proper decision theory is based on a loss function which incorporates the magnitude of gains/losses rather than just their existence. https://en.wikipedia.org/wiki/Loss_function https://en.wikipedia.org/wiki/Loss_function