3 ms·

The Monty Hall problem and what it has to do with A/B testing

• deleted 14y ago
  [deleted]
  • deleted 14y ago
    [deleted]
• deleted 14y ago
  [deleted]
• bthomas 14y ago
  I think the author draws the wrong conclusion from the example of churn. The company didn't want to implement the loyalty program unless it led to a reduction of 10%. After 5000 samples, the observed reduction was 8%. The probability that the true reduction is >= 10% is likely very small.
• jules 14y ago
  All of this madness is elegantly tackled by Bayesian statistics. There are usually two roadblocks to using a Bayesian approach: 1. It's computationally expensive. 2. You need a prior. #1 is not an issue at all here since the problem is tiny. Companies like Visual Website Optimizer are in an excellent spot for #2: because they have data from many customers, they can find a very good prior.
• klochner 14y ago
  Now, what is the comparison between [The Monty Hall] problem and the sample size blog? Well, both are about the usage of additional information to optimize decision making. Am I the only one feeling disappointed with the payoff here? The answer ends up being "almost nothing". It's the geek equivalent of TV gossip programs advertising "Find out next what crazy thing Tom Cruise said to Tiger woods . . . " with the result being "They actually didn't talk."
  • keesschippers 14y ago
    Kevin, Bare in mind this article was written as a follow up on another article that amply describes the concept of statistical power. I think a comparison between the two is justified. Both are about the use of additional information to enhance decision making and both are obviously difficult to grasp. There is a conceptual resemblance between the problem many scientists had a long time ago understanding the dynamics of the probabilities involved in the Monty Hall Problem and the problem many marketing data analists have nowadays trying to understand the dynamics of statistical power when designing experiments. The practical resemblance between the two is that failing to apply the logic behind the Monty Hall Problem will make you throw away half of what you could have had. Not applying the logic of statistical power makes many data analysts and thus organizations throw away half the knowlegde they could have had. Apart from the resemblance between the two puzzles, the practical consequences of such misconceptions are quite worrying to say the least. It’s for that reason I make these efforts to explain the matter. Regards, Kees
• jedberg 14y ago
  In summary: Make sure your sample size is large enough so that it is possible to get a statistically significant result.
  • keesschippers 14y ago
    Indeed, or, if that is not possible, consider lowering reliability. Business situations are different from academic settings where an alpha of 5% is more or less the rule. We do not look for publications in scientific journals, nor do we engage for instance in medical research – and even there -, we are in a business situation trying to optimize decision making. Up to us to decide for an alpha of 10% or 20%, as long as we are fully aware of the consequences of such choices. I am not advocating ‘going easy’ on statistical significance, I am in favour of weighing consequences and such can only be properly done if one masters the dynamics of reliabilty and power of a test. That could also mean you wanted a reliability of 99% and a power of 99%, it all depends on the risks you want take with your specific research questions.