5 ms·

I appreciate what you're trying to say here, but you've basically just described the prototypical way not to run an A/B test. Ending a test as soon as you reach

I appreciate what you're trying to say here, but you've basically just described the prototypical way not to run an A/B test. Ending a test as soon as you reach significance ensures a high rate of false positives, and doesn't clearly tell you when to end the experiment if there is no effect. See also: http://www.evanmiller.org/how-not-to-run-an-ab-test.html http://www.evanmiller.org/how-not-to-run-an-ab-test.html

btilly 10y ago
You have just pattern-matched to a response without understanding. And you're wrong. See http://www.evanmiller.org/sequential-ab-testing.html http://www.evanmiller.org/sequential-ab-testing.html for a very similar methodology presented by the very person you are citing. The difference is that I am aiming to always produce an answer from the test, even if it is a statistical coin flip, while he aims to only produce an answer roughly 5% of the time if there is no difference.
- in_cahoots 10y ago
  Maybe I am not understanding your post, but aren't you just declaring a winner after N trials, even if that is not significant? That seems to be a critical distinction here.
  - btilly 10y ago
    Right. We are trying to make a business decision, not conduct science. If you go with whichever is ahead, you'll reliably produce correct answers for wins at that you can't reliably measure. Yes, there will be mistakes, but the mistakes are guaranteed to be pretty small. If you insist on a higher standard, you'll learn which choices you were confident of..but now need to figure out what to do when the statistics didn't give you a final answer. I think that the first choice is more useful. Evan prefers to clearly distinguish your coin flips from solid results. But in the end you have to make a decision, and it isn't material what decision you make if the conversion rates are close.
    - in_cahoots 10y ago
      I'm sorry, I wouldn't call that framework A/B testing also much as running two experiments and seeing who's ahead after some arbitrary period of time.
      - rawnlq 10y ago
        I think this post summarizes it really well: http://bjk5.com/post/12829339471/ab-testing-still-works-sarcastic-phew http://bjk5.com/post/12829339471/ab-testing-still-works-sarc... If you have any sort of decent dashboarding the cost of a wrong decision is really not all that bad compared to the cost of being a purist.
        btilly 10y ago
        Oh boy. Let's replace statistics with guesses off of graphs! That visible "zone of stupidity" is based on how long it takes to make a conversion which has everything to do with how your product works, and nothing to do with how statistics works. There is absolutely no difference between the graph you expect that leads to an accidental wrong decision and one that detects a correct difference - the patterns that you think you see don't mean what your brain will decide they do. And more importantly, if you stop in 1/4 of the time that you would have been willing to run the test, the potential loss when you are wrong is twice the worst errors that you could make if you put more effort into it. Have you ever been in an organization that rolled out an innocuous looking change that killed 15% of the business? I have. Over the 10 months it took to find the offending subject line, there was, shall we say, "significant turnover in the executive team". Math exists for a reason. Either learn it, or believe people who have learned it. Don't substitute pretty pictures that you don't understand, then call it an explanation.
        rawnlq 10y ago
        I admit I don't know the math well so I am curious to know how to fix my intuition: Let's say you want to figure out the unknown bias on two coins. You flip both continuously and plot the percentage of heads you see. Due to law of large numbers these percentages will eventually converge to true probabilities (which is how I am interpreting the graphs in that blog post). The bad case is if the two coins are actually "flatlined" in the wrong order so as a pattern matching human you mistakenly believe the rates have converged prematurely. I don't know how to work out the math on this but let's say a "flatline" is visually 100 points or so with no significant slope. Then this should be pretty rare right?
- geoelectric 10y ago
  I take it you're the person who wrote the original to which your linked article responds?
  - btilly 10y ago
    Yes. It took a lot of work to get down to the super simple version. :-)

btilly 10y ago

You have just pattern-matched to a response without understanding. And you're wrong. See http://www.evanmiller.org/sequential-ab-testing.html http://www.evanmiller.org/sequential-ab-testing.html for a very similar methodology presented by the very person you are citing. The difference is that I am aiming to always produce an answer from the test, even if it is a statistical coin flip, while he aims to only produce an answer roughly 5% of the time if there is no difference.

in_cahoots 10y ago
Maybe I am not understanding your post, but aren't you just declaring a winner after N trials, even if that is not significant? That seems to be a critical distinction here.
- btilly 10y ago
  Right. We are trying to make a business decision, not conduct science. If you go with whichever is ahead, you'll reliably produce correct answers for wins at that you can't reliably measure. Yes, there will be mistakes, but the mistakes are guaranteed to be pretty small. If you insist on a higher standard, you'll learn which choices you were confident of..but now need to figure out what to do when the statistics didn't give you a final answer. I think that the first choice is more useful. Evan prefers to clearly distinguish your coin flips from solid results. But in the end you have to make a decision, and it isn't material what decision you make if the conversion rates are close.
  - in_cahoots 10y ago
    I'm sorry, I wouldn't call that framework A/B testing also much as running two experiments and seeing who's ahead after some arbitrary period of time.
    - rawnlq 10y ago
      I think this post summarizes it really well: http://bjk5.com/post/12829339471/ab-testing-still-works-sarcastic-phew http://bjk5.com/post/12829339471/ab-testing-still-works-sarc... If you have any sort of decent dashboarding the cost of a wrong decision is really not all that bad compared to the cost of being a purist.
      - btilly 10y ago
        Oh boy. Let's replace statistics with guesses off of graphs! That visible "zone of stupidity" is based on how long it takes to make a conversion which has everything to do with how your product works, and nothing to do with how statistics works. There is absolutely no difference between the graph you expect that leads to an accidental wrong decision and one that detects a correct difference - the patterns that you think you see don't mean what your brain will decide they do. And more importantly, if you stop in 1/4 of the time that you would have been willing to run the test, the potential loss when you are wrong is twice the worst errors that you could make if you put more effort into it. Have you ever been in an organization that rolled out an innocuous looking change that killed 15% of the business? I have. Over the 10 months it took to find the offending subject line, there was, shall we say, "significant turnover in the executive team". Math exists for a reason. Either learn it, or believe people who have learned it. Don't substitute pretty pictures that you don't understand, then call it an explanation.
        rawnlq 10y ago
        I admit I don't know the math well so I am curious to know how to fix my intuition: Let's say you want to figure out the unknown bias on two coins. You flip both continuously and plot the percentage of heads you see. Due to law of large numbers these percentages will eventually converge to true probabilities (which is how I am interpreting the graphs in that blog post). The bad case is if the two coins are actually "flatlined" in the wrong order so as a pattern matching human you mistakenly believe the rates have converged prematurely. I don't know how to work out the math on this but let's say a "flatline" is visually 100 points or so with no significant slope. Then this should be pretty rare right?
geoelectric 10y ago
I take it you're the person who wrote the original to which your linked article responds?
- btilly 10y ago
  Yes. It took a lot of work to get down to the super simple version. :-)