5 ms·

Confidence.js – make sense of your A/B test results

49 points by bvanvugt 13y ago

jessicaraygun 13y ago
Hey! I'm the author and I'd be happy to answer any questions you might have. Confidence.js is based on the A/B testing code that we use at sendwithus. A/B testing math is hard and we've worked really hard on making this great. Let us know what you think!
- sharnett 13y ago
  A nit pick: I believe your 'readable' interpretation of the confidence interval isn't quite right: http://en.wikipedia.org/wiki/Confidence_interval#Meaning_and_interpretation http://en.wikipedia.org/wiki/Confidence_interval#Meaning_and...
  - jessicaraygun 13y ago
    Hi sharnett, Thanks for bringing that to my attention - I agree that our explanation is a little more convoluted than it needs to be. I'll update that later today to be more "readable" ;)
- mattmanser 13y ago
  I think making it an object has made the code unnecessarily complicated. I think you could almost 1/2 code with a simple function. This is where I personally think javascript programming is going wrong at the moment, so much boiler plate. You're also programming a bit too defensively. Ask yourself, do you honestly need all those checks? What would actually happen if you got rid of the checks? I think your code would be clearer without them and the programmer would still realize he'd made a mistake and passed in nulls.
  - jessicaraygun 13y ago
    Thanks for the feedback! Confidence.js comes from how we operate on A/B tests at sendwithus - we think of them as variants associated with results. While the code could definitely boil down to a few simple math functions, we find objects to be a good way to organize our data models.
    - hiisi 13y ago
      Also, "getRequiredSampleSize" is called twice for each variant - in the "getResult"'s loop and then inside "hasEnoughData". Same thing for "getVariant". Would you welcome a pull request? Your project is great, I'd totally use at work. Also, it was to learn the math behind that. Thanks!
      - jessicaraygun 13y ago
        Glad you like it, happy to have you contribute :)
- teleclimber 13y ago
  Thank-you for sharing this. Can you recommend a resource that explains the Math for people who slept through statistics class? I actually need to brush up on this topic for a new project so anything you can recommend would be greatly appreciated. Thanks!
  - jessicaraygun 13y ago
    Stats class R sleepy... Real world applications like A/B testing are definitely more interesting. We did a lot of digging and there aren't a lot of resources that are easy to digest - but this one really stood out to us: http://visualwebsiteoptimizer.com/split-testing-blog/what-you-really-need-to-know-about-mathematics-of-ab-split-testing/ http://visualwebsiteoptimizer.com/split-testing-blog/what-yo... I'd love to write a summary of everything I've learned - maybe a blog post in the near future!
  - tagawa 13y ago
    This slide deck worked for me, with the explanations in understandable chunks: http://elem.com/~btilly/effective-ab-testing/ http://elem.com/~btilly/effective-ab-testing/
dsl 13y ago
This is awesome and solves a huge problem. Every startup constantly gets told to A/B test everything, but nobody ever tells them that startups rarely have enough initial traffic to do any meaningful testing. As a result you end up spending a lot of work on what is effectively a random landing page or email template that might actually be worse than what you started with.
- teleclimber 13y ago
  Absolutely true. The other "gotcha" is optimizing for a short term metric (converting visitor to trial) but failing to track whether that cohort actually converts as well to long term paying customers.
- kylerush 13y ago
  Low traffic levels do not mean that you cannot do any meaningful testing. It means that you cannot reliably detect subtle effects in the conversion rate. Whereas Google may be able to reliably detect a 0.005% effect in the conversion rate, a small startup may only be able to reliably detect a 60% effect in the conversion rate. A small startup should arguably not be chasing after small effects in the conversion rate anyway. Evan Miller has a great sample size calculator to illustrate this: http://www.evanmiller.org/ab-testing/sample-size.html http://www.evanmiller.org/ab-testing/sample-size.html In the example above, Google's MDE is 0.005% and the small startup's MDE is 60%. If the small startup's baseline conversion rate is 5%, then with 95% confidence and 80% statistical power, they can reliably detect a 60% effect (positive or negative) with 1,968 visits. It's not just the 60% increase that you should be mindful of. I bet you all small startups want to know if the change they made to their homepage decreases the conversion rate by 60% or more. With an a/b test, they can do that. Another point to keep in mind is that you can learn a lot from a statistical tie (not enough data to conclude there is a difference). In fact, no matter what your traffic levels are, most of your experiments will be a statistical tie. It's important to learn what doesn't work just as it's important to know what does work. This can really help you with prioritization. In the example above, the small startup can use the statistical tie to draw the conclusion that some tasks in their product roadmap will result in less than a 60% effect in the conversion rate and can be prioritized as such. But ultimately it will be quite difficult for the small startup to produce variations that have at least a 60% effect in the conversion rate, but it's not unheard of and again, you can learn from a statistical tie. Just because you have low levels of traffic doesn't mean you can't learn anything from a/b testing.
cleverjake 13y ago
Was expecting this to actually be https://github.com/spumko/confidence https://github.com/spumko/confidence, from WM Labs. Strange both would choose the same name for A/B testing
peter_mcrae 13y ago
Awesome work! Will look at replacing some hand rolled stuff internally. One gap I see with a lot of A/B testing analysis is that it only solves for conversion. While conversion is a great metric for many tests, in my experience, revenue is often the metric that matters most. Whether it's traditional eCommerce or selling tiered subscriptions, a lot of testing is geared towards 1) getting the customer to buy and 2) getting them into a more expensive product or plan. In the subscription scenario, some sort of customer life time value model is even better. I don't pretend to know all the math, but the calcs I've seen focused on revenue (AOV * conversion) need order level data (as opposed to aggregate) so it's not as easy to solve generically.
- jessicaraygun 13y ago
  You're right about revenue being a very important metric, and I think optimizing for conversions has its place too. I would be interested to hear the results of your repurposing!

49 points by bvanvugt 13y ago

Hey! I'm the author and I'd be happy to answer any questions you might have. Confidence.js is based on the A/B testing code that we use at sendwithus. A/B testing math is hard and we've worked really hard on making this great. Let us know what you think!

sharnett 13y ago
A nit pick: I believe your 'readable' interpretation of the confidence interval isn't quite right: http://en.wikipedia.org/wiki/Confidence_interval#Meaning_and_interpretation http://en.wikipedia.org/wiki/Confidence_interval#Meaning_and...
- jessicaraygun 13y ago
  Hi sharnett, Thanks for bringing that to my attention - I agree that our explanation is a little more convoluted than it needs to be. I'll update that later today to be more "readable" ;)
mattmanser 13y ago
I think making it an object has made the code unnecessarily complicated. I think you could almost 1/2 code with a simple function. This is where I personally think javascript programming is going wrong at the moment, so much boiler plate. You're also programming a bit too defensively. Ask yourself, do you honestly need all those checks? What would actually happen if you got rid of the checks? I think your code would be clearer without them and the programmer would still realize he'd made a mistake and passed in nulls.
- jessicaraygun 13y ago
  Thanks for the feedback! Confidence.js comes from how we operate on A/B tests at sendwithus - we think of them as variants associated with results. While the code could definitely boil down to a few simple math functions, we find objects to be a good way to organize our data models.
  - hiisi 13y ago
    Also, "getRequiredSampleSize" is called twice for each variant - in the "getResult"'s loop and then inside "hasEnoughData". Same thing for "getVariant". Would you welcome a pull request? Your project is great, I'd totally use at work. Also, it was to learn the math behind that. Thanks!
    - jessicaraygun 13y ago
      Glad you like it, happy to have you contribute :)
teleclimber 13y ago
Thank-you for sharing this. Can you recommend a resource that explains the Math for people who slept through statistics class? I actually need to brush up on this topic for a new project so anything you can recommend would be greatly appreciated. Thanks!
- jessicaraygun 13y ago
  Stats class R sleepy... Real world applications like A/B testing are definitely more interesting. We did a lot of digging and there aren't a lot of resources that are easy to digest - but this one really stood out to us: http://visualwebsiteoptimizer.com/split-testing-blog/what-you-really-need-to-know-about-mathematics-of-ab-split-testing/ http://visualwebsiteoptimizer.com/split-testing-blog/what-yo... I'd love to write a summary of everything I've learned - maybe a blog post in the near future!
- tagawa 13y ago
  This slide deck worked for me, with the explanations in understandable chunks: http://elem.com/~btilly/effective-ab-testing/ http://elem.com/~btilly/effective-ab-testing/

dsl 13y ago

This is awesome and solves a huge problem. Every startup constantly gets told to A/B test everything, but nobody ever tells them that startups rarely have enough initial traffic to do any meaningful testing. As a result you end up spending a lot of work on what is effectively a random landing page or email template that might actually be worse than what you started with.

teleclimber 13y ago
Absolutely true. The other "gotcha" is optimizing for a short term metric (converting visitor to trial) but failing to track whether that cohort actually converts as well to long term paying customers.
kylerush 13y ago
Low traffic levels do not mean that you cannot do any meaningful testing. It means that you cannot reliably detect subtle effects in the conversion rate. Whereas Google may be able to reliably detect a 0.005% effect in the conversion rate, a small startup may only be able to reliably detect a 60% effect in the conversion rate. A small startup should arguably not be chasing after small effects in the conversion rate anyway. Evan Miller has a great sample size calculator to illustrate this: http://www.evanmiller.org/ab-testing/sample-size.html http://www.evanmiller.org/ab-testing/sample-size.html In the example above, Google's MDE is 0.005% and the small startup's MDE is 60%. If the small startup's baseline conversion rate is 5%, then with 95% confidence and 80% statistical power, they can reliably detect a 60% effect (positive or negative) with 1,968 visits. It's not just the 60% increase that you should be mindful of. I bet you all small startups want to know if the change they made to their homepage decreases the conversion rate by 60% or more. With an a/b test, they can do that. Another point to keep in mind is that you can learn a lot from a statistical tie (not enough data to conclude there is a difference). In fact, no matter what your traffic levels are, most of your experiments will be a statistical tie. It's important to learn what doesn't work just as it's important to know what does work. This can really help you with prioritization. In the example above, the small startup can use the statistical tie to draw the conclusion that some tasks in their product roadmap will result in less than a 60% effect in the conversion rate and can be prioritized as such. But ultimately it will be quite difficult for the small startup to produce variations that have at least a 60% effect in the conversion rate, but it's not unheard of and again, you can learn from a statistical tie. Just because you have low levels of traffic doesn't mean you can't learn anything from a/b testing.

cleverjake 13y ago

Was expecting this to actually be https://github.com/spumko/confidence https://github.com/spumko/confidence, from WM Labs. Strange both would choose the same name for A/B testing

peter_mcrae 13y ago

Awesome work! Will look at replacing some hand rolled stuff internally. One gap I see with a lot of A/B testing analysis is that it only solves for conversion. While conversion is a great metric for many tests, in my experience, revenue is often the metric that matters most. Whether it's traditional eCommerce or selling tiered subscriptions, a lot of testing is geared towards 1) getting the customer to buy and 2) getting them into a more expensive product or plan. In the subscription scenario, some sort of customer life time value model is even better. I don't pretend to know all the math, but the calcs I've seen focused on revenue (AOV * conversion) need order level data (as opposed to aggregate) so it's not as easy to solve generically.

jessicaraygun 13y ago
You're right about revenue being a very important metric, and I think optimizing for conversions has its place too. I would be interested to hear the results of your repurposing!