4 ms·
I love when smart companies with great brands double business results with a few hours of A/B testing. The more this happens, the more you have to think: wait
by patio11 15y ago
I love when smart companies with great brands double business results with a few hours of A/B testing. The more this happens, the more you have to think: wait a second, if ten years of obsessive development and marketing and branding gets us X sales, and 10 hours of A/B testing added X sales to that... what the heck was I doing last week again. Answer: something that did not double sales.
I bang this drum frequently, but let me bang it one more time: you should be A/B testing.
- davidw 15y ago> you should be A/B testing. 37 signals should be A/B testing, because they have the volume to do it. Many of us don't, or aren't sure. Anyone want to comment on what sort of numbers you should be looking at, roughly, in order to get statistically significant results?
- Periodic 15y agoI'm interested in this too. It sounds like it would be hard to do A/B testing pre-launch with our tiny volume, and unless we get a significant amount of volume immediately post-launch we won't have much data. However, I do plan to get a framework in place for A/B testing immediately after launch as we go into a refinement phase.
- jcampbell1 15y agoI have found you need about 200 conversions and to let the experiments run for a 2 months to get rid of the novelty effect (regardless if a winner has been found). A conversion must also be a real sale, by that, I mean people make the mistake of using visits or free signups, and the landing page is optimized for clicks and free accounts not revenue. A perfect example of a failure here is Adwords.com. The Adwords landing page is designed to get people to signup for an account, rather than to generate revenue. 99% of adwords signups lead to dead accounts, because google does a bad job of setting customer expectations on the marketing site.
- patio11 15y agoIt's sensitive to your volume, conversion rates, and the magnitude of the difference in conversion rates between A and B. In general, volume is good, higher conversion rates (or easier intermediate conversions, like to email submit rather than to purchase) are good, and large differences in conversion rates are good. It would take a lot of time to statistically significantly discriminate between .001% and .0012% on volumes of 5 visitors a day. People like easy answers, though, so my easy answer is "A/B testing is worthwhile if you have 100 visitors a day. Otherwise, spend the bulk of your time finding 100 visitors a day first."
- arnorhs 15y agoIs 100 a day enough? Wouldn't it take forever to get statistical significance? (unless the results are really dramatic)
- metachris 15y agoPerhaps. Let's say you run the test for 5 days (500 users), and 80% of conversions happen on variant A, it does provide you with something to work with and experiment further.
- LargeWu 15y agoIt depends on how much variance you normally have, and the size of the change you hope to measure. http://en.wikipedia.org/wiki/Statistical_power http://en.wikipedia.org/wiki/Statistical_power
- noelwelsh 15y agoCheck out Ron Kohavi's publications for a way to calculate this: http://robotics.stanford.edu/~ronnyk/ronnyk-bib.html http://robotics.stanford.edu/~ronnyk/ronnyk-bib.html (He has a bunch of publications that are just the same thing repackaged, so it probably doesn't matter which one you read.) For example, if you have a conversion rate of 5% and want to detect an increase of 5% you need about 120'000 hits.
- chaz 15y agoOne simple and clever way to test for significant results is to run an A/A/B test. When your two A buckets converge, you've probably done enough testing.
- mitko 15y agoor just calculate statistical significance?
- joelg87 15y ago> Anyone want to comment on what sort of numbers you should be looking at, roughly, in order to get statistically significant results? This is a very important point. We've found you have to go with your gut until a point. We've just started doing lots of A/B testing for Buffer (http://bufferapp.com http://bufferapp.com). We are now getting 300-500 signups per day, and we currently have around a 10% conversion so you can work out how much traffic that is. We've tried A/B testing in the past without success, and I think now is the right time for us with the right volume for A/B testing to have a significant impact. One important thing: we've started A/B testing aggressively (a new test almost daily, now onto our 4th test. We've tested 3 different variations and had no significant results. I expect this to take a long time, and posts like this make it seem easy to get significant changes in conversion. A/B testing is hard.
- adityakothadiya 15y agoI don't think so this particular case is just "few hours of A/B testing". They were testing pretty drastic design changes - and it takes lot more than few hours to come up with design concepts, copyright text, and then actual implementation. These were not typical "test different landing page title or action call button color" tests. These were brand new design changes. So for established companies, this is absolutely the right approach - as your product is already ready, now just increase the sales. But for many early stage companies, when product development itself is constantly evolving and demanding, spending so much time on different design iterations is not always possible. Agree, their results are phenomenal, but not sure how many companies understand this before they try.