4 ms·

Not always - a lot depends on your existing conversion rate. See this blog post for details http://visualwebsiteoptimizer.com/split-testing-blog/what-you-really

Not always - a lot depends on your existing conversion rate. See this blog post for details http://visualwebsiteoptimizer.com/split-testing-blog/what-you-really-need-to-know-about-mathematics-of-ab-split-testing/ http://visualwebsiteoptimizer.com/split-testing-blog/what-yo...

bemmu 17y ago
Okay, I followed the instructions in that post, and unless I made a mistake you cannot show an improvement from 10% -> 11% conversion rate if you only had 2000 users to play with. Let's say that from 1000 visitors I have learned that my conversion rate is 10%. Standard error = 0.00948683298. So with 95% confidence conversion rate is actually between 8.14% and 11.86%. I run an experiment on 1000 visitors and seemingly have a improved conversion rate of 11%. But with 95% confidence it is actually between 9.06% and 12.94%. This heavily overlaps the previous result, so the result isn't significant. My takeaway from this is that it isn't worthwhile to do very small tests, but it is possible to get reliable results when doing fewer, bigger changes.
- paraschopra 17y ago
  > My takeaway from this is that it isn't worthwhile to do very small tests, but it is possible to get reliable results when doing fewer, bigger changes. Yup, that is right. If you have less traffic, testing for drastic, bigger changes will definitely give you statistically significant differences (either positive or negative). In fact, even the bigger sites can choose to include just a fraction of their traffic (say 2%) and then test for dramatically different designs/callouts/etc.
- btilly 17y ago
  My takeaway from this is that it isn't worthwhile to do very small tests, but it is possible to get reliable results when doing fewer, bigger changes. That is exactly wrong. You don't want to run a few big tests. Instead run multiple minor tests at the same time. Make your tests be small changes - an email subject line, the size of a button, the help text you're using, etc. And test them in parallel to let you answer questions more quickly. This strategy is not perfect because there may be interaction effects between your different tests. With larger traffic volumes you can address that problem. With smaller ones you need to hope that interactions are not significant. Usually interactions are not that significant, so you'll get away with the simple strategy. What sorts of things should you test? Here are some ideas. Button size, page layout, help text, adding/removing form questions, email text, email subject lines, landing pages, registration pages, and so on and so forth. Interestingly the overall aesthetics of your site can be tested but seldom show a result. You tend to get payoffs from stuff that directly affects user interactions at points where users are not very motivated to continue with your site. If you're going to be at the SXSW conference next month, the CEO of Freshbooks has a panel on A/B testing on Sunday that will give you some great concrete ideas of what has worked for a variety of companies. I highly recommend it. (Disclaimer, I'm on said panel.)
  - paraschopra 17y ago
    I think what he is saying is that running minor test on extremely small traffic isn't going to give significant results soon and I would agree with him. Though I agree with you that doing minor tests is a better strategy given you have enough traffic for that.
    - btilly 17y ago
      Define "minor test". I say that because the biggest bang for the buck that I've seen from A/B tests have tended to be really small changes. Take out an unneeded form element. Take out the verbose explanations of questions so the form looked less intimidating. Choose an email subject line that attracted the right group of people for that email. Most developers would think of those as minor changes. None took more than a couple of hours to code. But I've seen those changes result in 10-40% lifts to the bottom line. That's not a minor change to the business as a whole. More importantly the figures that were quoted suggest about 800 conversions/day. The patience to let tests run 2-4 weeks can result in measuring that level of improvement. And after you have a few wins of that size, you start a virtuous cycle where every improvement makes it easier to raise traffic volume, which makes it easier to run A/B tests, which makes it easier to find further improvements.

bemmu 17y ago

Okay, I followed the instructions in that post, and unless I made a mistake you cannot show an improvement from 10% -> 11% conversion rate if you only had 2000 users to play with. Let's say that from 1000 visitors I have learned that my conversion rate is 10%. Standard error = 0.00948683298. So with 95% confidence conversion rate is actually between 8.14% and 11.86%. I run an experiment on 1000 visitors and seemingly have a improved conversion rate of 11%. But with 95% confidence it is actually between 9.06% and 12.94%. This heavily overlaps the previous result, so the result isn't significant. My takeaway from this is that it isn't worthwhile to do very small tests, but it is possible to get reliable results when doing fewer, bigger changes.

paraschopra 17y ago
> My takeaway from this is that it isn't worthwhile to do very small tests, but it is possible to get reliable results when doing fewer, bigger changes. Yup, that is right. If you have less traffic, testing for drastic, bigger changes will definitely give you statistically significant differences (either positive or negative). In fact, even the bigger sites can choose to include just a fraction of their traffic (say 2%) and then test for dramatically different designs/callouts/etc.
btilly 17y ago
My takeaway from this is that it isn't worthwhile to do very small tests, but it is possible to get reliable results when doing fewer, bigger changes. That is exactly wrong. You don't want to run a few big tests. Instead run multiple minor tests at the same time. Make your tests be small changes - an email subject line, the size of a button, the help text you're using, etc. And test them in parallel to let you answer questions more quickly. This strategy is not perfect because there may be interaction effects between your different tests. With larger traffic volumes you can address that problem. With smaller ones you need to hope that interactions are not significant. Usually interactions are not that significant, so you'll get away with the simple strategy. What sorts of things should you test? Here are some ideas. Button size, page layout, help text, adding/removing form questions, email text, email subject lines, landing pages, registration pages, and so on and so forth. Interestingly the overall aesthetics of your site can be tested but seldom show a result. You tend to get payoffs from stuff that directly affects user interactions at points where users are not very motivated to continue with your site. If you're going to be at the SXSW conference next month, the CEO of Freshbooks has a panel on A/B testing on Sunday that will give you some great concrete ideas of what has worked for a variety of companies. I highly recommend it. (Disclaimer, I'm on said panel.)
- paraschopra 17y ago
  I think what he is saying is that running minor test on extremely small traffic isn't going to give significant results soon and I would agree with him. Though I agree with you that doing minor tests is a better strategy given you have enough traffic for that.
  - btilly 17y ago
    Define "minor test". I say that because the biggest bang for the buck that I've seen from A/B tests have tended to be really small changes. Take out an unneeded form element. Take out the verbose explanations of questions so the form looked less intimidating. Choose an email subject line that attracted the right group of people for that email. Most developers would think of those as minor changes. None took more than a couple of hours to code. But I've seen those changes result in 10-40% lifts to the bottom line. That's not a minor change to the business as a whole. More importantly the figures that were quoted suggest about 800 conversions/day. The patience to let tests run 2-4 weeks can result in measuring that level of improvement. And after you have a few wins of that size, you start a virtuous cycle where every improvement makes it easier to raise traffic volume, which makes it easier to run A/B tests, which makes it easier to find further improvements.