6 ms·

Architecture update: leaner, meaner and faster A/B tests

11 points by paraschopra 17y ago

btilly 17y ago
Until now you could run only one test per page. While this is the way it should actually be... Um, no. A/B testing is a balancing act between theory and pragmatism. Everyone wants to get the perfect answer, but we need to get it fast enough to make useful business decisions. When your website doesn't have that much traffic this means you need to make compromises. Running multiple tests at a time on a page is such a compromise. The one time you absolutely cannot justify multiple tests at the same time due to predictable interaction effects is when you've assigned users a sequence and use the sequence number mod n to figure out which version they get. That's bad, so don't do that. Even if the tests are on different pages you'll get interaction effects. Assign people to each test randomly instead.
- paraschopra 17y ago
  To clarify, I meant that you coudln't run multiple tests per page _using_ VWO. Of course, it has been done previously. Yes, running multiple tests is risky and we discourage it because even if you gain speed, the interaction effects may dissolve any results you get. Again, your last point is excellent. Assigning user ids and then choosing variations dependent on it is a poor strategy. In VWO, we assign variations randomly and then remember (via a cookie) which variation did the user see.
  - patio11 17y ago
    Yes, running multiple tests is risky and we discourage it because even if you gain speed, the interaction effects may dissolve any results you get. Ben has IIRC previously written, and I agree, that interaction effects are typically minimal in practice unless you have catastrophic implementation choices. Since you're taking care of the implementation and it isn't catastrophic, you can safely ignore them. Don't advise your users against using your software in a manner likely to increase the value they get out of it. Interaction effects are worth an asterix in the documentation, but not much more.
    - paraschopra 17y ago
      Yes, you are probably true. But I have seen users permanently implementing a variation on the slightest hint of improvement, so it is better to take a strict stance because they are going to ignore it anyway.
briancray 17y ago
I've been using VWO, and highly recommend it.
danskil 17y ago
I like the service and just applied for an invite. The demo on the main page shows changing text can you also change css, swap images, or use different html elements?
- paraschopra 17y ago
  Yes, see this post for details http://visualwebsiteoptimizer.com/split-testing-blog/web-designers-and-developers-we-have-changed-ab-testing-forever/ http://visualwebsiteoptimizer.com/split-testing-blog/web-des...
nopassrecover 17y ago
This is the second time today I've seen a service using one of the admin panel templates from ThemeForest. In any case it's not the interface that matters so much in this case.
- paraschopra 17y ago
  Yep, admin interface from ThemeForest provide a really good looking base for apps such as VWO to build their UI. I love ThemeForest and highly recommend it for all kinds of design needs.

11 points by paraschopra 17y ago

Until now you could run only one test per page. While this is the way it should actually be... Um, no. A/B testing is a balancing act between theory and pragmatism. Everyone wants to get the perfect answer, but we need to get it fast enough to make useful business decisions. When your website doesn't have that much traffic this means you need to make compromises. Running multiple tests at a time on a page is such a compromise. The one time you absolutely cannot justify multiple tests at the same time due to predictable interaction effects is when you've assigned users a sequence and use the sequence number mod n to figure out which version they get. That's bad, so don't do that. Even if the tests are on different pages you'll get interaction effects. Assign people to each test randomly instead.

paraschopra 17y ago
To clarify, I meant that you coudln't run multiple tests per page _using_ VWO. Of course, it has been done previously. Yes, running multiple tests is risky and we discourage it because even if you gain speed, the interaction effects may dissolve any results you get. Again, your last point is excellent. Assigning user ids and then choosing variations dependent on it is a poor strategy. In VWO, we assign variations randomly and then remember (via a cookie) which variation did the user see.
- patio11 17y ago
  Yes, running multiple tests is risky and we discourage it because even if you gain speed, the interaction effects may dissolve any results you get. Ben has IIRC previously written, and I agree, that interaction effects are typically minimal in practice unless you have catastrophic implementation choices. Since you're taking care of the implementation and it isn't catastrophic, you can safely ignore them. Don't advise your users against using your software in a manner likely to increase the value they get out of it. Interaction effects are worth an asterix in the documentation, but not much more.
  - paraschopra 17y ago
    Yes, you are probably true. But I have seen users permanently implementing a variation on the slightest hint of improvement, so it is better to take a strict stance because they are going to ignore it anyway.

briancray 17y ago

I've been using VWO, and highly recommend it.

danskil 17y ago

I like the service and just applied for an invite. The demo on the main page shows changing text can you also change css, swap images, or use different html elements?

paraschopra 17y ago
Yes, see this post for details http://visualwebsiteoptimizer.com/split-testing-blog/web-designers-and-developers-we-have-changed-ab-testing-forever/ http://visualwebsiteoptimizer.com/split-testing-blog/web-des...

nopassrecover 17y ago

This is the second time today I've seen a service using one of the admin panel templates from ThemeForest. In any case it's not the interface that matters so much in this case.

paraschopra 17y ago
Yep, admin interface from ThemeForest provide a really good looking base for apps such as VWO to build their UI. I love ThemeForest and highly recommend it for all kinds of design needs.