5 ms·

I like the idea and it clearly worked for you, but there's a problem. You're not controlling your variables. You're changing 20 things at once, and then doing

I like the idea and it clearly worked for you, but there's a problem. You're not controlling your variables. You're changing 20 things at once, and then doing it again to get wildly different tests. That's cool, but you're not going to get the same insight into WHY people convert better until you start controlling your tests more. I get why you're not doing that, you want to change big things drastically and see what happens, but you're kind of shooting in the dark that way. It might work; it might not. Even on the current test you're running, there are multiple different elements between a and b. You can do that, but to do it well you'd need to have a/b/c/d/...so that you can figure out if it's one change, a combination of changes, or what.

darrennix 13y ago
You're right. When a design wins, we have no idea why. We've decided that, for now, we'd rather leave our curiosity unsatisfied rather than slow down the pace of experimentation by 4-6x, which is what would happen if we isolated each experiment variable.
- ameister14 13y ago
  That makes sense. I'm curious about staging, though, as far as making this a repeatable process for others. Do you have a particular conversion rate you've decided to shoot for? Or are you going to do this until you get the sense that you've found the best overall design and make small controlled changes from there? The latter is the impression I got from the blog post. Of course if you end up getting a ton more uniques in the process you can run more tests at once to isolate variables; are you looking to wait till then?
  - darrennix 13y ago
    No, we don't have a specific number we're shooting for. We're just trying everything we can think of and seeing what sticks. We expect that we'll rapidly exhaust the supply of radical ideas and then we'll be left with two or three winning UX paradigms. From there we'll begin doing "normal" A/B testing in a linear fashion.
gingerlime 13y ago
For me that's one of the biggest mistakes someone can make when looking at doing A/B tests. This desire to understand the universe gets in the way of improving your website / app / business / bank balance. Yet, it's a common pitfall for many. What's the purpose of those tests? To make you a better UX specialist? To scientifically prove a theory? Or to get the conversion rate up??
- ameister14 13y ago
  The purpose of the test is to figure out what direction to go in in order to improve the conversion rate. If I understand you right, you question the reason you generally control the variables and want to figure out why something converts better than something else and not just observe the results. Well, firstly, if you figure out what exactly people like about one design versus another, it's easier to reproduce a higher conversion rate in other parts of your site. It also goes towards a further understanding of your customer and what exactly they want from you. If you have a bunch of services on a page and you change several of the offerings at once, as well as the content describing the services and the pictures displayed with them, observing a jump in conversion as you do so, what does that tell you? Do people not want some of the services you offered before? Not necessarily. Do they like the new photos, or does the copy better explain the services? Or is it that they like some of the new services listed? Do they like all of them, or just like some so much it doesn't matter that the others are there? What if you change your price at the same time you change the design of your page and see conversion drop. Was it wrong to change your price, or was the redesign wrong? Controlling your variables makes it so that you don't have to go back to answer all these questions. You get a better gradual understanding of your customer and can more accurately predict performance changes based on future redesign, making you more likely to increase conversion with each new test.
  - gingerlime 13y ago
    That's well and good in theory. In practice, it's an excruciatingly long process (unless you at a massive scale already) and you're only likely to achieve local maxima. Whereas big sweeping changes, which might change more than a single feature but the whole message, are more likely to give you a conversion boost. Once you find out what works and produces a big win, then you optimize more gradually and try to figure out what in particular it is. But it's usually not that important at that stage. Your big changes will reveal the stuff that's important to your customers.
RA_Fisher 13y ago
Exactly. This is why it's important to isolate experiments. Ideally you want what is like a horse stall on your main funnel. Make that stall just 10% of the total funnel so drastic experiments don't hit too many users. I've worked from Optimizely-style, which is Chi-square / Fisherian, to Bayesian to survival. Each time my req. sample sizes generally came down an order of magnitude (having started at 200,000). That's for conversion rates between 2-6%.
- thenomad 13y ago
  I know of Chi-square and Bayesian methods for A/B testing, but what's "survival" as a method? Sounds very interesting...

darrennix 13y ago

You're right. When a design wins, we have no idea why. We've decided that, for now, we'd rather leave our curiosity unsatisfied rather than slow down the pace of experimentation by 4-6x, which is what would happen if we isolated each experiment variable.

ameister14 13y ago
That makes sense. I'm curious about staging, though, as far as making this a repeatable process for others. Do you have a particular conversion rate you've decided to shoot for? Or are you going to do this until you get the sense that you've found the best overall design and make small controlled changes from there? The latter is the impression I got from the blog post. Of course if you end up getting a ton more uniques in the process you can run more tests at once to isolate variables; are you looking to wait till then?
- darrennix 13y ago
  No, we don't have a specific number we're shooting for. We're just trying everything we can think of and seeing what sticks. We expect that we'll rapidly exhaust the supply of radical ideas and then we'll be left with two or three winning UX paradigms. From there we'll begin doing "normal" A/B testing in a linear fashion.

gingerlime 13y ago

For me that's one of the biggest mistakes someone can make when looking at doing A/B tests. This desire to understand the universe gets in the way of improving your website / app / business / bank balance. Yet, it's a common pitfall for many. What's the purpose of those tests? To make you a better UX specialist? To scientifically prove a theory? Or to get the conversion rate up??

ameister14 13y ago
The purpose of the test is to figure out what direction to go in in order to improve the conversion rate. If I understand you right, you question the reason you generally control the variables and want to figure out why something converts better than something else and not just observe the results. Well, firstly, if you figure out what exactly people like about one design versus another, it's easier to reproduce a higher conversion rate in other parts of your site. It also goes towards a further understanding of your customer and what exactly they want from you. If you have a bunch of services on a page and you change several of the offerings at once, as well as the content describing the services and the pictures displayed with them, observing a jump in conversion as you do so, what does that tell you? Do people not want some of the services you offered before? Not necessarily. Do they like the new photos, or does the copy better explain the services? Or is it that they like some of the new services listed? Do they like all of them, or just like some so much it doesn't matter that the others are there? What if you change your price at the same time you change the design of your page and see conversion drop. Was it wrong to change your price, or was the redesign wrong? Controlling your variables makes it so that you don't have to go back to answer all these questions. You get a better gradual understanding of your customer and can more accurately predict performance changes based on future redesign, making you more likely to increase conversion with each new test.
- gingerlime 13y ago
  That's well and good in theory. In practice, it's an excruciatingly long process (unless you at a massive scale already) and you're only likely to achieve local maxima. Whereas big sweeping changes, which might change more than a single feature but the whole message, are more likely to give you a conversion boost. Once you find out what works and produces a big win, then you optimize more gradually and try to figure out what in particular it is. But it's usually not that important at that stage. Your big changes will reveal the stuff that's important to your customers.

RA_Fisher 13y ago

Exactly. This is why it's important to isolate experiments. Ideally you want what is like a horse stall on your main funnel. Make that stall just 10% of the total funnel so drastic experiments don't hit too many users. I've worked from Optimizely-style, which is Chi-square / Fisherian, to Bayesian to survival. Each time my req. sample sizes generally came down an order of magnitude (having started at 200,000). That's for conversion rates between 2-6%.

thenomad 13y ago
I know of Chi-square and Bayesian methods for A/B testing, but what's "survival" as a method? Sounds very interesting...