4 ms·

For me that's one of the biggest mistakes someone can make when looking at doing A/B tests. This desire to understand the universe gets in the way of improving

For me that's one of the biggest mistakes someone can make when looking at doing A/B tests. This desire to understand the universe gets in the way of improving your website / app / business / bank balance. Yet, it's a common pitfall for many. What's the purpose of those tests? To make you a better UX specialist? To scientifically prove a theory? Or to get the conversion rate up??

ameister14 13y ago
The purpose of the test is to figure out what direction to go in in order to improve the conversion rate. If I understand you right, you question the reason you generally control the variables and want to figure out why something converts better than something else and not just observe the results. Well, firstly, if you figure out what exactly people like about one design versus another, it's easier to reproduce a higher conversion rate in other parts of your site. It also goes towards a further understanding of your customer and what exactly they want from you. If you have a bunch of services on a page and you change several of the offerings at once, as well as the content describing the services and the pictures displayed with them, observing a jump in conversion as you do so, what does that tell you? Do people not want some of the services you offered before? Not necessarily. Do they like the new photos, or does the copy better explain the services? Or is it that they like some of the new services listed? Do they like all of them, or just like some so much it doesn't matter that the others are there? What if you change your price at the same time you change the design of your page and see conversion drop. Was it wrong to change your price, or was the redesign wrong? Controlling your variables makes it so that you don't have to go back to answer all these questions. You get a better gradual understanding of your customer and can more accurately predict performance changes based on future redesign, making you more likely to increase conversion with each new test.
- gingerlime 13y ago
  That's well and good in theory. In practice, it's an excruciatingly long process (unless you at a massive scale already) and you're only likely to achieve local maxima. Whereas big sweeping changes, which might change more than a single feature but the whole message, are more likely to give you a conversion boost. Once you find out what works and produces a big win, then you optimize more gradually and try to figure out what in particular it is. But it's usually not that important at that stage. Your big changes will reveal the stuff that's important to your customers.
  - ameister14 13y ago
    I agree; shot in the dark big changes can be very useful to give you a starting point to optimize from. I wouldn't say they are more likely to give you a conversion boost, though. Why would that be the case? Let's say I design my product and page. It's well thought out and I put some time into it, and the UX isn't bad. Why would changing that drastically be more likely to give me a boost than small changes? Now, the shift might be larger with big changes, that I'd agree with. With a small change you're talking about small increases leading to an aggregate large increase. Then again, if you're established, you're also not risking your customer base. I'd agree that you can go with this as a concept at the start, I just think, as it appears you do, that after you've established a good base you need to go more controlled from there.

ameister14 13y ago

The purpose of the test is to figure out what direction to go in in order to improve the conversion rate. If I understand you right, you question the reason you generally control the variables and want to figure out why something converts better than something else and not just observe the results. Well, firstly, if you figure out what exactly people like about one design versus another, it's easier to reproduce a higher conversion rate in other parts of your site. It also goes towards a further understanding of your customer and what exactly they want from you. If you have a bunch of services on a page and you change several of the offerings at once, as well as the content describing the services and the pictures displayed with them, observing a jump in conversion as you do so, what does that tell you? Do people not want some of the services you offered before? Not necessarily. Do they like the new photos, or does the copy better explain the services? Or is it that they like some of the new services listed? Do they like all of them, or just like some so much it doesn't matter that the others are there? What if you change your price at the same time you change the design of your page and see conversion drop. Was it wrong to change your price, or was the redesign wrong? Controlling your variables makes it so that you don't have to go back to answer all these questions. You get a better gradual understanding of your customer and can more accurately predict performance changes based on future redesign, making you more likely to increase conversion with each new test.

gingerlime 13y ago
That's well and good in theory. In practice, it's an excruciatingly long process (unless you at a massive scale already) and you're only likely to achieve local maxima. Whereas big sweeping changes, which might change more than a single feature but the whole message, are more likely to give you a conversion boost. Once you find out what works and produces a big win, then you optimize more gradually and try to figure out what in particular it is. But it's usually not that important at that stage. Your big changes will reveal the stuff that's important to your customers.
- ameister14 13y ago
  I agree; shot in the dark big changes can be very useful to give you a starting point to optimize from. I wouldn't say they are more likely to give you a conversion boost, though. Why would that be the case? Let's say I design my product and page. It's well thought out and I put some time into it, and the UX isn't bad. Why would changing that drastically be more likely to give me a boost than small changes? Now, the shift might be larger with big changes, that I'd agree with. With a small change you're talking about small increases leading to an aggregate large increase. Then again, if you're established, you're also not risking your customer base. I'd agree that you can go with this as a concept at the start, I just think, as it appears you do, that after you've established a good base you need to go more controlled from there.