7 ms·

I think one problem is that a better design would move the button above the fold without ruining the spacing, and therefore achieve a better result with even hi

by strken 2y ago

I think one problem is that a better design would move the button above the fold without ruining the spacing, and therefore achieve a better result with even higher lift, but someone focused on just the numbers wouldn't understand this. The fact that the A/B test has a big green number next to it doesn't mean you should stop iterating after one improvement.

carlmr 2y ago
It still seems like a valid use-case for AB testing. Ideally, you should maybe redo the design, something you could AB test if it helps. My guess is yes, because consistency in design usually makes people assume better quality.
- BoredPositron 2y ago
  A/B tests suck because you are testing against two cases which are probably not the best case. If you take your learnings of the A/B test and iterate your design that's a viable strategy but proposing a shit design and insisting on deploying is wrong.
  - bruce511 2y ago
    That's like saying that comparing a veggie burger to a normal burger sucks because neither are ice cream. A/B tests, by definition, test between A and B. It is very likely that neither is the best option. But how will you find the best option if you don't measure options against each other.
    - BoredPositron 2y ago
      The problem is that in 90% of companies the decision is between A/B and not the iterations of them.
      - bruce511 2y ago
        I'm going to assume the 90% number was simply hyperbole. Because it's trivially false in any number of ways; Firstly many businesses have never heard of A/B testing, much less apply rigorous application of it to proposed changes. Secondly many businesses have found their niche and don't change anything. There's a reason "that's not how we do it here" is a cliche. Thirdly a whole slew of businesses are greater changing things all the time. My supermarket can't seem yo help themselves iterating on product placement in the store. Blaming testing in general, or A/B testing specifically for some companies being unwilling to change, or iterate, seems to be missing the actual problem. Frankly, with regard yo web sites and software I'd prefer a little -less- change. I just get used to something and whoops, there's a "redesign" so I can learn it all again.
bruce511 2y ago
A/B testing has something to say about (ideally) a single choice. It has nothing to say about optimal solutions. Nothing about A/B testing suggests that you have reached an optimal point, or that you should lock in what you have as being the "best". Now that the button position (or tighter layout) has been noted to have a material effect, more tests can be run to determine any more improvements.
II2II 2y ago
The issue is that A/B testing only looks at outcomes, not reasons. There is a possibility that having the sign-up button above the fold wasn't the contributing factor here, or that it was only a contributing factor. Having to scroll through an estimate may lead a potential customer to believe that pricing is too complex or, worse, that the vendor is trying to hide something. Perhaps there are other reasons. The problem is that A/B testing will only tell you the what and not the why.
- nottorp 2y ago
  If it was your average modern startup page where you have to scroll down 3 screens to see anything besides generic marketing drivel and licensed stock photos of smiling corporate drones, of course reducing whitespace so people can see what the product is about will increase signups.
onlyrealcuzzo 2y ago
Okay - but the design without padding converted better than the one with padding. A/B tests don't give you the best possible choice of all choices. They just give you the better choice between A & B. The business shouldn't delay rolling something out that increases conversions *significantly* because a better design *might* exist. You can A/B test "better" designs in the future until one converts better.

by strken 2y ago

carlmr 2y ago

It still seems like a valid use-case for AB testing. Ideally, you should maybe redo the design, something you could AB test if it helps. My guess is yes, because consistency in design usually makes people assume better quality.

BoredPositron 2y ago
A/B tests suck because you are testing against two cases which are probably not the best case. If you take your learnings of the A/B test and iterate your design that's a viable strategy but proposing a shit design and insisting on deploying is wrong.
- bruce511 2y ago
  That's like saying that comparing a veggie burger to a normal burger sucks because neither are ice cream. A/B tests, by definition, test between A and B. It is very likely that neither is the best option. But how will you find the best option if you don't measure options against each other.
  - BoredPositron 2y ago
    The problem is that in 90% of companies the decision is between A/B and not the iterations of them.
    - bruce511 2y ago
      I'm going to assume the 90% number was simply hyperbole. Because it's trivially false in any number of ways; Firstly many businesses have never heard of A/B testing, much less apply rigorous application of it to proposed changes. Secondly many businesses have found their niche and don't change anything. There's a reason "that's not how we do it here" is a cliche. Thirdly a whole slew of businesses are greater changing things all the time. My supermarket can't seem yo help themselves iterating on product placement in the store. Blaming testing in general, or A/B testing specifically for some companies being unwilling to change, or iterate, seems to be missing the actual problem. Frankly, with regard yo web sites and software I'd prefer a little -less- change. I just get used to something and whoops, there's a "redesign" so I can learn it all again.

bruce511 2y ago

A/B testing has something to say about (ideally) a single choice. It has nothing to say about optimal solutions. Nothing about A/B testing suggests that you have reached an optimal point, or that you should lock in what you have as being the "best". Now that the button position (or tighter layout) has been noted to have a material effect, more tests can be run to determine any more improvements.

II2II 2y ago

The issue is that A/B testing only looks at outcomes, not reasons. There is a possibility that having the sign-up button above the fold wasn't the contributing factor here, or that it was only a contributing factor. Having to scroll through an estimate may lead a potential customer to believe that pricing is too complex or, worse, that the vendor is trying to hide something. Perhaps there are other reasons. The problem is that A/B testing will only tell you the what and not the why.

nottorp 2y ago
If it was your average modern startup page where you have to scroll down 3 screens to see anything besides generic marketing drivel and licensed stock photos of smiling corporate drones, of course reducing whitespace so people can see what the product is about will increase signups.

onlyrealcuzzo 2y ago

Okay - but the design without padding converted better than the one with padding. A/B tests don't give you the best possible choice of all choices. They just give you the better choice between A & B. The business shouldn't delay rolling something out that increases conversions *significantly* because a better design *might* exist. You can A/B test "better" designs in the future until one converts better.