3 ms·

Absolutely true! An A/B test enthusiast in our team once significantly reduced padding on the pricing page to bring the signup button above the fold and used th

Absolutely true! An A/B test enthusiast in our team once significantly reduced padding on the pricing page to bring the signup button above the fold and used the increase in signup button clicks as a proof of success of the experiment. Of course, the pricing page became plain ugly, but that didn't matter, because "Signups are increasing!!" In this case, I do agree that the calculator is a bit daunting if you're not used to all the terms, but what should be done with it should have been an intuitive decision ("what can we do to simplify the calculator?") Not a fan of A/B testing culture that everything needs to be statistically analyzed and proved.

bruce511 2y ago
>> Of course, the pricing page became plain ugly, but that didn't matter, because "Signups are increasing!!" I'm not sure I'm following you here, so perhaps you'd care to elaborate? The GP critique was that it was perhaps just creating a problem elsewhere later on. I'm not seeing the similarity to your case where the change is cosmetic not functional. The issue of whitespace (padding) is subjective (see the conversation recently between the old and new windows control panels) but "scrolling down" does seem to be something that should potentially be avoided. If sign-ups are increasing is that not the goal of the page? Is there reason to believe that the lack of padding is going to be a problem for those users?
- strken 2y ago
  I think one problem is that a better design would move the button above the fold without ruining the spacing, and therefore achieve a better result with even higher lift, but someone focused on just the numbers wouldn't understand this. The fact that the A/B test has a big green number next to it doesn't mean you should stop iterating after one improvement.
  - carlmr 2y ago
    It still seems like a valid use-case for AB testing. Ideally, you should maybe redo the design, something you could AB test if it helps. My guess is yes, because consistency in design usually makes people assume better quality.
    - BoredPositron 2y ago
      A/B tests suck because you are testing against two cases which are probably not the best case. If you take your learnings of the A/B test and iterate your design that's a viable strategy but proposing a shit design and insisting on deploying is wrong.
      - bruce511 2y ago
        That's like saying that comparing a veggie burger to a normal burger sucks because neither are ice cream. A/B tests, by definition, test between A and B. It is very likely that neither is the best option. But how will you find the best option if you don't measure options against each other.
        BoredPositron 2y ago
        The problem is that in 90% of companies the decision is between A/B and not the iterations of them.
        bruce511 2y ago
        I'm going to assume the 90% number was simply hyperbole. Because it's trivially false in any number of ways; Firstly many businesses have never heard of A/B testing, much less apply rigorous application of it to proposed changes. Secondly many businesses have found their niche and don't change anything. There's a reason "that's not how we do it here" is a cliche. Thirdly a whole slew of businesses are greater changing things all the time. My supermarket can't seem yo help themselves iterating on product placement in the store. Blaming testing in general, or A/B testing specifically for some companies being unwilling to change, or iterate, seems to be missing the actual problem. Frankly, with regard yo web sites and software I'd prefer a little -less- change. I just get used to something and whoops, there's a "redesign" so I can learn it all again.
  - bruce511 2y ago
    A/B testing has something to say about (ideally) a single choice. It has nothing to say about optimal solutions. Nothing about A/B testing suggests that you have reached an optimal point, or that you should lock in what you have as being the "best". Now that the button position (or tighter layout) has been noted to have a material effect, more tests can be run to determine any more improvements.
  - II2II 2y ago
    The issue is that A/B testing only looks at outcomes, not reasons. There is a possibility that having the sign-up button above the fold wasn't the contributing factor here, or that it was only a contributing factor. Having to scroll through an estimate may lead a potential customer to believe that pricing is too complex or, worse, that the vendor is trying to hide something. Perhaps there are other reasons. The problem is that A/B testing will only tell you the what and not the why.
    - nottorp 2y ago
      If it was your average modern startup page where you have to scroll down 3 screens to see anything besides generic marketing drivel and licensed stock photos of smiling corporate drones, of course reducing whitespace so people can see what the product is about will increase signups.
  - onlyrealcuzzo 2y ago
    Okay - but the design without padding converted better than the one with padding. A/B tests don't give you the best possible choice of all choices. They just give you the better choice between A & B. The business shouldn't delay rolling something out that increases conversions *significantly* because a better design *might* exist. You can A/B test "better" designs in the future until one converts better.
CookieCrisp 2y ago
I feel like that example is missing some context - if signups did increase then their experiment was successful - we aren’t here to make pretty pages, we’re here to make money.
- shubhamjain 2y ago
  The problem is that it's easy to prove that signups are increasing, and lot harder to prove that there was a measurable increase in number of paying users. Most A/B tests focus on the former, very few on the latter. We had a free plan, and most users who signed up never made a single API request. So, assuming that the increase in signups is driving more business is just foolhardy.
  - blackoil 2y ago
    You can always track signup/paying-users ratio. Purpose of landing/pricing page is to get the users to sign-up. Unless some dark pattern or misinformation is used to confuse users into sign-up, more users is a positive thing.
    - TeMPOraL 2y ago
      The problem with A/B test optimization is that, unless you're extremely careful, they naturally lead you to apply dark patterns and misinformation.
      - foldr 2y ago
        Ok, but the example we're discussing is one where the signup button was simply moved to a different position on the page. That's not a 'dark pattern'.
  - lelanthran 2y ago
    > We had a free plan, and most users who signed up never made a single API request. That doesn't sound like a signup problem; what was the goal behind the free plan? Drive more paying users? Raise company profile? Lock in more users?
PaulHoule 2y ago
If I had a choice between ugly and rich and pretty and poor I'd be sorely tempted by ugly, particularly if I was making the decision for an organization.

bruce511 2y ago

>> Of course, the pricing page became plain ugly, but that didn't matter, because "Signups are increasing!!" I'm not sure I'm following you here, so perhaps you'd care to elaborate? The GP critique was that it was perhaps just creating a problem elsewhere later on. I'm not seeing the similarity to your case where the change is cosmetic not functional. The issue of whitespace (padding) is subjective (see the conversation recently between the old and new windows control panels) but "scrolling down" does seem to be something that should potentially be avoided. If sign-ups are increasing is that not the goal of the page? Is there reason to believe that the lack of padding is going to be a problem for those users?

strken 2y ago
I think one problem is that a better design would move the button above the fold without ruining the spacing, and therefore achieve a better result with even higher lift, but someone focused on just the numbers wouldn't understand this. The fact that the A/B test has a big green number next to it doesn't mean you should stop iterating after one improvement.
- carlmr 2y ago
  It still seems like a valid use-case for AB testing. Ideally, you should maybe redo the design, something you could AB test if it helps. My guess is yes, because consistency in design usually makes people assume better quality.
  - BoredPositron 2y ago
    A/B tests suck because you are testing against two cases which are probably not the best case. If you take your learnings of the A/B test and iterate your design that's a viable strategy but proposing a shit design and insisting on deploying is wrong.
    - bruce511 2y ago
      That's like saying that comparing a veggie burger to a normal burger sucks because neither are ice cream. A/B tests, by definition, test between A and B. It is very likely that neither is the best option. But how will you find the best option if you don't measure options against each other.
      - BoredPositron 2y ago
        The problem is that in 90% of companies the decision is between A/B and not the iterations of them.
        bruce511 2y ago
        I'm going to assume the 90% number was simply hyperbole. Because it's trivially false in any number of ways; Firstly many businesses have never heard of A/B testing, much less apply rigorous application of it to proposed changes. Secondly many businesses have found their niche and don't change anything. There's a reason "that's not how we do it here" is a cliche. Thirdly a whole slew of businesses are greater changing things all the time. My supermarket can't seem yo help themselves iterating on product placement in the store. Blaming testing in general, or A/B testing specifically for some companies being unwilling to change, or iterate, seems to be missing the actual problem. Frankly, with regard yo web sites and software I'd prefer a little -less- change. I just get used to something and whoops, there's a "redesign" so I can learn it all again.
- bruce511 2y ago
  A/B testing has something to say about (ideally) a single choice. It has nothing to say about optimal solutions. Nothing about A/B testing suggests that you have reached an optimal point, or that you should lock in what you have as being the "best". Now that the button position (or tighter layout) has been noted to have a material effect, more tests can be run to determine any more improvements.
- II2II 2y ago
  The issue is that A/B testing only looks at outcomes, not reasons. There is a possibility that having the sign-up button above the fold wasn't the contributing factor here, or that it was only a contributing factor. Having to scroll through an estimate may lead a potential customer to believe that pricing is too complex or, worse, that the vendor is trying to hide something. Perhaps there are other reasons. The problem is that A/B testing will only tell you the what and not the why.
  - nottorp 2y ago
    If it was your average modern startup page where you have to scroll down 3 screens to see anything besides generic marketing drivel and licensed stock photos of smiling corporate drones, of course reducing whitespace so people can see what the product is about will increase signups.
- onlyrealcuzzo 2y ago
  Okay - but the design without padding converted better than the one with padding. A/B tests don't give you the best possible choice of all choices. They just give you the better choice between A & B. The business shouldn't delay rolling something out that increases conversions *significantly* because a better design *might* exist. You can A/B test "better" designs in the future until one converts better.

CookieCrisp 2y ago

I feel like that example is missing some context - if signups did increase then their experiment was successful - we aren’t here to make pretty pages, we’re here to make money.

shubhamjain 2y ago
The problem is that it's easy to prove that signups are increasing, and lot harder to prove that there was a measurable increase in number of paying users. Most A/B tests focus on the former, very few on the latter. We had a free plan, and most users who signed up never made a single API request. So, assuming that the increase in signups is driving more business is just foolhardy.
- blackoil 2y ago
  You can always track signup/paying-users ratio. Purpose of landing/pricing page is to get the users to sign-up. Unless some dark pattern or misinformation is used to confuse users into sign-up, more users is a positive thing.
  - TeMPOraL 2y ago
    The problem with A/B test optimization is that, unless you're extremely careful, they naturally lead you to apply dark patterns and misinformation.
    - foldr 2y ago
      Ok, but the example we're discussing is one where the signup button was simply moved to a different position on the page. That's not a 'dark pattern'.
- lelanthran 2y ago
  > We had a free plan, and most users who signed up never made a single API request. That doesn't sound like a signup problem; what was the goal behind the free plan? Drive more paying users? Raise company profile? Lock in more users?

PaulHoule 2y ago

If I had a choice between ugly and rich and pretty and poor I'd be sorely tempted by ugly, particularly if I was making the decision for an organization.