5 ms·

I don't know what's more surprising, that one small change is worth millions or that A/B testing is still news to some companies.

by samd 16y ago

brianbreslin 16y ago
Wasn't there a famous post a few years back by amazon about their a/b tests resulting in 60million in sales increases, just by changing the button colors? Everyone working on ecommerce sites of this scale should be expected to know about this stuff
patio11 16y ago
I have spoken to a lot of companies who have A/B testing somewhere on their radar, but some combination of "That's nobody's job, so we haven't actually started doing it", (sometimes) organizational infighting (marketing wants to start, engineering has higher priorities and refuses to allocate resources to "twiddling text", etc), and uncertainty about outcomes means they don't start it yet. One of my big tasks at some consulting clients is convincing people to buy into A/B testing. The easy way is to show them how their website is leaking staggering sums of money, and present A/B testing as the cheap, reliable solution to that problem in the future. These are smart software companies who transact most of their business online, by the way. The broader economy which isn't primarily web-focused has many, many, many opportunities to improve their execution.
leftnode 16y ago
Yup, it's shocking. However, is this more of a problem with A/B testing or just bad user experience/design? I suppose an A/B test would've been to remove the Company field from the payment process and see how things have performed. I think companies are more concerned with more obvious A/B tests (button color, page headings) than with testing large changes to their site or testing more in depth things like user experience.
thibaut_barrere 16y ago
A/B anything (testing or dual projects at once etc) is seen in some places as a "lack of skills, otherwise you would know beforehand". (which I don't agree with, in case someone doubts!)
matwood 16y ago
I don't know what's more surprising, that one small change is worth millions or that A/B testing is still news to some companies. Yeah, my company has been doing A/B (we call it Champion/Challenger) testing for years.
njharman 16y ago
This wasn't A/B testing. This was fixing UI mistakes based on UX analysis. There was no 'B'. They weren't comparing A and B.
- CapitalistCartr 16y ago
  In this case, the A/B testing would have been with that field and without.
  - pavel_lishin 16y ago
    How would you know which fields to use?
- samd 16y ago
  UX analysis and A/B testing go hand in hand. There'd be no way to know that the Company field was a UI mistake if it had not been tested.
aplusbi 16y ago
A/B testing probably wouldn't have caught this.

I don't know what's more surprising, that one small change is worth millions or that A/B testing is still news to some companies.

by samd 16y ago

I don't know what's more surprising, that one small change is worth millions or that A/B testing is still news to some companies.

brianbreslin 16y ago

Wasn't there a famous post a few years back by amazon about their a/b tests resulting in 60million in sales increases, just by changing the button colors? Everyone working on ecommerce sites of this scale should be expected to know about this stuff

patio11 16y ago

I have spoken to a lot of companies who have A/B testing somewhere on their radar, but some combination of "That's nobody's job, so we haven't actually started doing it", (sometimes) organizational infighting (marketing wants to start, engineering has higher priorities and refuses to allocate resources to "twiddling text", etc), and uncertainty about outcomes means they don't start it yet. One of my big tasks at some consulting clients is convincing people to buy into A/B testing. The easy way is to show them how their website is leaking staggering sums of money, and present A/B testing as the cheap, reliable solution to that problem in the future. These are smart software companies who transact most of their business online, by the way. The broader economy which isn't primarily web-focused has many, many, many opportunities to improve their execution.

leftnode 16y ago

Yup, it's shocking. However, is this more of a problem with A/B testing or just bad user experience/design? I suppose an A/B test would've been to remove the Company field from the payment process and see how things have performed. I think companies are more concerned with more obvious A/B tests (button color, page headings) than with testing large changes to their site or testing more in depth things like user experience.

thibaut_barrere 16y ago

A/B anything (testing or dual projects at once etc) is seen in some places as a "lack of skills, otherwise you would know beforehand". (which I don't agree with, in case someone doubts!)

matwood 16y ago

I don't know what's more surprising, that one small change is worth millions or that A/B testing is still news to some companies. Yeah, my company has been doing A/B (we call it Champion/Challenger) testing for years.

njharman 16y ago

This wasn't A/B testing. This was fixing UI mistakes based on UX analysis. There was no 'B'. They weren't comparing A and B.

CapitalistCartr 16y ago
In this case, the A/B testing would have been with that field and without.
- pavel_lishin 16y ago
  How would you know which fields to use?
samd 16y ago
UX analysis and A/B testing go hand in hand. There'd be no way to know that the Company field was a UI mistake if it had not been tested.

aplusbi 16y ago

A/B testing probably wouldn't have caught this.