6 ms·

>We actually don't do a ton of A/B testing or worry much about open and click rates, and here's why: Our revenue model is subscription based. We don't take any

by jpster 9y ago

>We actually don't do a ton of A/B testing or worry much about open and click rates, and here's why: Our revenue model is subscription based. We don't take any commissions or have any ads in the premium emails. Our only incentive is to keep premium subscribers happy. I find this a surprising statement -- aren't the emails the primary channel for trying to convince a free user to upgrade to paid? Why not try A/B testing to try and boost that percentage from the ~10% to something higher?

napworth 9y ago
In most cases, A/B testing will actually do negative damage than if you didn't test it at all.
- jdpedrie 9y ago
  [citation needed]
- eddigo 9y ago
  And I would bet in a large subset of those cases it's because the person implementing those tests and the person interpreting the results aren't the same. And that neither understands basic statistical hypothesis testing.
- jpster 9y ago
  Depends on if the tester actually knows what they're doing. https://blog.kissmetrics.com/your-ab-tests-are-illusory/ https://blog.kissmetrics.com/your-ab-tests-are-illusory/
  - ssharp 9y ago
    In most cases, flying a plane yourself will result in more negative damage than having a trained pilot fly you.
- ssharp 9y ago
  Interested in further thought here.
rangersanger 9y ago
He did say they "don't do a ton," so they are definitely doing some, maybe that's where? That said, I do appreciate the sentiment here. This is a different context entirely, but in my ecomm past a/b testing was often employed by those seeking the politically safest growth path. I considered it proactive CYA where Product Managers would prefer premature optimization over having an opinion on what's best for the user. I witnessed what I considered an unhealthy amount of a/b testing resources spent on micro adjustments eg moving a buy button a few pixels or changing the color slightly. There was very little calculated risk taking, which can be enabled with A/B testing employed differently. There's absolutely a place for those kinds of optimizations but I think they come much later in product maturity than when many organizations actually begin using them. Then there's an entire other discussion about understanding the math behind building a relevant test and interpreting the results.
- bkidwell 9y ago
  You're right. We do some A/B testing when we're running promotions (test headlines, CTAs, etc.) but that's about it. There is a lot that goes into correctly running split tests on the website and interpreting those results. We decided that once something is working good enough we'd focus on other areas of the business. However, now that the team is larger and we have resources to dedicate to testing and optimizing we will likely start doing this by the end of the year. Like you said, moving buttons and changing colors is really just a waste of time. You have to spend the time to do the analytics research, get customer feedback, etc. to first understand the problem so you can come up with a solution. Changing the color of a button doesn't solve a problem. Rather than wasting resources on doing conversion optimization the wrong way, we've decided to hold off until we can do it the right way.

by jpster 9y ago

napworth 9y ago

In most cases, A/B testing will actually do negative damage than if you didn't test it at all.

jdpedrie 9y ago
[citation needed]
eddigo 9y ago
And I would bet in a large subset of those cases it's because the person implementing those tests and the person interpreting the results aren't the same. And that neither understands basic statistical hypothesis testing.
jpster 9y ago
Depends on if the tester actually knows what they're doing. https://blog.kissmetrics.com/your-ab-tests-are-illusory/ https://blog.kissmetrics.com/your-ab-tests-are-illusory/
- ssharp 9y ago
  In most cases, flying a plane yourself will result in more negative damage than having a trained pilot fly you.
ssharp 9y ago
Interested in further thought here.

rangersanger 9y ago

He did say they "don't do a ton," so they are definitely doing some, maybe that's where? That said, I do appreciate the sentiment here. This is a different context entirely, but in my ecomm past a/b testing was often employed by those seeking the politically safest growth path. I considered it proactive CYA where Product Managers would prefer premature optimization over having an opinion on what's best for the user. I witnessed what I considered an unhealthy amount of a/b testing resources spent on micro adjustments eg moving a buy button a few pixels or changing the color slightly. There was very little calculated risk taking, which can be enabled with A/B testing employed differently. There's absolutely a place for those kinds of optimizations but I think they come much later in product maturity than when many organizations actually begin using them. Then there's an entire other discussion about understanding the math behind building a relevant test and interpreting the results.

bkidwell 9y ago
You're right. We do some A/B testing when we're running promotions (test headlines, CTAs, etc.) but that's about it. There is a lot that goes into correctly running split tests on the website and interpreting those results. We decided that once something is working good enough we'd focus on other areas of the business. However, now that the team is larger and we have resources to dedicate to testing and optimizing we will likely start doing this by the end of the year. Like you said, moving buttons and changing colors is really just a waste of time. You have to spend the time to do the analytics research, get customer feedback, etc. to first understand the problem so you can come up with a solution. Changing the color of a button doesn't solve a problem. Rather than wasting resources on doing conversion optimization the wrong way, we've decided to hold off until we can do it the right way.