4 ms·

I'm also guessing that someone offended by the profanity would likely not be a happy customer. Which highlights an interesting potential limit of A/B testing fo

I'm also guessing that someone offended by the profanity would likely not be a happy customer. Which highlights an interesting potential limit of A/B testing for conversions. It may take some time after the point of sale to determine whether you've created a happy (and therefor loyal, returning) customer or not. So imagine in this case that profanity is likely to turn away 2 customers but presence of profanity will tilt 1 additional conversion -- so overall conversion rate of non-profane form could be better (2 vs 1 on some margin). But what if the non-profane converts are ultimately dissatisfied with the product whereas the the profane convert is likely to be happy? Perhaps the long term relationship with the latter customer is worth more than the two unhappy customers, even though a naive A/B test would push you to limit profanity for the short term conversion benefit.

sbov 15y ago
This is something about A/B tests that always bothers me. They always seem to assume that if variation A is best for a proxy stat (signups, views, etc) it's also best for what I'm really interested in (longterm value). I still use them though - not sure there's much you can do about it anyways.
- jplewicke 15y ago
  On the other hand, if you can eventually measure long-term value and you can keep track of users over a long enough cycle, then you can eventually run an A/B test on long term value. This depends on accurately tracking users, having a quantifiable way to calculate long-term value, and eliminating other sources of error, but it should be absolutely doable. You can also measure how well your proxy stat does correspond to long-term value and adjust your expectations accordingly.