3 ms·

How much difference did no testimonial vs male testimonial vs female testimonial make?

• patio11 16y ago
  A statistically significant difference and then a statistically significant difference. A/B tests don't guarantee that the observed magnitude of the difference is meaningful. (To satisfy your curiosity, it was Glenda 25.12% vs Dave 23.38%, significant at 90% confidence, and I don't have numbers handy for the testimonial vs. no testimonial test. That was prior to me tracking the results in a systematic fashion. It doesn't look like it is written anywhere on my blog, either.)
  • jules 16y ago
    There seems to be a tension between running the test longer and collecting more profits. If you run the test longer you are less likely to pick the bad option, at the expense of trying the bad option for a longer period of time. Have you thought about a mathematical solution to this problem? Perhaps there is a good way to decide when to stop automatically.
  • davidblair 16y ago
    What do you use as your p-value cutoff? I've heard many arguments for .05 being the highest acceptable for research but it seems like you have found an acceptable p-value that is out of that range by quite a bit.
    • bryanh 16y ago
      I figure that if the test is at 90% confidence, it usually means we can't really make a really really bad decision going with the favorite. So what if that tiny increase in conversions wasn't really there, I figure its time to move on to another, possibly more lucrative test. Maybe Patrick has a different or more fleshed out opinion on this.