4 ms·

Well done; A/B testing is valuable, but under a very specific set of circumstances. The true purpose of AB testing is to say, "Does my conversion go up or down

Well done; A/B testing is valuable, but under a very specific set of circumstances. The true purpose of AB testing is to say, "Does my conversion go up or down if I make that button green?" If it goes up you keep it. You put in some variables, and A/B testing can tell you which varables will produce the best result. But it's important not to forget that the results of these tests were (intentionally) limited to certain variables. The reason being a founder is so hard and the reason your intuition as a founder is so valuable is because answers that require finding and solving pains can't (yet) be mathematically found. To us a very tired analogy, A/B testing will tell you people want a faster horse instead of a car.

yummyfajitas 14y ago
But it's important not to forget that the results of these tests were (intentionally) limited to certain variables. I recently made the argument that you should cook up a single metric which captures all variables of interest, and use it everywhere. Put all variables of interest into them, and carefully define what tradeoffs you are willing to make. http://www.chrisstucchio.com/blog/2013/metrics_manifesto.html http://www.chrisstucchio.com/blog/2013/metrics_manifesto.htm... As for the tired analogy, founders intuition gives you the idea of cars. A/B testing can still demonstrate that cars are superior.
- darkxanthos 14y ago
  Since you're proposing a calculated metric, it's advisable to just log all three and do the calculation afterwards. That way if the business decides that number is silly at some point it can be changed and your results recalculated. Also trying to come up with a simple formula that relates different units of business value can be VERY difficult. Any literature on perspectives around that?
  - yummyfajitas 14y ago
    Also trying to come up with a simple formula that relates different units of business value can be VERY difficult. Deciding what you actually want and what tradeoffs you are willing to make can be hard. Then again, if you don't know what you want you are unlikely to achieve it.

yummyfajitas 14y ago

But it's important not to forget that the results of these tests were (intentionally) limited to certain variables. I recently made the argument that you should cook up a single metric which captures all variables of interest, and use it everywhere. Put all variables of interest into them, and carefully define what tradeoffs you are willing to make. http://www.chrisstucchio.com/blog/2013/metrics_manifesto.html http://www.chrisstucchio.com/blog/2013/metrics_manifesto.htm... As for the tired analogy, founders intuition gives you the idea of cars. A/B testing can still demonstrate that cars are superior.

darkxanthos 14y ago
Since you're proposing a calculated metric, it's advisable to just log all three and do the calculation afterwards. That way if the business decides that number is silly at some point it can be changed and your results recalculated. Also trying to come up with a simple formula that relates different units of business value can be VERY difficult. Any literature on perspectives around that?
- yummyfajitas 14y ago
  Also trying to come up with a simple formula that relates different units of business value can be VERY difficult. Deciding what you actually want and what tradeoffs you are willing to make can be hard. Then again, if you don't know what you want you are unlikely to achieve it.