5 ms·

I'm Split Testing ... Why Haven't I Doubled My Revenue Yet?

48 points by dcancel 16y ago

zemaj 16y ago
If this holds for more situations, I guess the conclusion I would draw is that spilt testing should be done as far down the funnel as possible to generate the most return.
- StavrosK 16y ago
  If you're losing 90% of the people on the first step and 50% of that 10% on the second step, is the second step really what you need to be optimising?
  - markitechtMA 16y ago
    Possibly, yes. The 'start at the cart' theory of optimizations says that you should absolutely start with optimizing that 50% and working 'backwards.' The idea being that it is a lot easier to close sales that are in process than to generate more sales leads that may or may not be qualified enough to actually initiate a purchase. Focus on reeling in the fish on the hook, to put it grossly, as opposed to putting more hooks in the water. Pretty easy to see the other side of this though, and like the correct answer to any poker question, the answer ends up being "it depends". =)
JangoSteve 16y ago
Ah, it's a rhetorical question. I was ready to read a rant; well played, sir. If you're short on time, check out the last graph, it's gold.
po 16y ago
Another tactic besides trying to move out the curve is: screw the skeptical aholes and create another product that the early-adopters will go for. If you have people tearing holes in their pants trying to get their wallet out faster, capitalize on it. Focus on that very small wedge on the left. Let's call this "the Apple strategy".
btilly 16y ago
Here is an alternate theory. Stick the numbers post-conversion in to http://elem.com/~btilly/effective-ab-testing/g-test-calculator.html http://elem.com/~btilly/effective-ab-testing/g-test-calculat... (41 successes out of 638 trials versus 35 successes from 416 trials) and the conclusion of unequal performance has 72.42% confidence. Meaning that more than 1 time in 4 you'd have a difference that big or bigger by chance. In other words the entire basis of this post could be a chance statistical fluctuation that should be ignored. It is true that there can be effects where pushing less qualified leads through the top stage of the funnel doesn't get them to the end. However my experience with A/B testing is that it is more common for the extra people put in the system by an A/B test at the top to convert the rest of the way relatively similarly. But not always! Which is why if you have sufficient volume you should always measure to actual sales. There is no other way to be absolutely sure that you are improving end sales. However in this example that would mean running the test for something like 20x as long. In that case it makes sense to be pragmatic, test from one step of the funnel to the next, and then pivot on the answers you get. Furthermore to start you should focus on the top of the funnel for the simple reason that higher volumes will get you answers faster there - you can easily try a dozen ideas before you could test one idea deeper in the funnel. Once you've improved your site enough to get a better percentage of actual sales, you'll be able to purchase more traffic. Doing both of those things will put you in a position to conduct more rigorous A/B tests to eke out more subtle differences. But that is down the road. Focus on testing what is easiest in the quickest possible way first.
- JangoSteve 16y ago
  In other words the entire basis of this post could be a chance statistical fluctuation that should be ignored. I agree that the particular stats referenced in the article may not be statistically valid, but I wouldn't argue that those stats were a supporting detail rather than the entire basis. The main point as I understood it was to illustrate, more or less, why a 100% increase in conversions to the purchasing page does not equal a 100% increase in conversions from the purchasing page to actual purchase. They are saying that once you start attracting traffic beyond the early adopters, your additional traffic is now comprised of a different group of people who exhibit fundamentally different behavior in how likely they are to make a purchase even once they've hit the purchasing page.

48 points by dcancel 16y ago

zemaj 16y ago

If this holds for more situations, I guess the conclusion I would draw is that spilt testing should be done as far down the funnel as possible to generate the most return.

StavrosK 16y ago
If you're losing 90% of the people on the first step and 50% of that 10% on the second step, is the second step really what you need to be optimising?
- markitechtMA 16y ago
  Possibly, yes. The 'start at the cart' theory of optimizations says that you should absolutely start with optimizing that 50% and working 'backwards.' The idea being that it is a lot easier to close sales that are in process than to generate more sales leads that may or may not be qualified enough to actually initiate a purchase. Focus on reeling in the fish on the hook, to put it grossly, as opposed to putting more hooks in the water. Pretty easy to see the other side of this though, and like the correct answer to any poker question, the answer ends up being "it depends". =)

JangoSteve 16y ago

Ah, it's a rhetorical question. I was ready to read a rant; well played, sir. If you're short on time, check out the last graph, it's gold.

po 16y ago

Another tactic besides trying to move out the curve is: screw the skeptical aholes and create another product that the early-adopters will go for. If you have people tearing holes in their pants trying to get their wallet out faster, capitalize on it. Focus on that very small wedge on the left. Let's call this "the Apple strategy".

btilly 16y ago

Here is an alternate theory. Stick the numbers post-conversion in to http://elem.com/~btilly/effective-ab-testing/g-test-calculator.html http://elem.com/~btilly/effective-ab-testing/g-test-calculat... (41 successes out of 638 trials versus 35 successes from 416 trials) and the conclusion of unequal performance has 72.42% confidence. Meaning that more than 1 time in 4 you'd have a difference that big or bigger by chance. In other words the entire basis of this post could be a chance statistical fluctuation that should be ignored. It is true that there can be effects where pushing less qualified leads through the top stage of the funnel doesn't get them to the end. However my experience with A/B testing is that it is more common for the extra people put in the system by an A/B test at the top to convert the rest of the way relatively similarly. But not always! Which is why if you have sufficient volume you should always measure to actual sales. There is no other way to be absolutely sure that you are improving end sales. However in this example that would mean running the test for something like 20x as long. In that case it makes sense to be pragmatic, test from one step of the funnel to the next, and then pivot on the answers you get. Furthermore to start you should focus on the top of the funnel for the simple reason that higher volumes will get you answers faster there - you can easily try a dozen ideas before you could test one idea deeper in the funnel. Once you've improved your site enough to get a better percentage of actual sales, you'll be able to purchase more traffic. Doing both of those things will put you in a position to conduct more rigorous A/B tests to eke out more subtle differences. But that is down the road. Focus on testing what is easiest in the quickest possible way first.

JangoSteve 16y ago
In other words the entire basis of this post could be a chance statistical fluctuation that should be ignored. I agree that the particular stats referenced in the article may not be statistically valid, but I wouldn't argue that those stats were a supporting detail rather than the entire basis. The main point as I understood it was to illustrate, more or less, why a 100% increase in conversions to the purchasing page does not equal a 100% increase in conversions from the purchasing page to actual purchase. They are saying that once you start attracting traffic beyond the early adopters, your additional traffic is now comprised of a different group of people who exhibit fundamentally different behavior in how likely they are to make a purchase even once they've hit the purchasing page.