5 ms·

"If you work at a software company. . . you should be doing A/B testing for the same reason you have an accountant do your taxes. It is a low-risk low-investmen

by aresant 10y ago

"If you work at a software company. . . you should be doing A/B testing for the same reason you have an accountant do your taxes. It is a low-risk low-investment high-certainty way to increase economic returns." Spot on, and you should 100% support Patrick & Nick and buy their books / videos / etc. But A/B testing is not as simple as hiring an accountant, running A/B testing as part of your corporate culture requires: - Front end developer to install / manage testing engine. - UX designer to measure / understand where holes may be in your flow. - Competent web designer to execute instructions from UX designer. - Statistically sound processes to know when to "call" a test (no Optimizely does NOT do this) - SEO / marketing people to make sure that your A/B test didn't just break marketing flow - etc Yes, anybody could go run a headline test. But running A/B testing with any sort of regularity, scale, and success is a complicated problem.

btilly 10y ago
Meh, you make it sound too hard. Here is a perfectly adequate A/B testing regime. Assign visitors into buckets randomly. Track conversion counts only. Declare the winner of an A/B test whenever one gets 100 conversions ahead, or whichever is ahead after 10,000 conversions. Do this with every test. As long as there are no obvious interaction effects, you can run multiple tests in parallel. If you wish, you can replace 10,000 with N where N is however many conversions you expect to get in a month. Replace 100 by the square root of N. Real differences below 2/sqrt(N) are basically coin flips. Even after many tests, you are very unlikely to have ever made an error as big as 5/sqrt(N). If you want to do something more sophisticated than this, you need competent people. But this is worlds better than what most companies do. (And yes, it is better than what Optimizely does for you by default.)
- ken47 10y ago
  Can you explain the statistics behind this methodology?
  - btilly 10y ago
    Essentially the same as http://www.evanmiller.org/sequential-ab-testing.html http://www.evanmiller.org/sequential-ab-testing.html except that he aims to not produce an answer most of the time by chance, while I aim to always produce an answer even if it is statistically a coin flip. (Because, after all, you need to actually choose a headline in the end...)
- robbiemitchell 10y ago
  You're describing the math of A/B testing, but not the work of analyzing data to decide what experiment to do next, designing the experiment, building and integrating it, then assigning visitors to the buckets and so forth.
  - btilly 10y ago
    The technical end of building / integrating / assigning visitors and so on tends to be relatively simple if it is done by competent people who know how to not overcomplicate things. Coming up with ideas to test tends to be fairly easy if you've got a competent product person. Perhaps I'm biased. I've built multiple A/B testing systems for multiple companies, and know what to do. However it didn't seem hard for me the first time either.
- in_cahoots 10y ago
  I appreciate what you're trying to say here, but you've basically just described the prototypical way not to run an A/B test. Ending a test as soon as you reach significance ensures a high rate of false positives, and doesn't clearly tell you when to end the experiment if there is no effect. See also: http://www.evanmiller.org/how-not-to-run-an-ab-test.html http://www.evanmiller.org/how-not-to-run-an-ab-test.html
  - btilly 10y ago
    You have just pattern-matched to a response without understanding. And you're wrong. See http://www.evanmiller.org/sequential-ab-testing.html http://www.evanmiller.org/sequential-ab-testing.html for a very similar methodology presented by the very person you are citing. The difference is that I am aiming to always produce an answer from the test, even if it is a statistical coin flip, while he aims to only produce an answer roughly 5% of the time if there is no difference.
    - in_cahoots 10y ago
      Maybe I am not understanding your post, but aren't you just declaring a winner after N trials, even if that is not significant? That seems to be a critical distinction here.
      - btilly 10y ago
        Right. We are trying to make a business decision, not conduct science. If you go with whichever is ahead, you'll reliably produce correct answers for wins at that you can't reliably measure. Yes, there will be mistakes, but the mistakes are guaranteed to be pretty small. If you insist on a higher standard, you'll learn which choices you were confident of..but now need to figure out what to do when the statistics didn't give you a final answer. I think that the first choice is more useful. Evan prefers to clearly distinguish your coin flips from solid results. But in the end you have to make a decision, and it isn't material what decision you make if the conversion rates are close.
        10y ago
- deleted 10y ago
  [deleted]
- dhbradshaw 10y ago
  This seems like a nice application of two related principles: 1) The harder it is to tell which choice is better, the less it matters which you choose, and 2) a decision has value even in the absence of proof that it's the very best decision because it frees up resources for the next decision, which may be more important.
- cle 10y ago
  Depending on the implementation of "assign visitors into buckets randomly", this should include a test to ensure the populations are correctly randomized. "Unsophisticated" users of these heuristics are prone to make terrible decisions if the treatment populations are not properly randomized (this can be due to systemic bias caused by e.g. caching, or treatment effects like survival bias).
netcan 10y ago
Hiring an accountant is not really easy either, though it's made easier by the fact that accountants are fairly well established. If accounting had been invented recently, it would be hard to find someone who knows what to do and it would be hard to get the rest of the company to operate in a way that allows him to do it. The more companies do this, the easier it well get. The fewer people do this, the more of a competitive edge it is.
ssharp 10y ago
Even if you think Optimizely does a bad job of calculating test results, I'm assuming (I'm a VWO loyalist) it still gives you the raw numbers and it still provides a lot of other value. Just handling the basic logistics and measurement of the test is well worth the price. You're mostly right in what types of resources you need, however, I'd add that the "competent web designer" should also be competent in jQuery. For a lot of tests, running jQuery DOM manipulations is required/preferable over the visual designer. Those visual designers tend to fail when dealing with dynamic content and have the potential to halt your entire operation. Of course, you may also have teammates (or you yourself) who can do more than one of these things.
ryandrake 10y ago
Totally agree, a few bullet points you almost touched upon, too: 1. Management buy-in to always be testing and always be investing in testing - It's one thing to have the boss wake up tomorrow and decide "We should do some A/B testing", have the team go off and implement a few tests, then when that's done, move onto the next feature-of-the-month. To really make it pay off, A/B testing needs to be done all the time, to the point where it drives priorities and it makes no sense to commit to the next thing until the tests confirm what that thing should be. Which brings me to: 2. A product design culture that integrates the results of A/B testing full circle back into product decisions - Why even do these tests if results don't feed back into the product? If testing says header bar A's choices perform/monetize better than header bar B, but the designer sticks with header bar B simply because of his artistic "gut sense", then why are you even investing in testing?
- wastedhours 10y ago
  Another big aspect of the management buy-in pieces is making sure they're happy with the underlying outcome of testing: there'll be a loser. As in, it's quite difficult to get them comfortable with the notion that they've missed out on revenue/users as a result of a test, even if the winner will provide outsized gains.
- foobarian 10y ago
  I would add 1a. Management buy-in to devote resources to clean up complete/unsuccessful tests. I witnessed first hand the ball of mud resulting in 10 years of test upon test upon test. It's a huge impediment to quick iteration; simple changes are hard and hard changes are impossible. It's especially bad because test code tends to be poorly designed and documented due to the get-it-out-the-door mentality.

by aresant 10y ago

btilly 10y ago

Meh, you make it sound too hard. Here is a perfectly adequate A/B testing regime. Assign visitors into buckets randomly. Track conversion counts only. Declare the winner of an A/B test whenever one gets 100 conversions ahead, or whichever is ahead after 10,000 conversions. Do this with every test. As long as there are no obvious interaction effects, you can run multiple tests in parallel. If you wish, you can replace 10,000 with N where N is however many conversions you expect to get in a month. Replace 100 by the square root of N. Real differences below 2/sqrt(N) are basically coin flips. Even after many tests, you are very unlikely to have ever made an error as big as 5/sqrt(N). If you want to do something more sophisticated than this, you need competent people. But this is worlds better than what most companies do. (And yes, it is better than what Optimizely does for you by default.)

ken47 10y ago
Can you explain the statistics behind this methodology?
- btilly 10y ago
  Essentially the same as http://www.evanmiller.org/sequential-ab-testing.html http://www.evanmiller.org/sequential-ab-testing.html except that he aims to not produce an answer most of the time by chance, while I aim to always produce an answer even if it is statistically a coin flip. (Because, after all, you need to actually choose a headline in the end...)
robbiemitchell 10y ago
You're describing the math of A/B testing, but not the work of analyzing data to decide what experiment to do next, designing the experiment, building and integrating it, then assigning visitors to the buckets and so forth.
- btilly 10y ago
  The technical end of building / integrating / assigning visitors and so on tends to be relatively simple if it is done by competent people who know how to not overcomplicate things. Coming up with ideas to test tends to be fairly easy if you've got a competent product person. Perhaps I'm biased. I've built multiple A/B testing systems for multiple companies, and know what to do. However it didn't seem hard for me the first time either.
in_cahoots 10y ago
I appreciate what you're trying to say here, but you've basically just described the prototypical way not to run an A/B test. Ending a test as soon as you reach significance ensures a high rate of false positives, and doesn't clearly tell you when to end the experiment if there is no effect. See also: http://www.evanmiller.org/how-not-to-run-an-ab-test.html http://www.evanmiller.org/how-not-to-run-an-ab-test.html
- btilly 10y ago
  You have just pattern-matched to a response without understanding. And you're wrong. See http://www.evanmiller.org/sequential-ab-testing.html http://www.evanmiller.org/sequential-ab-testing.html for a very similar methodology presented by the very person you are citing. The difference is that I am aiming to always produce an answer from the test, even if it is a statistical coin flip, while he aims to only produce an answer roughly 5% of the time if there is no difference.
  - in_cahoots 10y ago
    Maybe I am not understanding your post, but aren't you just declaring a winner after N trials, even if that is not significant? That seems to be a critical distinction here.
    - btilly 10y ago
      Right. We are trying to make a business decision, not conduct science. If you go with whichever is ahead, you'll reliably produce correct answers for wins at that you can't reliably measure. Yes, there will be mistakes, but the mistakes are guaranteed to be pretty small. If you insist on a higher standard, you'll learn which choices you were confident of..but now need to figure out what to do when the statistics didn't give you a final answer. I think that the first choice is more useful. Evan prefers to clearly distinguish your coin flips from solid results. But in the end you have to make a decision, and it isn't material what decision you make if the conversion rates are close.
      - 10y ago
deleted 10y ago
[deleted]
dhbradshaw 10y ago
This seems like a nice application of two related principles: 1) The harder it is to tell which choice is better, the less it matters which you choose, and 2) a decision has value even in the absence of proof that it's the very best decision because it frees up resources for the next decision, which may be more important.
cle 10y ago
Depending on the implementation of "assign visitors into buckets randomly", this should include a test to ensure the populations are correctly randomized. "Unsophisticated" users of these heuristics are prone to make terrible decisions if the treatment populations are not properly randomized (this can be due to systemic bias caused by e.g. caching, or treatment effects like survival bias).

netcan 10y ago

Hiring an accountant is not really easy either, though it's made easier by the fact that accountants are fairly well established. If accounting had been invented recently, it would be hard to find someone who knows what to do and it would be hard to get the rest of the company to operate in a way that allows him to do it. The more companies do this, the easier it well get. The fewer people do this, the more of a competitive edge it is.

ssharp 10y ago

Even if you think Optimizely does a bad job of calculating test results, I'm assuming (I'm a VWO loyalist) it still gives you the raw numbers and it still provides a lot of other value. Just handling the basic logistics and measurement of the test is well worth the price. You're mostly right in what types of resources you need, however, I'd add that the "competent web designer" should also be competent in jQuery. For a lot of tests, running jQuery DOM manipulations is required/preferable over the visual designer. Those visual designers tend to fail when dealing with dynamic content and have the potential to halt your entire operation. Of course, you may also have teammates (or you yourself) who can do more than one of these things.

ryandrake 10y ago

Totally agree, a few bullet points you almost touched upon, too: 1. Management buy-in to always be testing and always be investing in testing - It's one thing to have the boss wake up tomorrow and decide "We should do some A/B testing", have the team go off and implement a few tests, then when that's done, move onto the next feature-of-the-month. To really make it pay off, A/B testing needs to be done all the time, to the point where it drives priorities and it makes no sense to commit to the next thing until the tests confirm what that thing should be. Which brings me to: 2. A product design culture that integrates the results of A/B testing full circle back into product decisions - Why even do these tests if results don't feed back into the product? If testing says header bar A's choices perform/monetize better than header bar B, but the designer sticks with header bar B simply because of his artistic "gut sense", then why are you even investing in testing?

wastedhours 10y ago
Another big aspect of the management buy-in pieces is making sure they're happy with the underlying outcome of testing: there'll be a loser. As in, it's quite difficult to get them comfortable with the notion that they've missed out on revenue/users as a result of a test, even if the winner will provide outsized gains.
foobarian 10y ago
I would add 1a. Management buy-in to devote resources to clean up complete/unsuccessful tests. I witnessed first hand the ball of mud resulting in 10 years of test upon test upon test. It's a huge impediment to quick iteration; simple changes are hard and hard changes are impossible. It's especially bad because test code tends to be poorly designed and documented due to the get-it-out-the-door mentality.