3 ms·

Great job on the product and video - really easy and slick. What's the use of real-time results for A/B testing? Most likely, it's only going to lead to errone

by revorad 16y ago

Great job on the product and video - really easy and slick. What's the use of real-time results for A/B testing? Most likely, it's only going to lead to erroneous conclusions, as discussed in this thread - http://news.ycombinator.com/item?id=1778104 http://news.ycombinator.com/item?id=1778104.

dsiroker 16y ago
Great point. Jumping to pre-mature conclusions is certainly an easy mistake to make. During the Obama campaign we were frustrated by having to wait 24 - 48 hours to get results for our experiments. We were lucky to be getting a ton of traffic and hence be able to get statistically significant results in just a few days. Another big benefit of real-time results is helping our customers answer the question "is this working correctly?" Seeing the data come in in real-time helps reassure folks that the tool is working properly and collecting data.
- revorad 16y ago
  During the Obama campaign Whoaa why isn't there a crazy big testimonial screaming that on the homepage? [EDIT: Sorry I misunderstood as you having used optimizely for the campaign, but it sounds like you probably thought of the idea as a result of your frustration.] I can understand customers getting frustrated for having to wait. But, that's probably what will make them jump to conclusions. I don't know how much you want to get into advocating "best practices", but it would probably be good to ask the user to set their minimum number of visitors beforehand. The "is this working correctly?" question sounds like a very good practical issue you picked up during your beta testing.
btilly 16y ago
That thread is seriously wrong. Here is an excerpt from an email discussion I had recently that touched on http://www.evanmiller.org/how-not-to-run-an-ab-test.html http://www.evanmiller.org/how-not-to-run-an-ab-test.html. Evan Miller has a point, but not as good of one as he thinks. It is true that multiple peeks mean that eventually any test will find significance at any level you want. However in A/B tests the peeks are not independent. This greatly weakens the effect he is talking about. Section 7 of my presentation, starting at http://elem.com/~btilly/effective-ab-testing/#slide59 http://elem.com/~btilly/effective-ab-testing/#slide59, is about the question of how long it takes for a test to complete. For that I ran numerical experiments with constant peeking, literally every time you add one to A and one to B you peek again. You can see graphs of how many errors there were, and how long it takes to get an answer. Here are key points: - Be suspicious of tests that end quickly. Run them a bit longer on general principal. (In general I'd call 500 people a very small test.) - Nobody can predict how long a test will take. Even if you know the actual improvement, you still can't predict time to within an order of magnitude. - If a test has been running for a long time you know the true difference is small, so there is no harm in accepting whatever answer it gives.

by revorad 16y ago

dsiroker 16y ago

Great point. Jumping to pre-mature conclusions is certainly an easy mistake to make. During the Obama campaign we were frustrated by having to wait 24 - 48 hours to get results for our experiments. We were lucky to be getting a ton of traffic and hence be able to get statistically significant results in just a few days. Another big benefit of real-time results is helping our customers answer the question "is this working correctly?" Seeing the data come in in real-time helps reassure folks that the tool is working properly and collecting data.

revorad 16y ago
During the Obama campaign Whoaa why isn't there a crazy big testimonial screaming that on the homepage? [EDIT: Sorry I misunderstood as you having used optimizely for the campaign, but it sounds like you probably thought of the idea as a result of your frustration.] I can understand customers getting frustrated for having to wait. But, that's probably what will make them jump to conclusions. I don't know how much you want to get into advocating "best practices", but it would probably be good to ask the user to set their minimum number of visitors beforehand. The "is this working correctly?" question sounds like a very good practical issue you picked up during your beta testing.

btilly 16y ago

That thread is seriously wrong. Here is an excerpt from an email discussion I had recently that touched on http://www.evanmiller.org/how-not-to-run-an-ab-test.html http://www.evanmiller.org/how-not-to-run-an-ab-test.html. Evan Miller has a point, but not as good of one as he thinks. It is true that multiple peeks mean that eventually any test will find significance at any level you want. However in A/B tests the peeks are not independent. This greatly weakens the effect he is talking about. Section 7 of my presentation, starting at http://elem.com/~btilly/effective-ab-testing/#slide59 http://elem.com/~btilly/effective-ab-testing/#slide59, is about the question of how long it takes for a test to complete. For that I ran numerical experiments with constant peeking, literally every time you add one to A and one to B you peek again. You can see graphs of how many errors there were, and how long it takes to get an answer. Here are key points: - Be suspicious of tests that end quickly. Run them a bit longer on general principal. (In general I'd call 500 people a very small test.) - Nobody can predict how long a test will take. Even if you know the actual improvement, you still can't predict time to within an order of magnitude. - If a test has been running for a long time you know the true difference is small, so there is no harm in accepting whatever answer it gives.