3 ms·

Great point. Jumping to pre-mature conclusions is certainly an easy mistake to make. During the Obama campaign we were frustrated by having to wait 24 - 48 hou

Great point. Jumping to pre-mature conclusions is certainly an easy mistake to make. During the Obama campaign we were frustrated by having to wait 24 - 48 hours to get results for our experiments. We were lucky to be getting a ton of traffic and hence be able to get statistically significant results in just a few days. Another big benefit of real-time results is helping our customers answer the question "is this working correctly?" Seeing the data come in in real-time helps reassure folks that the tool is working properly and collecting data.

revorad 16y ago
During the Obama campaign Whoaa why isn't there a crazy big testimonial screaming that on the homepage? [EDIT: Sorry I misunderstood as you having used optimizely for the campaign, but it sounds like you probably thought of the idea as a result of your frustration.] I can understand customers getting frustrated for having to wait. But, that's probably what will make them jump to conclusions. I don't know how much you want to get into advocating "best practices", but it would probably be good to ask the user to set their minimum number of visitors beforehand. The "is this working correctly?" question sounds like a very good practical issue you picked up during your beta testing.