We had thousands of visitors and hundreds of conversions (I don't know the exact numbers off the top of my head). We let it run long enough that the confidence

We had thousands of visitors and hundreds of conversions (I don't know the exact numbers off the top of my head). We let it run long enough that the confidence intervals for the results fell below .1%, for this particular test, we would have let it run longer, but the results let us know we didn't actually need to for the email test. Since it was likely to be worse, we just shut it down. We are (unfortunately) well aware of how statistically significant things look with too little data and on the flip side, how to optimize for the wrong conversion goal (see my other post: http://georgesaines.com/2010/09/28/how-we-increased-conversion-actions-218-and-increased-sales-0/ http://georgesaines.com/2010/09/28/how-we-increased-conversi...) We use Visual Website Optimizer so we have to set a test, agree on a number of conversions we have to see before making our first judgment, and the adjust our expectations based on the number of times we peak.