Thanks, I'm glad you enjoyed the book! (Author here -- the website got its first publicity here on HN.) Regarding AB testing, you might be interested in this r

Thanks, I'm glad you enjoyed the book! (Author here -- the website got its first publicity here on HN.) Regarding AB testing, you might be interested in this recent research, which uses real data from Optimizely to estimate how often people get AB test false positives because they stopped as soon as they hit significance: https://ssrn.com/abstract=3204791 https://ssrn.com/abstract=3204791 > Specifically, about 73% of experimenters stop the experiment just when a positive effect reaches 90% confidence. Also, approximately 75% of the effects are truly null. Improper optional stopping increases the false discovery rate (FDR) from 33% to 40% among experiments p-hacked at 90% confidence