The thing about an A/B test is that you don't have to put up with it because you don't know you're in it. So I don't think there's much friction in executing on

The thing about an A/B test is that you don't have to put up with it because you don't know you're in it. So I don't think there's much friction in executing one as long as you're bucketing the population in a sane way (e.g., don't show two different things to the same person at two different times). I hear you on the misuse of statistics, but unless you're running multivariate experiments, this isn't much of an issue. Yes, if you can't even describe an experiment in a table/matrix, then it's probably too complicated and the results difficult to parse. I think most A/B tests don’t fall into this category and are far simpler. Let me try to offer a practical A/B test for this checkbox. Hypothesis: We believe that making the checkbox animated and reactive will result in people creating more to-do list items right after using the checkbox because they love it so much. Test: To verify that we will deploy the new checkbox to half the users, and half the users get the normal one. Run this for a duration of two weeks. Metric: We will measure the amount of to-do list items created right after someone completes a to-do list item. Criteria: We are right if people with the new checkbox create 1.2x more to-do list items than the ones with the old one. -- The above is not a difficult experiment to create or analyze.