The short and the simple (if somewhat controversial) answer is because the real world is complex and probability is a measure of subjective belief. So without

The short and the simple (if somewhat controversial) answer is because the real world is complex and probability is a measure of subjective belief. So without getting TOO philosophical, for cases which don't fall into relatively simple text-book examples (differences between coin tosses in two groups, random draws of cards, radioactive decay, well-defined distributions, that sort of thing...for which generally agreed upon settled best practice analytical solutions probably do exist), or which don't involve making some rather outrageous assumptions (lets assume away our non-survey error), you have in practice disagreements about what you're even experimenting over, what is held constant, what even are the significance of the results, what are the relevant distributions, what is the non-survey error, how to update your beliefs based upon the results, what are reasonable prior beliefs beforehand, real world restrictions on how you can design your experiment, etc. It would be my contention that there's not a settled best practice because its inherently NOT settle-able. There is no objective answer to the reference class problem. There are multiple consistent hypothesis that can equally fit/explain data. Two reasonable rational minds can disagree over the implications of results. Options have trade-offs and there are differing levels of uncertainty over all things. Of course, if I'm at work and I'm asked to design a sample, collection, or experiment, that's not what I say. I generally start at analytical solutions, and gradually step down into simulations and monte-carlo methods depending on how complex it gets. And i'll ask them what question they hope to answer, and what are the consequences of being right/wrong, and how much resources they have to expend in finding the answer and what are the restrictions on what they can do and what do they have to work with. And people who ask me to design samples don't talk in languages of prior or subjective beliefs or refer to reference class problems or relevant distributions for a significance test or updating beliefs etc. Even though that's what I often know they're having problems with. But this is hacker news, so maybe on here I can say: because the real world is really complex and a lot of it is inherently subjective. (also, in the real world, a lot of people who ask for the a sample design REALLY don't like the amount of resources required for their 'officially' stated desired outcomes, so like a lot of real world practice and service delivery, best practice often becomes how to optimally deal with champagne tastes on a beer budget serving princes who ask for contradictory goals).