3 ms·
Stats class role of thumb: if you need to calculate the relative probability of two outcomes, you can get to within about 10% once you get 100 samples of each o
by yafbum 3y ago
Stats class role of thumb: if you need to calculate the relative probability of two outcomes, you can get to within about 10% once you get 100 samples of each outcome (so, need more samples overall if the distribution is skewed).
- gear54rus 3y agoIts interesting that even in this thread the 2 answers differ by an order of magnitude lol
- gipp 3y agoEh it's really the same rule, just applying a different threshold.
- marcosdumay 3y agoThe problem is that the sensitivity to the number growth is supposed to be exponential. So if you need 100 samples for "within 10% of the value", then 10 samples should give you almost completely random behavior. In reality, it depends on your actual distribution, but the OP from this thread here is unreasonably conservative for something described as a "rule of thumb". Almost always, if you have at least 10 of every category, you can already discover every interesting thing that a rule of thumb will allow. And you probably could go with less. But if you want precision, you can't get it with rules of thumb.
- CaptainNegative 3y agoThe dependence on sample size is not exponential, it's sublinear. The heuristic rate of convergence to keep in mind is the square root of the sample size, i.e. getting 10x more samples shrinks the margin of error (in a multiplicative sense) by sqrt(10) ≈ 3ish. The exponential bit applies to the probability densities as a function of the bounds themselves, i.e. how likely you are to fall x units away from the mean typically decreases exponentially with (some polynomial in) x. Of course, this is all assuming a whole bunch of standard conditions on the data you're looking at (independence, identically distributed, bounded variance, etc.) and may not hold if these are violated.
- deleted 3y ago[deleted]
- jcranmer 3y agoFWIW, the threshold I learned was 20 in each bucket, so now you have 3 answers.
- koolba 3y agoJust apply it recursively. Let’s get 100 samples of comments suggesting the number of samples to use. Then average those.
- Keirmot 3y agoI think both answers are referencing the Central Limit Theorem, that states [simplified] that once you get over 30 samples for each independent variable, you will get a normal distribution.