3 ms·

>On a sample of thousands, a 3% difference is big. Not really. Only if it is many, many thousands. Assuming a totally random acceptance rate of 1/5: a = 0;

>On a sample of thousands, a 3% difference is big. Not really. Only if it is many, many thousands. Assuming a totally random acceptance rate of 1/5: a = 0; b = 0; for (c of Array(1000)) { if (Math.random() > .8) a++; if (Math.random() > .8) b++; } console.log(`a=${a}, b=${b}, a is ${(a/b - 1)*100}% more likely than b`) > a=209, b=201, a is 3.9800995024875663% more likely than b literally the first run. And even in absolute terms, I got this on the third run: >a=192, b=219, a is -12.328767123287676% more likely than b That's an absolute difference of 2.7%. Again, 100% random data.

timr 4y ago
> That's an absolute difference of 2.7%. Again, 100% random data. I think I get what you're going for here -- you're trying to simulate a coin flip? -- but what you've actually done is made successive draws from a uniform random number generator. The software is designed to return numbers that fall along the interval [0,1) with equal probability. Thresholding the numbers and dividing their counts is not a meaningful transformation; the result is still just a uniformly distributed random number. It's like...the ratio of heads in two identical, unfair coins or something. If all "random numbers" were uniform like this, then no, we wouldn't expect an X% difference to be any more or less likely based on the magnitude of the underlying sample. But when we're talking about something like a a population mean, then the behavior of the errors on estimates is very different indeed, and most estimates cluster around the true (aka population) value: https://online.stat.psu.edu/stat415/lesson/9/9.4 https://online.stat.psu.edu/stat415/lesson/9/9.4 As the sample size for an experiment of this sort gets larger, the bell curve of expected errors gets sharper and sharper, and it becomes increasingly less likely to see errors >= X, for any value X. In the limit of large N, the distribution of sample errors around a known mean approach a normal distribution: https://www.jmp.com/en_us/statistics-knowledge-portal/t-test/t-distribution.html https://www.jmp.com/en_us/statistics-knowledge-portal/t-test... For what it's worth, the expected proportion of N heads in M coin flips is modeled using the binomial distribution, which is also bell-shaped and illustrates the same idea: https://en.wikipedia.org/wiki/Binomial_distribution https://en.wikipedia.org/wiki/Binomial_distribution
- LordDragonfang 4y ago
  That's precisely what this is trying to model, yes. The standard computational way to simulate a binary event with probability p is to call rand() and check if rand(0,1) < p (or > 1-p, what I did). Or as you called it, an unfair coin flip. This model is built on the assumptions that if candidates are actually totally equally likely to be picked (the null hypothesis for the experiment above), any given candidate has a p=.2 chance of being hired (given an arbitrary but reasonable hire vs interview ratio of 1:5). Which is just a weighted coin flip. This is indeed a binomial distribution, and my point is that results ±3% of the mean (p*M), even at M=1000, are still fairly probable. When comparing two such results, it's almost expected.
  - timr 4y ago
    The part where you did rand() < 0.8 ? 1 : 0 is fine. That's a Bernoulli trial with p=0.8 The part where you did this in a loop, with two calls per iteration, and then divided the counts and called it a percentage is wrong. It's certainly not a Binomial distribution. It's just the ratio of two binomial random variables.
- chronial 4y ago
  > I think I get what you're going for here -- you're trying to simulate a coin flip? -- but what you've actually done is made successive draws from a uniform random number generator. The software is designed to return numbers that fall along the interval [0,1) with equal probability. Thresholding the numbers and dividing their counts is not a meaningful transformation; This is wrong. That is a very meaningful transformation. It is the standard way (https://stats.stackexchange.com/questions/240338 https://stats.stackexchange.com/questions/240338) to turn a uniform distribution into a Bernoulli distribution. Getting a single value with Bernoulli distribution is called a Bernoulli trial (https://en.wikipedia.org/wiki/Bernoulli_trial https://en.wikipedia.org/wiki/Bernoulli_trial). Repeating this gives you a Binomial distribution (see your own wikipedia link). Long story short: GPs code is a perfectly valid way of sampling the Bernoulli distribution. It is inefficient because it needs so many random values, but it mimics the actual process happening in real life making it easier to understand than generating a Binomial sample from the Binomial distribution's CDF.
deleted 4y ago
[deleted]

timr 4y ago

> That's an absolute difference of 2.7%. Again, 100% random data. I think I get what you're going for here -- you're trying to simulate a coin flip? -- but what you've actually done is made successive draws from a uniform random number generator. The software is designed to return numbers that fall along the interval [0,1) with equal probability. Thresholding the numbers and dividing their counts is not a meaningful transformation; the result is still just a uniformly distributed random number. It's like...the ratio of heads in two identical, unfair coins or something. If all "random numbers" were uniform like this, then no, we wouldn't expect an X% difference to be any more or less likely based on the magnitude of the underlying sample. But when we're talking about something like a a population mean, then the behavior of the errors on estimates is very different indeed, and most estimates cluster around the true (aka population) value: https://online.stat.psu.edu/stat415/lesson/9/9.4 https://online.stat.psu.edu/stat415/lesson/9/9.4 As the sample size for an experiment of this sort gets larger, the bell curve of expected errors gets sharper and sharper, and it becomes increasingly less likely to see errors >= X, for any value X. In the limit of large N, the distribution of sample errors around a known mean approach a normal distribution: https://www.jmp.com/en_us/statistics-knowledge-portal/t-test/t-distribution.html https://www.jmp.com/en_us/statistics-knowledge-portal/t-test... For what it's worth, the expected proportion of N heads in M coin flips is modeled using the binomial distribution, which is also bell-shaped and illustrates the same idea: https://en.wikipedia.org/wiki/Binomial_distribution https://en.wikipedia.org/wiki/Binomial_distribution

LordDragonfang 4y ago
That's precisely what this is trying to model, yes. The standard computational way to simulate a binary event with probability p is to call rand() and check if rand(0,1) < p (or > 1-p, what I did). Or as you called it, an unfair coin flip. This model is built on the assumptions that if candidates are actually totally equally likely to be picked (the null hypothesis for the experiment above), any given candidate has a p=.2 chance of being hired (given an arbitrary but reasonable hire vs interview ratio of 1:5). Which is just a weighted coin flip. This is indeed a binomial distribution, and my point is that results ±3% of the mean (p*M), even at M=1000, are still fairly probable. When comparing two such results, it's almost expected.
- timr 4y ago
  The part where you did rand() < 0.8 ? 1 : 0 is fine. That's a Bernoulli trial with p=0.8 The part where you did this in a loop, with two calls per iteration, and then divided the counts and called it a percentage is wrong. It's certainly not a Binomial distribution. It's just the ratio of two binomial random variables.
chronial 4y ago
> I think I get what you're going for here -- you're trying to simulate a coin flip? -- but what you've actually done is made successive draws from a uniform random number generator. The software is designed to return numbers that fall along the interval [0,1) with equal probability. Thresholding the numbers and dividing their counts is not a meaningful transformation; This is wrong. That is a very meaningful transformation. It is the standard way (https://stats.stackexchange.com/questions/240338 https://stats.stackexchange.com/questions/240338) to turn a uniform distribution into a Bernoulli distribution. Getting a single value with Bernoulli distribution is called a Bernoulli trial (https://en.wikipedia.org/wiki/Bernoulli_trial https://en.wikipedia.org/wiki/Bernoulli_trial). Repeating this gives you a Binomial distribution (see your own wikipedia link). Long story short: GPs code is a perfectly valid way of sampling the Bernoulli distribution. It is inefficient because it needs so many random values, but it mimics the actual process happening in real life making it easier to understand than generating a Binomial sample from the Binomial distribution's CDF.

deleted 4y ago

[deleted]