35 ms·

Stats below: If there is no bias, the lost packages reports would have equal probability of having come from the athiest group or the non-athiest group. The (

Stats below: If there is no bias, the lost packages reports would have equal probability of having come from the athiest group or the non-athiest group. The (frequentist) probability that the ratio would be 9:1 or worse (given 10 lost packages) is 10/(2^10) which is approximately 0.98%. If you're the type to care about p-values, here we have p < 0.01. Suppose there indeed exists this bias as it appears (9 times more likely to lose athiest package), we might be curious to ask: how likely is it that we would have detected a statistically significant (p < 0.05) bias with just 10 packages lost total? Doing a similar calculation as above can check that we only see a statistical significance bias if the loss discrepancy is (10:0, 9:1, 1:9, or 0:10). Now if we suppose the bias is 9:1, how likely are we to have seen one of those events? If we add up the probability of these events (10:0, 9:1, 1:9, or 0:10) occurring under a biased scenario, we get about a 74% chance of us detecting the bias. A standard requirement is for this number to be at least 80%, our result is close but would like to see more data before publishing in the Journal of Atheist Footwear. The latter metric is called statistical power or the probability of not getting a false negative, whereas p-value is the probability of a false positive.

tnecniv 10y ago
I'm glad you ran this analysis. I was about to do something similar!
joemi 10y ago
I appreciate the statistics even though I don't really understand them all that well. Since you seem fairly knowledgable of statistics (though I know a lot of people here are too, so other chime in too if you want), could a case be made that we shouldn't be comparing 9 lost decorative tape packages out of 178 with 1 lost non-decorative tape package out of 178, but instead be comparing 9 directly with 1? When 10 packages And if you're curious, please see my other reply as to why I think, even including statistical significance of one type being lost more than the other, we can't really draw a "religious bias" conclusion.
- alexbeloi 10y ago
  Yep, the numbers I ran are already doing exactly what you suggest. Looking exclusively at the 10 lost packages and considering how likely it should be that 9 of them would belong to one category, compared to our expectation (which is that they would be equally likely to have come from either category, coming from the fact that the same number of packages were shipped with Atheist labeling as without). The statistics is entirely identical to flipping a coin 10 times and asking the following questions: * Given that I saw 9 heads and 1 tail, how likely is it that the coin is a fair coin. This is the p-value which was about 0.0098 (less than 1% chance) * Given that we are only tossing 10 coins, how likely are we to have seen a statistically significant bias with the predicted bias in place. This is the statistical power, which was 74%. As you suggest in your other comment, the stats only point to a bias, the origin of the bias is left for people to decide. That said, the packages were sent in pairs, so subdividing the country into states and offices is irrelevant since a given states saw the same number of packages from each category as did each given office, the statistics are saying that on average (across the whole country) there is a bias. To ease your worries a little, this is not saying the USPS hates Atheist packages 9x more than non-Atheist packages, most of the packages were delivered successfully. All this is saying is that, for the ones that were lost, it is very likely that they were lost because of their labeling. This is the effect you would expect to see if 97.3% of USPS employees treated the packages identically and the other 2.7% have a strong bias. If we had more specifics about the delayed packages as well, we could get a more nuanced picture, I suspect we would find that 80+% of USPS employees are unbiased, some 10-15% might have some bias against Atheist labeling (would delay an Atheist labeled package), and 2-3% have a strong bias against Atheist labeling (would lose an Atheist labeled package). If I was at USPS, I would push to conduct an experiment to try to identify which offices were most likely to have some bias one way or another (not necessarily specifically about Atheism, one could test whatever they wanted presumably).

tnecniv 10y ago

I'm glad you ran this analysis. I was about to do something similar!

joemi 10y ago

I appreciate the statistics even though I don't really understand them all that well. Since you seem fairly knowledgable of statistics (though I know a lot of people here are too, so other chime in too if you want), could a case be made that we shouldn't be comparing 9 lost decorative tape packages out of 178 with 1 lost non-decorative tape package out of 178, but instead be comparing 9 directly with 1? When 10 packages And if you're curious, please see my other reply as to why I think, even including statistical significance of one type being lost more than the other, we can't really draw a "religious bias" conclusion.

alexbeloi 10y ago
Yep, the numbers I ran are already doing exactly what you suggest. Looking exclusively at the 10 lost packages and considering how likely it should be that 9 of them would belong to one category, compared to our expectation (which is that they would be equally likely to have come from either category, coming from the fact that the same number of packages were shipped with Atheist labeling as without). The statistics is entirely identical to flipping a coin 10 times and asking the following questions: * Given that I saw 9 heads and 1 tail, how likely is it that the coin is a fair coin. This is the p-value which was about 0.0098 (less than 1% chance) * Given that we are only tossing 10 coins, how likely are we to have seen a statistically significant bias with the predicted bias in place. This is the statistical power, which was 74%. As you suggest in your other comment, the stats only point to a bias, the origin of the bias is left for people to decide. That said, the packages were sent in pairs, so subdividing the country into states and offices is irrelevant since a given states saw the same number of packages from each category as did each given office, the statistics are saying that on average (across the whole country) there is a bias. To ease your worries a little, this is not saying the USPS hates Atheist packages 9x more than non-Atheist packages, most of the packages were delivered successfully. All this is saying is that, for the ones that were lost, it is very likely that they were lost because of their labeling. This is the effect you would expect to see if 97.3% of USPS employees treated the packages identically and the other 2.7% have a strong bias. If we had more specifics about the delayed packages as well, we could get a more nuanced picture, I suspect we would find that 80+% of USPS employees are unbiased, some 10-15% might have some bias against Atheist labeling (would delay an Atheist labeled package), and 2-3% have a strong bias against Atheist labeling (would lose an Atheist labeled package). If I was at USPS, I would push to conduct an experiment to try to identify which offices were most likely to have some bias one way or another (not necessarily specifically about Atheism, one could test whatever they wanted presumably).