4 ms·

I don’t know much about stats, but the idea that a theory is bunk because no studies have shown it’s bunk is pretty absurd. How many studies fail to show an ass

I don’t know much about stats, but the idea that a theory is bunk because no studies have shown it’s bunk is pretty absurd. How many studies fail to show an association between narcotics abuse and crime? Zero? Then it must be fake?

MontyCarloHall 5y ago
Let’s say there’s a test that, by pure chance, has a probability of p=1/N of returning an unexpected outcome. If we run that test N times, we expect on average one unexpected outcome, solely due to random chance. I haven’t read the linked paper yet, but its authors’ argument is that they have orthogonal evidence that p in this case should be considerably larger than 1/24, so given 24 total studies on the lead crime hypothesis, at least one should have returned a negative result purely by chance. The validity of this claim depends entirely on whether their argument that p>>1/24 is sound.
- dllthomas 5y ago
  Though even if it's 1/12, the chance of no negative results by chance is still >10%, right?
jonas21 5y ago
I think their argument is a little more subtle than that -- the issue isn't really that there are no negative results, but rather that the shape of the distribution of results is funny. They're saying you'd expect the results to follow a normal distribution centered at some value. Instead, we see a large number of studies clustered just above zero and a right tail, but no left tail. I think they're arguing that this is because the left side of the distribution is unpublished, and without knowing what it looks like, you can't come to a conclusion.
6gvONxR4sf7o 5y ago
Picture a perfect bell curve where the whole left half has been chopped off. If you had an a priori reason to believe it’s a bell curve, then it’s easy to spot the fact that the left half has been chopped off, and easy to spot where the mean would have been.

MontyCarloHall 5y ago

Let’s say there’s a test that, by pure chance, has a probability of p=1/N of returning an unexpected outcome. If we run that test N times, we expect on average one unexpected outcome, solely due to random chance. I haven’t read the linked paper yet, but its authors’ argument is that they have orthogonal evidence that p in this case should be considerably larger than 1/24, so given 24 total studies on the lead crime hypothesis, at least one should have returned a negative result purely by chance. The validity of this claim depends entirely on whether their argument that p>>1/24 is sound.

dllthomas 5y ago
Though even if it's 1/12, the chance of no negative results by chance is still >10%, right?

jonas21 5y ago

I think their argument is a little more subtle than that -- the issue isn't really that there are no negative results, but rather that the shape of the distribution of results is funny. They're saying you'd expect the results to follow a normal distribution centered at some value. Instead, we see a large number of studies clustered just above zero and a right tail, but no left tail. I think they're arguing that this is because the left side of the distribution is unpublished, and without knowing what it looks like, you can't come to a conclusion.

6gvONxR4sf7o 5y ago

Picture a perfect bell curve where the whole left half has been chopped off. If you had an a priori reason to believe it’s a bell curve, then it’s easy to spot the fact that the left half has been chopped off, and easy to spot where the mean would have been.