4 ms·

So we have 800 scientists signing a paper, but there are on the order of 7 million scientists worldwide. To prove the hypothesis "scientists rise up against st

So we have 800 scientists signing a paper, but there are on the order of 7 million scientists worldwide. To prove the hypothesis "scientists rise up against statistical significance" with 95% confidence level and a .1 confidence interval, we need a sample size of 857,462. A sample size of just 800 is clearly not statistically significant, so the paper is meaningless and the hypothesis can be rejected. Am I doing this right?

humbledrone 8y ago
What you want is a one-tailed Binomial test: https://en.wikipedia.org/wiki/Binomial_test https://en.wikipedia.org/wiki/Binomial_test You need these ingredients: - The scientists polled for whether they're rising up need to be randomly sampled from the population of 7M - A threshold for the fraction of scientists rising up to consider an uprising to be occurring (e.g. 90% of scientists rising up indicates an uprising) Then you can poll your scientists and feed the results into the Binomial test to find out whether there is statistical evidence to support the hypothesis that an uprising is occurring. Note that the sample size to do this might be quite small. In general, the required sample size is not dependent on the size of the population, but rather on how close the actual fraction of scientists rising up is to your chosen threshold. If the true fraction of scientists rising up is much smaller than your threshold, say 10%, and your threshold is 90%, it might only take a random sample of 10-20 scientists to be confident there's no uprising. But if the true fraction is 89.99999% it would take a huge sample to be confident that no uprising is occurring.
pulisse 8y ago
> Am I doing this right? Sampling error is independent of population size, so, no.
- ilyaeck 8y ago
  Also, the forces influencing the society are not statistical, at least not in this sense. A small minority in a group is definitely capable of challenging old norms and installing new ones. In fact, that is how it usually happens.
d-sc 8y ago
I think you provide a convincing case of the problems of statistical significance: Without context on the 800 scientists that signed, we don’t know whether it’s stastically significant or not. 800 Nobel prize winners would certainly mean more than 800 graduate students. The discipline they were in could matter. I didn’t check you’re math, and I’m sure it’s fine. Raw statistics cannot reject the hypothesis (or null).
- amluto 8y ago
  Huh? This has nothing to do with statistical significance. Also, it sounds like the authors didn’t sample scientists at all. They solicited signatures with some minimal qualifications. The 800 number wasn’t claimed to prove anything.
amluto 8y ago
Something I find helpful when doing statistics is to clearly state exactly what you’re trying to show or calculate before you calculate anything. “with 95% confidence level and a .1 confidence interval” is not clearly stated. Here’s something you might credibly calculate: if you ask n scientists a yes/no question, but actually only half of scientists worldwide think “yes”, then what’s the chance that m or more of them in your sample say “yes”? The answer is the p value, and it’s not very hard to calculate. The point of this article is that many, many papers do this calculation and ascribe totally inappropriate meaning to it. And, of course, when you clearly state what you actually calculated, it’s pretty obvious that it’s a terrible way to draw conclusions about what fraction of scientists think “yes”. This is the point of the Nature paper. P.S. I don’t know what you’re trying to calculate, but I think you did it wrong. If you want your calculation checked, can you clarify your question?
- jpatokal 8y ago
  It was meant to be tongue firmly in cheek. Poe's law strikes again...
lisper 8y ago
To say nothing of the selection bias at work here.
mancerayder 8y ago
So we have 800 scientists signing a paper, but there are on the order of 7 million scientists worldwide. Hence rising up against statistical significance.

humbledrone 8y ago

What you want is a one-tailed Binomial test: https://en.wikipedia.org/wiki/Binomial_test https://en.wikipedia.org/wiki/Binomial_test You need these ingredients: - The scientists polled for whether they're rising up need to be randomly sampled from the population of 7M - A threshold for the fraction of scientists rising up to consider an uprising to be occurring (e.g. 90% of scientists rising up indicates an uprising) Then you can poll your scientists and feed the results into the Binomial test to find out whether there is statistical evidence to support the hypothesis that an uprising is occurring. Note that the sample size to do this might be quite small. In general, the required sample size is not dependent on the size of the population, but rather on how close the actual fraction of scientists rising up is to your chosen threshold. If the true fraction of scientists rising up is much smaller than your threshold, say 10%, and your threshold is 90%, it might only take a random sample of 10-20 scientists to be confident there's no uprising. But if the true fraction is 89.99999% it would take a huge sample to be confident that no uprising is occurring.

pulisse 8y ago

> Am I doing this right? Sampling error is independent of population size, so, no.

ilyaeck 8y ago
Also, the forces influencing the society are not statistical, at least not in this sense. A small minority in a group is definitely capable of challenging old norms and installing new ones. In fact, that is how it usually happens.

d-sc 8y ago

I think you provide a convincing case of the problems of statistical significance: Without context on the 800 scientists that signed, we don’t know whether it’s stastically significant or not. 800 Nobel prize winners would certainly mean more than 800 graduate students. The discipline they were in could matter. I didn’t check you’re math, and I’m sure it’s fine. Raw statistics cannot reject the hypothesis (or null).

amluto 8y ago
Huh? This has nothing to do with statistical significance. Also, it sounds like the authors didn’t sample scientists at all. They solicited signatures with some minimal qualifications. The 800 number wasn’t claimed to prove anything.

amluto 8y ago

Something I find helpful when doing statistics is to clearly state exactly what you’re trying to show or calculate before you calculate anything. “with 95% confidence level and a .1 confidence interval” is not clearly stated. Here’s something you might credibly calculate: if you ask n scientists a yes/no question, but actually only half of scientists worldwide think “yes”, then what’s the chance that m or more of them in your sample say “yes”? The answer is the p value, and it’s not very hard to calculate. The point of this article is that many, many papers do this calculation and ascribe totally inappropriate meaning to it. And, of course, when you clearly state what you actually calculated, it’s pretty obvious that it’s a terrible way to draw conclusions about what fraction of scientists think “yes”. This is the point of the Nature paper. P.S. I don’t know what you’re trying to calculate, but I think you did it wrong. If you want your calculation checked, can you clarify your question?

jpatokal 8y ago
It was meant to be tongue firmly in cheek. Poe's law strikes again...

lisper 8y ago

To say nothing of the selection bias at work here.

mancerayder 8y ago

So we have 800 scientists signing a paper, but there are on the order of 7 million scientists worldwide. Hence rising up against statistical significance.