3 ms·
Right. And the frequentist version is also useful. If you test one person and the the test is positive, then that person is the president (p=0.00001). If you
by klodolph 13y ago
Right. And the frequentist version is also useful.
If you test one person and the the test is positive, then that person is the president (p=0.00001).
If you test a thousand people, and the test is positive for one of them, then that person is the president (p=0.01).
So you don't really need Bayesian logic to reason that you should test fewer people if you want a more significant result. (Note I'm not saying you don't need Bayes' Theorem, which everyone uses.)
Edit: I think most people on HN get their knowledge of frequentist and bayesian statistics from XKCD #1132. That's sad.
- jey 13y ago> So you don't really need Bayesian logic to reason that you should test fewer people if you want a more significant result. That's misleading by the use of the word "significant" which apparently means something in frequentism than it does in normal speech. I certainly wouldn't use "significant" in that way as a non-frequentist, I would instead rephrase what you said as: > So you don't really need Bayesian logic to reason that you should test fewer people if you want more confirmation bias in your result. And that's a statement I can definitely get behind!
- klodolph 13y agoThat's a good point. Conversely, when you run a test and get a result which is "not statistically significant", that doesn't mean that the measured effect is not significant. For example, if I test a drug and say "it does not cause cancer (p=0.17)", that's wrong. Yes, the correlation with cancer is not statistically significant, but it is significant in the normal sense of the word.
- _delirium 13y agoIt can definitely be used misleadingly, but it's not too out of line with normal scientific usage of the term. The "significant" in "significant figures" is the same: if a number has "11 significant figures", it doesn't mean the 11th digit is significant in the sense of being important or having a big impact, just that the 11th digit is within the measurement precision (as propagated through any subsequent calculations).
- pcrh 13y agoFurthermore, practicing biomedical researchers rarely, if ever, take a single study as "proof" of a hypothesis, no matter what the P-value is. Replication and plausibility in the context of other studies are taken into account, however these additional parameters are difficult to reduce to numbers. A recent blog post (link below) describes the actual practice of (good) biomedical research fairly well. http://www.tcpinnovations.com/drugbaron/the-all-new-good-old-fashioned-solution-to-the-replication-crisis-in-science/ http://www.tcpinnovations.com/drugbaron/the-all-new-good-old...
- Houshalter 13y agoThe point is not the number of people that are tested, but the prior probability that the person tested is the president. The person wandering around the white house already has a decent probability of being president. Some guy found in the middle of the country wearing every day clothes, not so much.
- GFK_of_xmaspast 13y agooh that's not fair, lots of people here read that lesswrong site (http://lesswrong.com/lw/qa/the_dilemma_science_or_bayes/ http://lesswrong.com/lw/qa/the_dilemma_science_or_bayes/)
- ronaldx 13y ago>If you test one person and the the test is positive, then that person is the president (p=0.00001). >If you test a thousand people, and the test is positive for one of them, then that person is the president (p=0.01). I am frequently a frequentist, but this particular test doesn't make sense to me. Without any prior belief, the person you tested is still no more likely to be the president than any of the other 3199 who would test positive.
- dougabug 13y agoThe parent poster is wrong, for the reason you surmise.
- dougabug 13y agoIf you test one person at random, the odds of that person being the President equals the probability of a true positive divided by the probability that the test provides a Positive result (i.e. the sum of the probabilities for True and False Positives). The probability of a true positive is the probability that test subject is the President times the conditional probability that the subject tests positive given that they are the President. If the subject is chosen at random from the population of the US, then the a priori odds of this person being the President is about 1/314M times a conditional probability of testing positive of 1. The probability of a false negative is the a priori odds of a random person being the President, times the conditional probability of a non-President testing positive. By hypothesis, this is (1-1/314M)0.00001. So the odds of a subject chosen at random from the overall US population must be approximately (1/314M)/((1/314M)+(1-1/314M)0.00001), which simplifies to 1/(1+(314M-1)*0.00001, or about 1/3141. The ~99.97% chance that the test result is a false positive for a subject chosen at random is a consequence of the prior probability of that subject being the President in the first place being over 3000 times lower than the probability of a positive test result. In other words, though the sensitivity of the test is perfect, the specificity of the test is insufficient to isolate a condition as rare as the Presidency. It has nothing to do with testing one person being inherently more decisive than testing more than one. Only narrowing the pool of candidates in a way which will increase the prior probability that the subject is the President will improve the situation, not random decimation to a single test subject. The GP gave several correct examples such as limiting the test to those who were much more likely to be the President in the first place (eg. a person randomly found sitting in the President's chair at the Oval Office might have a prior probability of being the President of greater than 0.01, millions of times greater than that of a randomly selected person in the US).
- comicjk 13y agoI think you're mistaken about what frequentist p-values mean. The definition you're using is the one we would like to have - the probability that X is president given the data. But what a p-value actually gives you is the chance of seeing the given data given that X is president. And if X is the president, we have p=0, since there are no false negatives. So a p-value is useless in this example, although better frequentist measures can handle it.
- klodolph 13y ago> seeing the given data given that X is president You've got it backwards. The p-value is the chance that we see the given data given that X is not president.
- vbs_redlof 13y agoLess wrong, but still off. The p-value is the probability of committing a type-1 error (false positive: we declare X to be president when he really isn't) given a sample and a null hypothesis. P-values are not the 'chances' of seeing the given data, it is a test-statistic that describe how reliable your probability estimates are for a given sample and null.
- onetwofiveten 13y agoActually I think we're still a little off. The p-value is the probability of seeing a test statistic at least as extreme as the observed sample statistic, under the assumption that the null hypothesis is true. This can be restated in terms of tests and errors. The p-value is the probability of committing a type-1 error for a statistical test, where the threshold of that test is chosen to be equal to the sample statistic. In more intuitive terms, it's the probability of a type-1 error under the null hypothesis for the most extreme test that the sample data is able to pass. In your explanation, where you said "given a sample" you should say "given a sample size", to make it clear that the type-1 error probability is not conditional on the sample data. If you condition on the observed data, then the probability of passing a statistical test is going to be 1 or 0 depending on whether or not the data passes the test. It is the test itself, not the error probability, that depends on the sample data. Which is also something you should have specified.
- nkurz 13y agoIt seems awkward that p-value does not account for the setting in which the test is performed. That is, if I test 1000 people at a high school football game in Peoria, Illinois, and one of them comes out positive (p=0.01), there is a very different likelihood that I have found the true president than if I test 1000 people at a official dinner in the White House and find a match (also p=0.01). In fact, I think I'd be willing to wager more on the chance that the individual tested in the White House is actually the president than the random football fan in Peoria, even if only a single person in Peoria was tested (p=0.00001). Of course, this is only an issue if p-value is being used to express relative confidence in a conclusion, which it shouldn't (?) be. Still, how does a frequentist account for a choice of venue like this?
- vbs_redlof 13y agoP-values are not comparable across different null hypotheses, they are a property of a particular sample given a null hypothesis. You can compare p-values only by making very strong assumptions that the samples come from the same data-generating process. In almost all cases this is unlikely. A frequentist usually builds a more descriptive model by adding more variables (accounting for obvious factors that contribute to differences in observed frequencies) and increasing sample size to increase statistical power (reduce false negatives). This is not applicable in this case because there is only one president and the tests have low power. The test is bogus because you can't 'sample' and determine to probability of X==president effectively.
- klodolph 13y agoFrequentist approach would be to design an experiment, determine rules for interpreting data, and then consider the probability that the experiment lead to the correct conclusion. In this case, the experiment is really "who is president?" and not "is person X president?" The "who is president?" question does not really have a null hypothesis, so it makes no sense to talk about p-values. Instead, we can talk about two different results: identified the president correctly, and identified the president incorrectly. We then have to design the experiment in such a way that probabilities can be calculated. But this is not possible if you say the repeated experiment is testing a bunch of people to see if one is president--because the probability depends on who actually is president, and that's not known. So you consider the experiment to be "the president goes about their business and ends up in some random place, we get amnesia, and then run the test." We can simulate this experiment because we can come up with some probability distribution for where the president is. We can't come up with a probability for who the president is, because that's unknown and either 0% or 100%--only Bayesians would let you do that. Then you can choose the order of people to test to maximize the probability that the president will be identified correctly. And then you say things like, * The test had a 92% chance of succeeding. * We only had to test 14 people, and in such cases, the test had a 98% chance of succeeding.