9 ms·
There's a really interesting mathematical 'paradox' with false positives along the same line as the birthday paradox. It's an important one also that many docto
by somenameforme 5y ago
There's a really interesting mathematical 'paradox' with false positives along the same line as the birthday paradox. It's an important one also that many doctors get wrong. Let's say there is a terrible disease that 0.01% (1 in 10,000) of the population is infected with, and we develop a test that is 99% accurate. You go to the doctor and get a test. Oh no, it's positive. What are the chances that it's a false positive?
Intuition would tell you 1% since the test is 99% accurate. Intuition, as it often is, is wrong. Take a sample size of 'x', we'll use 10,000. How many people will be infected? 1. How many false positives will there be? 1% * 10,000 = 100. So there will be 100 false positives, and 1 real positive. So it turns out there's a greater than 99% chance that your result is a false positive, even though the test is genuinely 99% accurate.
Here's a fun one. So we decide to run the test twice. And AGAIN it comes up positive. What are the odds that it's a false positive? Again, the exact same logic holds - there are 101 people we are testing. There will be about 1 false positive and about 1 real infection. So the odds of it being a false positive once again are about 50%.
- User23 5y agoA good example is screening men for breast cancer. It's possible for men to get breast cancer, but it's so vanishingly rare that the overwhelming majority of positive results will be false.
- reincarnate0x14 5y agoThis is, incidentally, the same basic reason why so many knee-jerk policies produce systems that can never do anything but fail. When trying to find needles in hay stacks, since the needle (let's say terrorists trying to board airplanes) is so tiny relative to the hay (the millions of other people doing same), the people responsible for reacting are essentially trained only on false positives and still miss the real events when they (highly infrequently) occur. Obviously with low-impact false positives like requiring more expensive or invasive, but not really dangerous, additional medical tests this is an easily accepted and accounted for set of outcomes, but as we try to do things like apply ML to ever more specific cases of complex questions it's worth keeping in mind.
- deleted 5y ago[deleted]
- majormajor 5y agoGoing a level further, the claim that the systems fail is itself tough to support! Let's say two million people have been seriously inconvenienced by airport screening. We have to ask if that's an acceptable price to pay for the benefits? But do we know the benefits? Can we know the benefits? We actually don't know if the policy deterred anything, or even if someone who wanted to do something bad had their knife or liquid explosive or whatever confiscated and just abandoned the plan. It's purely a hypothetical so you're going to support the view you want: "we don't need to do all this because we don't actually have a big threat" or "we have successfully prevented any more attacks." After all, we often see discussions here of foolish companies that underfund their security teams because "why are we paying so much for this when nobody has cyberattacked us?" ;)
- potatoman22 5y agoFalse positives lead to poor use of resources
- idrios 5y agoThere's a version of this that I was once told and now repeat to everyone else: When you have a condition that affects only 1% of the population, a test that reads negative 100% of the time will be 99% accurate.
- thaumasiotes 5y agoBut tests are not generally described in terms of accuracy. They're described in terms of sensitivity and specificity, because those quantities are both "accuracy" but they're not closely related to each other. A test that is always negative has 0% sensitivity and 100% specificity. Nobody thinks that sounds good.
- cortesoft 5y agoThat is how tests are described in scientific circles, but not with the general public. An overall `accuracy` number is usually given.
- amelius 5y agoPerhaps multiply sensitivity by specificity then, to get a single number that the public can use?
- thaumasiotes 5y agoThat's not a number the public can use. What would it be supposed to mean?
- amelius 5y agoA fuzzy AND of sensitivity and specificity. Another option is to take the minimum (part of the Zadeh operators). https://en.wikipedia.org/wiki/Fuzzy_logic https://en.wikipedia.org/wiki/Fuzzy_logic
- 5y ago
- giovannibonetti 5y ago> So the odds of it being a false positive once again are about 50%. It's even worse in practice, given that the false positive rate is probably not independent between both measurements
- dontreact 5y agoYeah this is why in practice the followup to a mammogram is never going to be another mammogram. It's a different test which is more invasive but also produces far less false positives: taking a tissue sample.
- tmoertel 5y ago> Here's a fun one. So we decide to run the test twice. And AGAIN it comes up positive. What are the odds that it's a false positive? Again, the exact same logic holds - there are 101 people we are testing. There will be about 1 false positive and about 1 real infection. So the odds of it being a false positive once again are about 50%. This conclusion assumes that running the test twice is equivalent to running two independent tests for the same condition. For many real tests, getting a false positive once predicts a higher chance of getting another false positive in a repeated test.
- deleted 5y ago[deleted]
- reincarnate0x14 5y agoAlso depends a lot on the nature of the screening and what is being screened for. Prostate cancer being somewhat of an outlier as at least to my understanding, essentially all men will end up with some kind of tumor or another if they live long enough, but some large fraction of those won't become something that requires medical intervention before the individual would die from any number of other age-related causes. Outside my area of expertise but the relative size of the tissue that produces various breast cancers as well as the proximity to lymph nodes change the calculus on that a bit. But I would definitely expect overall screening protocols to evolve as methodologies are developed that can potentially exclude the false positives from another metholodgy. While test A + A may still produce false positives due the nature of the test failing the same way, if we have A + B + C that are all relatively low-impact and not horribly expensive, that may end up being a fairly good program if some combination of A, B, and C generally exclude the parameters that the other produces a false positive for.
- viraptor 5y ago> as well as the proximity to lymph nodes change the calculus on that a bit. That sounds interesting. Why does lymph node proximity matter?
- 5y ago
- giantg2 5y agoThis doesn't sound quite right. I think it's ignoring randomness and not accounting for the thresholds correctly.
- SamBam 5y agoI'm pretty sure it is quite right. It's standard Bayesian statistics. If the math is wrong, show us.
- giantg2 5y agoIf the right, show the proof (or at least the real math). Specifically, why would the 10,001st test carry a 50% chance of being a false positive?
- tetromino_ 5y agoYou misunderstood the scenario: 10,000 people would receive a test. Out of those, ~100 would be expected to test positive. Only those ~100 people would receive a second test. Thus, the population for tests 1..10,000 is different than the population for tests 10,001..10,100.
- giantg2 5y agoOk... and why would that second population carry a 50% risk of the second test being a false positive? Edit: I misunderstood the scenario.
- rndgermandude 5y agoIn this example? Because we know that one is a true positive. We tested 100 people with a false positive, and 1 with a true positive. If we test again, and the test is "fair", then of these 101 people, we will retest 99% (or 99) true negative, 1% (or 1) false positive, and 1 true positive. So two positives, one false positive and one true positive. 1 out of 2 is 50%. (Tho, there is a chance also that the true positive might come back as a false negative, then 1 out of 1 or 100% of positive results would be false positive)
- hadmatter 5y agoFunnily enough, accuracy is a somewhat inaccurate term when it comes to medical testing. What we are interested in are sensitivity, specificity [0] and prevalence of the disease in the population being tested. With those three we can get positive prediction value (PPV) and negative prediction value (NPV) [1]. But PPV and NPV always depend on prevalence! That's why we don't want to test healthy people for all kinds of exotic diseases. [0] https://en.wikipedia.org/wiki/Sensitivity_and_specificity https://en.wikipedia.org/wiki/Sensitivity_and_specificity [1] https://en.wikipedia.org/wiki/Positive_and_negative_predictive_values https://en.wikipedia.org/wiki/Positive_and_negative_predicti...
- dontreact 5y agoHowever, despite the high false positive rate, breast cancer screening is a net beneficial procedure (though not without controversy) in that when applied in a population, long term outcomes like mortality improve. (edit yes of course cancer is not beneficial thank you)
- deleted 5y ago[deleted]
- scottlamb 5y ago> However, despite the high false positive rate, breast cancer screening is a net beneficial procedure (though not without controversy) in that when applied in a population, long term outcomes like mortality improve. I've read that's also a statistically tricky statement. IIRC, they measure survival time from when the cancer was detected. So early detection by definition improves the survival stats even when there's no meaningful treatment. If I've read about this, I'm sure medical practitioners are well aware of it, but even so, I don't think it's easy to correct for it. How do you determine when a late-detected cancer actually started?
- dontreact 5y agoYou don't, you just measure long term outcomes like mortality in a treatment and control arm. Potentially you have to use a synthetic control. I am most familiar with NLST which actually was a long term randomized trial.
- deleted 5y ago[deleted]
- Apes 5y agoWhat's interesting to me is that the three big criteria tests are rated on don't take this into account at all. In a population of 1,000,000: TP=99 TN=989901 FP=9999 FN=1 Accuracy = TN + TP / (TP+TN+FP+FN) = 99% Sensitivity = TP / (TP + FN ) = 99% Specificity = TN / (TN + FP ) = 99% Looks like a great test across the board! However, the actual chance someone with a positive test result has the disease: TP / ( TP + FP ) = ~1% So why isn't this ratio part of the major test criteria?
- killjoywashere 5y agoPPV is absolutely something that's tracked. In fact, if you read the instructions for use (equivalent to "package insert") for lab diagnostics, you'll find the manufacturer's truth tables and can calculate the PPV if it's not in there. FDA keeps them online, so you don't even need to buy the diagnostic. Look the up.
- majormajor 5y agoI'm not sure 99% is considered the gold standard you think it is. Sure, a lot of our tests can't even do that well, but we know that 99% isn't a lot for large populations. We even have jargon like "five nines", "six sigma" around the fact that 99% isn't a lot.
- thaumasiotes 5y ago> So why isn't this ratio part of the major test criteria? Because it has nothing to do with the test. Sensitivity and specificity are facts about the test, and conceptually they will remain whatever they are if you start testing a different population. Whether doing a test has any value is a fact about the population ("olympic athletes are never anemic, so there's no point in testing them for anemia") that has nothing to do with the test. Why would you evaluate the test based on that?
- colinmhayes 5y ago> So why isn't this ratio part of the major test criteria? I think it's because this number depends on the percent of the population that is positive which can change after the trial is conducted.
- croes 5y agoAnd if you get two positive tests and were each time randomly selected for the test, the probability to be really positive is 99% https://github.com/khuyentran1401/Data-science/blob/master/statistics/bayesian_theorem.ipynb https://github.com/khuyentran1401/Data-science/blob/master/s...
- deleted 5y ago[deleted]
- tejohnso 5y agoSo we say something like, "the test is 99% accurate, giving a ~0.99 percent positive test result reliability across the tested population." Is there a specific metric word or phrase for this less than 1% positive test result reliability? Edit: Looks like hadmatter responded with the term I'm looking for!
- InitialLastName 5y agoNote that the basic Bayesian analysis relies on the samples being independent, so if the same test is re-run on the same patient, equipment, process and facility, the error is likely correlated to the previous test, so the chance that it's still a false positive is even higher than an independent analysis would suggest.
- jdthedisciple 5y agoImo you are exaggerating a bit: What you are describing is specificity vs sensitivity [1] which I am fairly certain doctors get educated about pretty early on in their studies. Once you know about that your examples shouldn't be surprising to anyone. [1] https://en.m.wikipedia.org/wiki/Sensitivity_and_specificity https://en.m.wikipedia.org/wiki/Sensitivity_and_specificity
- sarchertech 5y agoAny younger doctor has most certainly had sensitivity vs specificity hammered into them in med school. I remember reading studies about doctors not understanding the stats years ago, but my wife assures me that medical education has over corrected in the last decade or 2.
- dragonwriter 5y ago> Let's say there is a terrible disease that 0.01% (1 in 10,000) of the population is infected with, and we develop a test that is 99% accurate. You go to the doctor and get a test. Oh no, it's positive. What are the chances that it's a false positive? The correct answer is that the question doesn't give enough information to know. I mean, a test that always said no would be 99.99% accurate. A mere 99% accuracy could be no false negatives and 0.99% of all tests false positives, in which case the answer is that a positive result is 1% likely to be correct, and 99% likely to be a false positive. Or it could be, on the other extreme, 0.01% false negative and 0.98% of all tests false positives, in which case the chance that a positive result represents an actual case would be 0%. Presumably, though, approval processes for medical tests will constrain the actual distribution of results to be such that, a patient with positive result is not more likely to be clear of the condition than one with a negative result, so it won't be as bad as the latter extreme, but it's worth noting that your assumption (essentially that the false/true ratio is equal for negative and positive results) is not certain at all.
- Cpoll 5y ago> You go to the doctor and get a test. Nitpick here. There's nothing wrong with the math of Bayesian probability, but the setup is usually something like an STD test, and in those cases the prior is wrong. Our prior probability shouldn't be "0.01% (1 in 10,000) of the population is infected," we really want % of infection for the population that get tested. This is really obvious with disease tests because people don't typically get tests without reasons (symptoms, contact with infected persons, etc.). The math as-is would only work if we're sampling people randomly.
- WhitneyLand 5y agoWho defines “99% accurate”? Couldn’t it mean zero false positives and 1/100 positives missed? Couldn’t it also alternatively be defined as 99% accurate for a given person? Maybe there are reasons/conventions it is always as you originally describe, but if that’s the case wouldn’t it be more about poor communication to the public than a paradox?
- dundarious 5y agoI think it’s correct that “99% accurate” here means “99% true positive and 1% false positive”. Agreed that “accurate” is ironically an awfully imprecise word to use. As for why it is the case, I presume it is just the usual focus on counting your hits and not on your misses.
- evancoop 5y agoThis is even more terrifying when one considers the possibility that the result of the second test is not independent from the result of the first. That is, some trait within the population increases the probability of a false positive (and this is likely in any heterogeneous population). Now the conditional probability of infection, given TWO positive tests is still far below 50%. This is why a second test of a different type, even if it is less accurate, if it is more likely to be independent, is much preferred!
- Sparyjerry 5y agoI feel like the general use of accuracy assumes no false positives. When I hear "accuracy" in general conversation I assume it's 99% percent chance of a true positive and a 1% chance of false negative and ignore chance of a false positive and true negative when considering accuracy. Of course everyone already mentioned the various ratios that scientifically actually make up accuracy.
- acchow 5y agoIgnoring the difference between recall and precision - off the bat, it seems weird to rely on a test with some 1% error rate for a disease that has a 0.01% occurrence rate. Without doing the math, it seems like it would be mostly “wrong” in some sense (
- ironmagma 5y agoCan someone describe the core reason the second test is a false positive is not 1%? Why does the outside world (number of people being tested) affect my own test error probability?
- andreareina 5y agoIt's the same reason that the first test doesn't get you 1%, but the testing pool has changed. Instead of testing 10000 people without and 1 person with, you're testing 100 people without (and the same person with). It's a Bayesian update. Every (independent) positive result increases your confidence of having the condition by 100x. At first you have no information so you assume the base rate, 1:10000. At the first test, that goes to 100:10000. Second test gets you to 10000:10000 or even odds. Third positive result finally gets you to 1000000:10000, or a 99% confidence of having the condition.
- MrJohz 5y agoA maths teacher once told me that if I'm struggling to work through a logical problem in my head, I should look at the extreme numbers, because that often gives a clearer picture of what's happening. This has helped me a lot, and is how I would approach this: Let's say we've got a test that: * If you've got cancer, detects it 100% of the time (to make the maths easy) * If you haven't got cancer, gives you a (true) negative 99% of the time, and a (false) positive 1% of the time. For starters, consider a population of 100 people, all of whom have cancer. Our magical test always finds these cases, so we have an 100% positive rate, and a 0% false positive rate. (None of these can be false positives because we know they all really do have cancer.) Now consider a group of 200 people, 50% of whom have cancer, and 50% of whom don't. The group who have cancer will still all get detected, but if the 100 people who don't have cancer, one of them will be marked as having cancer even though they don't. In total, that's 101 people who will get a positive result. So our overall positivity rate is 101/200 (~50%), but our false positivity rate is the number of false positives divided by the number of all positives, so 1/101 (~1%) Now consider the case where we have 101 people, of whom only one has cancer. If we run our test a third time, we of course detect the person with cancer, but of the remaining 100 who don't have cancer, we will detect a second person. That's two positive cases in 101 people, so our positivity rate is ~2%. However, of those two positive cases we know only one really has cancer - the false positivity rate is 1/2, or 50%. So that's three situations where the only thing we've varied is the proportion of the population who have the condition we're scanning for. (Technically we also changed how big the population was in the first place but that was just to make the maths easier - it doesn't affect the results at all.) However, we get a different rate of false positives in every test, because the number of false positives is dependent on the ratio between the number of people with cancer and the number of people without. It's also worth noting that this obviously doesn't just apply to cancer, but also other situations. Most relevant to us in the last couple of years is probably Covid tests - you might have seen over the summers some scepticism about how valuable it is to do lots and lots of tests, and that's because during those times, the number of affected people was lower, and so these tests were more likely to produce less useful results. However, in winter, even relatively poor tests become more useful, because there are more true positives, and therefore a lower rate of false positives.
- silisili 5y agoWow, breaking down the percentages like that with a real example was eye opening to me. If I got a positive on a 99% accurate test I'd be convinced I was dying. This really puts things into perspective...thanks.
- msrenee 5y agoI'm still pissed that I have to get a pap smear every couple years in spite of being with the same man since I turned 18. Doctors won't prescribe birth control without one. I had a lovely biopsy of cervical tissue a couple years ago. Again, monogamous relationship with a man that started when we were both virgins. But I got tissue ripped out of my most sensitive region anyway. Why am I forced to submit to these screening tests for issues I'm not at risk for in order to receive tangentially related medical care?
- lostcolony 5y agoIf your gyno won't prescribe BC without a pap smear, look to get a new doctor. Insurance isn't mandating it; it's just the doc deciding they know better than you. Current medical advice does NOT stipulate the two should be related - https://www.acog.org/womens-health/experts-and-stories/ask-acog/do-i-need-to-have-a-pelvic-exam-to-get-birth-control https://www.acog.org/womens-health/experts-and-stories/ask-a... If you can't find a doc who won't decouple the two, try Planned Parenthood.
- msrenee 5y agoI may end up going through planned parenthood if that improves things. I really, really hate speculums up my cooch and have no predispositions to any issues in that area and there's absolutely no reason for me to be required to have a yearly exam to get my BC.