6 ms·

IMHO, some of those criticizing the article for failing to understand statistics are missing the point. The point is that people who get a "positive" result on

IMHO, some of those criticizing the article for failing to understand statistics are missing the point. The point is that people who get a "positive" result on these tests are often put through terrifying levels of anxiety when there is no actual problem; this anxiety is often exacerbated because they aren't informed of the false positive rate. This clearly has a harmful emotional effect on people, and explaining the false positives in Bayesian terms, or reframing it in terms of sensitivity and specificity, doesn't undo that damage. That potential harm needs to be explained to patients, and it needs to be weighed carefully against the potential benefits of the test (as is done for PSA tests for prostate cancer, which also have a high false positive rate). Given that potential for harm, it's not unreasonable to ask that these tests be more tightly regulated. To quote the OP: > In interviews, 14 patients who got false positives said the experience was agonizing. They recalled frantically researching conditions they’d never heard of, followed by sleepless nights and days hiding their bulging bellies from friends. Eight said they never received any information about the possibility of a false positive, and five recalled that their doctor treated the test results as definitive. (Edit: clarified)

midjji 5y ago
If you get a positive for a horrid cancer with a 90 percent false positives you should be afraid. Its lunacy for tests to be regulated beyond requiring rough false positive false negative rates, and if anything smacks of "I dont understand statistics and therefore have to protect my children from understanding statistics." The article is most likely written by some anti abortion idiot.
- teruakohatu 5y ago
  > if you get a positive for a horrid cancer with a 90 percent false positives you should be afraid. No you should not panic, anymore than you should celebrate and buy a yatch if you think you have 10% chance of winning a lottery. The doctor should phrase it as "the test indicates you have a 10% possibility of cancer. The majority of people who test positive do not have cancer. Further testing is required to confirm or rule it out." Going with the expected value of 10% of a horrid thing is still bad, or 10% of $100m is still $10m, is not applicable to a single non repeated event.
  - halpert 5y ago
    The GP wasn’t saying go out and kill yourself because you are definitely condemned. They were highlighting that you still have an extremely elevated chance of having cancer. If you got into your car and found out you had a 10% chance of being in a fatal accident, would you not worry? Sure, there is a 90% probability you’ll be fine, but the probability for most other people is 99.999%. Pretending like there isn’t a potential issue is dangerous and willful ignorance.
- thaumasiotes 5y ago
  > Its lunacy for tests to be regulated beyond requiring rough false positive false negative rates There is a good reason tests are described in terms of sensitivity and specificity ("if the answer in reality is yes, how often will the test say no?") rather than in terms of false positive or false negative rates ("if the test says no, how often is the true answer yes?"). The sensitivity and specificity are facts about the test which can conceptually stay constant[1] as you apply the test to different people. False positive and false negative rates do not have that property; they are facts about the group you're performing tests on just as much as they're facts about the test. [1] This is not to say that the sensitivity and specificity of a test do stay constant as you apply the test to different populations. Often they won't. But it is a theoretical possibility, and even that isn't true for false negative rates.
  - midjji 5y ago
    For a test manufacturer you are right, and they aren't responsible for accounting for general population frequencies, or conditional frequencies based on things like some sibling has it etc. But there is no need to regulate that. But the serivice in question isnt a test manufacturer, its diagnostics, and they are responsible for this. I think a company selling a diagnostics service should be required to prominently display reasonably accurate FP,TP,FN,TN rates for gen pop, as the next example will show. This is is not a legal problem, its a customer experience one, and as such its very much what the free market does well. A clever company could say that the tests service we provide works in a two stage process where a preliminary test is used to identify which conditions need to be investigated further, the results of which are presented in a meeting with one of our doctors were the results are explained and additional tests are performed to resolve any ambiguity. This combines the benefits of sensitivity, and specificity in a resource efficient way, and does not provide the user with any scary results before a certain answer is available. The FP,FN,TP,TN probabilities for gen pop for such a company would be much better than those of companies trying to do so in a single pass. But for these figures to mean anything regulating that they are available and reasonably accurate is key to ensure an efficient market. Another problem is of course that the tests themselves are shit, or the rates based on poor data. The requirement on reasonably accurately reported gen pop rates help with this however, and I see no way around legally forcing diagnostic services providers to prove this. And until someone does, my patent pending algorithm(){return false} tests for all rare genetic disorders with excellent accuracy^^
derbOac 5y ago
I've worked clinically and taught this in professional training programs. It's a classical clinical decision making problem. The solution is fairly straightforward I think: require the test companies to report the posterior probability of the illness given the test result. Probably with some kind of uncertainty interval.
- rscho 5y ago
  The posterior depends on the prior. So not very feasible in practice.
  - derbOac 5y ago
    Generally speaking in clinical settings there are rough estimates of base rates of diseases available, either through biological theory (eg, mutation rates) or empirical studies or both. Not always to be sure, but often enough to mitigate against some of the problems discussed.

midjji 5y ago

If you get a positive for a horrid cancer with a 90 percent false positives you should be afraid. Its lunacy for tests to be regulated beyond requiring rough false positive false negative rates, and if anything smacks of "I dont understand statistics and therefore have to protect my children from understanding statistics." The article is most likely written by some anti abortion idiot.

teruakohatu 5y ago
> if you get a positive for a horrid cancer with a 90 percent false positives you should be afraid. No you should not panic, anymore than you should celebrate and buy a yatch if you think you have 10% chance of winning a lottery. The doctor should phrase it as "the test indicates you have a 10% possibility of cancer. The majority of people who test positive do not have cancer. Further testing is required to confirm or rule it out." Going with the expected value of 10% of a horrid thing is still bad, or 10% of $100m is still $10m, is not applicable to a single non repeated event.
- halpert 5y ago
  The GP wasn’t saying go out and kill yourself because you are definitely condemned. They were highlighting that you still have an extremely elevated chance of having cancer. If you got into your car and found out you had a 10% chance of being in a fatal accident, would you not worry? Sure, there is a 90% probability you’ll be fine, but the probability for most other people is 99.999%. Pretending like there isn’t a potential issue is dangerous and willful ignorance.
thaumasiotes 5y ago
> Its lunacy for tests to be regulated beyond requiring rough false positive false negative rates There is a good reason tests are described in terms of sensitivity and specificity ("if the answer in reality is yes, how often will the test say no?") rather than in terms of false positive or false negative rates ("if the test says no, how often is the true answer yes?"). The sensitivity and specificity are facts about the test which can conceptually stay constant[1] as you apply the test to different people. False positive and false negative rates do not have that property; they are facts about the group you're performing tests on just as much as they're facts about the test. [1] This is not to say that the sensitivity and specificity of a test do stay constant as you apply the test to different populations. Often they won't. But it is a theoretical possibility, and even that isn't true for false negative rates.
- midjji 5y ago
  For a test manufacturer you are right, and they aren't responsible for accounting for general population frequencies, or conditional frequencies based on things like some sibling has it etc. But there is no need to regulate that. But the serivice in question isnt a test manufacturer, its diagnostics, and they are responsible for this. I think a company selling a diagnostics service should be required to prominently display reasonably accurate FP,TP,FN,TN rates for gen pop, as the next example will show. This is is not a legal problem, its a customer experience one, and as such its very much what the free market does well. A clever company could say that the tests service we provide works in a two stage process where a preliminary test is used to identify which conditions need to be investigated further, the results of which are presented in a meeting with one of our doctors were the results are explained and additional tests are performed to resolve any ambiguity. This combines the benefits of sensitivity, and specificity in a resource efficient way, and does not provide the user with any scary results before a certain answer is available. The FP,FN,TP,TN probabilities for gen pop for such a company would be much better than those of companies trying to do so in a single pass. But for these figures to mean anything regulating that they are available and reasonably accurate is key to ensure an efficient market. Another problem is of course that the tests themselves are shit, or the rates based on poor data. The requirement on reasonably accurately reported gen pop rates help with this however, and I see no way around legally forcing diagnostic services providers to prove this. And until someone does, my patent pending algorithm(){return false} tests for all rare genetic disorders with excellent accuracy^^

derbOac 5y ago

I've worked clinically and taught this in professional training programs. It's a classical clinical decision making problem. The solution is fairly straightforward I think: require the test companies to report the posterior probability of the illness given the test result. Probably with some kind of uncertainty interval.

rscho 5y ago
The posterior depends on the prior. So not very feasible in practice.
- derbOac 5y ago
  Generally speaking in clinical settings there are rough estimates of base rates of diseases available, either through biological theory (eg, mutation rates) or empirical studies or both. Not always to be sure, but often enough to mitigate against some of the problems discussed.