5 ms·

0.5% false positive rate is actually quite high. I don't have the numbers, but I'd be surprised if more than 1% of the people that would receive such a screenin

0.5% false positive rate is actually quite high. I don't have the numbers, but I'd be surprised if more than 1% of the people that would receive such a screening actually already have cancer, which would mean that it produces about as many false positives as false negatives. Picture a population of a million people, all receiving the test. 1% of them have cancer (unknown to them) - half of those people get a negative screening result and half get a positive one (51% successful identification), and 0.5% of the full population get a negative screening result. Those numbers are roughly equal. That doesn't make it useless by any means, but it's not nearly as 'specific' as it sounds on paper.

gradys 5y ago
There's an interesting discussion raging on elsewhere in the thread about the possibility for cultural adaptation that would enable us to cope with statements like "the cancer test was positive, but there's still a decent chance you don't have cancer". There's also an interesting question, taking your roughly 1:1 true to false positive ratio as an example, of whether the marginal true positive does more good for the world than the marginal false positive. Side-stepping those questions though, they probably won't give this test to literally every person in the population. How much does the picture improve when you give this only to people over 50? Or only smokers? You could still massively increase effective screening for cancer with a cheap and easy test if you combine it with enough population filtering to increase the true positive/false positive ratio to something like 70-90% rather than the 50%
- nevinera 5y ago
  That side-step was my actual assumption - I can't imagine that 1% of the _whole population_ currently has actionable cancer. But yes, the higher you drive the prior, the less the false-positives cost. That's a fairly normal situation - it's why they have to evaluate the cost-value of further-investigation against so many different population groups to determine the actual optimal usage. And yes, the question I was trying to prompt really is "what information would we need to have in order to know when this test is a net positive, from the patient's perspective" (and separately, "from the insurance company's perspective", since that's likely to have a very different answer).
randcraw 5y ago
I would very much like to know the confusion matrix for all these component tests: the false positive rate, false negative rate, i.e., the sensitivity and specificity of each. For example, the Prostate Specific Antigen (PSA) test is famous for a high false positive rate, inviting many men to worry needlessly or get unwarranted biopsies. The current state of any test and its history are important to know for both patient and doctor, so both can have an informed dialog about the options after a positive result. Also, each of these 50 tests are going to evolve over time, as will the accuracy of error rates for patient subpopulations. Race, age, ethnicity, co-morbidities and co-maladies will each shift the accuracy and precision of each test in ways that will make them much more useful if everyone stays well informed about all their merits and demerits.
moralestapia 5y ago
I feel it's a net win anyway, if the procedure is not invasive and cheap, you could take one every so often (yearly?) and if something shows up you just double check with a better method or w/e. Better than nothing, which is the current alternative.
- Robotbeat 5y ago
  Right. If the objection is that it has a high false positive rate and doctors don’t understand Bayesian statistics enough to take that into account, then the solution isn’t to ban the test but instead require clinicians who interpret the test to learn Bayesian statistics better.
- colinmhayes 5y ago
  The problem is people are idiots. They get a false positive and their doctors tells them "well there are a lot of false positives with this test" but all people hear is "you have cancer." Then they get an invasive biopsy because the healthcare system is glad to take your money.
- scarecrowboat 5y ago
  What if the followup procedure is invasive and expensive? It's possible that this procedure could be net negative in aggregate depending on the % of false positives and # of resulting needless followup procedures. Not to mention the emotional stress.
  - timy2shoes 5y ago
    Exactly! This reminds of the recent case of the AMA increasing the suggested age of regular mammograms. More testing means earlier detection of breast cancer. But a false positive means an invasive biopsy. These are the two sides that need to be balanced. The AMA decided that since the base rate of cancer is so low in women under 50, the posterior probability that you have cancer given a positive mammogram for the under 50 age group is considerably low. Therefore they decided that the increased costs (monetary and emotional) do not outweigh the benefits. From https://www.ama-assn.org/sites/ama-assn.org/files/corp/media-browser/public/about-ama/councils/Council%20Reports/council-on-science-public-health/a12-csaph6-screeningmammography.pdf https://www.ama-assn.org/sites/ama-assn.org/files/corp/media... "Screening mammography reduces mortality from breast cancer, including in women younger than age 50 years. However, screening mammography carries harms such as false positive results that can lead to additional imaging and invasive biopsy procedures, and overdiagnosis that could lead to treatment in patients who may not benefit from it. The USPSTF considered the balance of benefits and harms using a commissioned targeted systematic evidence review of randomized clinical trials and a decision analysis that compared the expected health outcomes of starting and ending mammography at different ages and using annual and biennial screening strategies; it concluded (in part) that routine screening should begin at age 50 years and continue biennially until age 74 years."
vmception 5y ago
So still get a second opinion just as you would have anyway. It’s still getting caught
bick_nyers 5y ago
If it's cheap enough then get it performed 3 times.

gradys 5y ago

There's an interesting discussion raging on elsewhere in the thread about the possibility for cultural adaptation that would enable us to cope with statements like "the cancer test was positive, but there's still a decent chance you don't have cancer". There's also an interesting question, taking your roughly 1:1 true to false positive ratio as an example, of whether the marginal true positive does more good for the world than the marginal false positive. Side-stepping those questions though, they probably won't give this test to literally every person in the population. How much does the picture improve when you give this only to people over 50? Or only smokers? You could still massively increase effective screening for cancer with a cheap and easy test if you combine it with enough population filtering to increase the true positive/false positive ratio to something like 70-90% rather than the 50%

nevinera 5y ago
That side-step was my actual assumption - I can't imagine that 1% of the _whole population_ currently has actionable cancer. But yes, the higher you drive the prior, the less the false-positives cost. That's a fairly normal situation - it's why they have to evaluate the cost-value of further-investigation against so many different population groups to determine the actual optimal usage. And yes, the question I was trying to prompt really is "what information would we need to have in order to know when this test is a net positive, from the patient's perspective" (and separately, "from the insurance company's perspective", since that's likely to have a very different answer).

randcraw 5y ago

I would very much like to know the confusion matrix for all these component tests: the false positive rate, false negative rate, i.e., the sensitivity and specificity of each. For example, the Prostate Specific Antigen (PSA) test is famous for a high false positive rate, inviting many men to worry needlessly or get unwarranted biopsies. The current state of any test and its history are important to know for both patient and doctor, so both can have an informed dialog about the options after a positive result. Also, each of these 50 tests are going to evolve over time, as will the accuracy of error rates for patient subpopulations. Race, age, ethnicity, co-morbidities and co-maladies will each shift the accuracy and precision of each test in ways that will make them much more useful if everyone stays well informed about all their merits and demerits.

moralestapia 5y ago

I feel it's a net win anyway, if the procedure is not invasive and cheap, you could take one every so often (yearly?) and if something shows up you just double check with a better method or w/e. Better than nothing, which is the current alternative.

Robotbeat 5y ago
Right. If the objection is that it has a high false positive rate and doctors don’t understand Bayesian statistics enough to take that into account, then the solution isn’t to ban the test but instead require clinicians who interpret the test to learn Bayesian statistics better.
colinmhayes 5y ago
The problem is people are idiots. They get a false positive and their doctors tells them "well there are a lot of false positives with this test" but all people hear is "you have cancer." Then they get an invasive biopsy because the healthcare system is glad to take your money.
scarecrowboat 5y ago
What if the followup procedure is invasive and expensive? It's possible that this procedure could be net negative in aggregate depending on the % of false positives and # of resulting needless followup procedures. Not to mention the emotional stress.
- timy2shoes 5y ago
  Exactly! This reminds of the recent case of the AMA increasing the suggested age of regular mammograms. More testing means earlier detection of breast cancer. But a false positive means an invasive biopsy. These are the two sides that need to be balanced. The AMA decided that since the base rate of cancer is so low in women under 50, the posterior probability that you have cancer given a positive mammogram for the under 50 age group is considerably low. Therefore they decided that the increased costs (monetary and emotional) do not outweigh the benefits. From https://www.ama-assn.org/sites/ama-assn.org/files/corp/media-browser/public/about-ama/councils/Council%20Reports/council-on-science-public-health/a12-csaph6-screeningmammography.pdf https://www.ama-assn.org/sites/ama-assn.org/files/corp/media... "Screening mammography reduces mortality from breast cancer, including in women younger than age 50 years. However, screening mammography carries harms such as false positive results that can lead to additional imaging and invasive biopsy procedures, and overdiagnosis that could lead to treatment in patients who may not benefit from it. The USPSTF considered the balance of benefits and harms using a commissioned targeted systematic evidence review of randomized clinical trials and a decision analysis that compared the expected health outcomes of starting and ending mammography at different ages and using annual and biennial screening strategies; it concluded (in part) that routine screening should begin at age 50 years and continue biennially until age 74 years."

vmception 5y ago

So still get a second opinion just as you would have anyway. It’s still getting caught

bick_nyers 5y ago

If it's cheap enough then get it performed 3 times.