5 ms·

Presenting the true positive rate without mentioning false positives seems problematic. I couldn’t find any information about that in the article, and they don’

Presenting the true positive rate without mentioning false positives seems problematic. I couldn’t find any information about that in the article, and they don’t link the paper / publication.

mtgx 8y ago
It's not problematic. It's on purpose and it's criminally misleading. If "AI companies" had to show their false positive rates, 99% of them would go bankrupt because then nobody would want to buy their bullshit.
samfriedman 8y ago
The paper itself [0] claims 82% specificity and 100% sensitivity. Meaning, 100% of positive cases in the test set were correctly identified, and 82% of negative cases in the test set were correctly identified. That corresponds to a 0% false negative rate and an 18% false positive rate. The test set is 40 imaging studies from 40 patients. [0] https://pubs.rsna.org/doi/pdf/10.1148/radiol.2018180958 https://pubs.rsna.org/doi/pdf/10.1148/radiol.2018180958
- SolaceQuantum 8y ago
  Wouldn’t 82% of negative cases being correctly identified mean that 18% of negative cases were incorrectly identified as positive(false positive), or in other words it will mistake 18% of healthy people as having Alzheimer’s, but will always identify an Alzheimer’s patient?
  - samfriedman 8y ago
    82% true negative rate == 18% false positive rate. Your description is correct. It's likely that those statistics will be somewhat worse when the algorithm is tested on a broader and more diverse test set, which the authors claim to plan to do in the future.
    - ThePhysicist 8y ago
      Thanks for digging up the paper, false positives are probably more acceptable than false negatives in this scenario (wrong positive diagnoses can be rectified with additional tests but a negative diagnosis will often not be confirmed by additional testing).
      - bloak 8y ago
        Both things matter, and you need to consider what the follow-up test might be and what it would cost not just financially but also in terms of pain, lost time, risk of harm and so on. There are medical tests which have such a high rate of false positives that in practice in the absence of other indications you would ignore a positive result, because the only possible follow-up test would be a non-risk-free biopsy, and therefore, for your peace of mind, it's probably better not to do the first test at all.
  - ThePhysicist 8y ago
    I think OP initially mixed up the numbers but now it's correct (18 % false positives).
- fencepost 8y ago
  A false positive rate (currently) that high may preclude this being used as a widely available diagnostic test, but if used within a patient base already showing other signs of problems it might be more acceptable. After all, if you're going to see a memory specialist you're already thinking there's a problem.
pge 8y ago
This is a key point. If the false positive rate is 18% (as noted in other comments), and the rate of Alzheimer's is 36/100k[1], then for every patient the test correctly identifies, there are 500 patients that tested positive but did not have Alzheimers. There needs to be significant improvement before such a test becomes useful. [1] that's mortality rate which I am using as a proxy for incidence rate, though the actual incidence rate is presumably higher (mortality rate from the CDC)
- gooseus 8y ago
  I'm not good at this (yet), but my understanding is that your prior for just this test would be the likelihood of having Alzheimer's based on the statistical occurrence within the general population. After this test you now have updated priors which may not be good enough to make a certain diagnosis, but wouldn't it still be useful combined with another independent test that has similar rates, since you'd be using these updated priors?
- bsder 8y ago
  That depends. I suspect that this test would only be used on those with a family history of Alzheimer's or similar. That alters the Bayesian priors quite a bit. In addition, you can monitor the progress of someone who "tests positive" over time and see if that is really true. Part of the issue with Alzheimer's treatments is that we may be intervening far too late. These kinds of tests may help that.
  - iseeyoubydesign 8y ago
    what ever , this is not science but gambling with peoples health to declare "look what we can do". They is a very irresponsible approach that amounts more to guess work than any actual discovery. In the end people get old. "Far to late" is because the person already lived a full life.
epmaybe 8y ago
Firstly, I agree. You can't hide from that, and research should be upfront with the limitations of the work. Other commenters have noted that the sensitivity is 100% with specificity 82% (false positive rate being 18%). I wanted to provide my own take on what that means for me as a future provider (I'm considering becoming a neurologist after medical school). I'm assuming to the data science crowd here, it's almost second nature to know what these numbers really mean in practice, but maybe I can provide some outside perspective. When considering screening tests in medicine, we have to always balance the benefits and risks of conducting these tests on patients. We have to figure out how reliable a test is, and how patients benefit from earlier detection. In this case, it's an imaging test, so noninvasive. However, it's a test that requires radiation (positron emission tomography or PET). So we already have to determine if that extra radiation dose is worth the increased risk for something like cancer. In an elderly population, it's almost certainly worth the scan given the radiation would not significantly impact their decades of accrued radiation damage. Now, what happens if a patient has a positive test, 6 years earlier than they would normally? Let's assume for a moment that it was a true positive test. They could receive treatment earlier, which could improve their quality of life. They could adjust their risk factors to slow down the progression of disease. The treatment could have side effects, such as increased blood pressure. If we apply that to a large population scale, would we increase mortality associated with heart disease by treating patients with suspected Alzheimer's earlier? Lastly, I wanted to touch on the effects of a screening test with a large amount of false positives. I would argue that a screening test should be allowed to have lots of false positives, simply because we can always do other tests (neuropsychiatric evaluation like MOCA score) to improve our accuracy of detection. However, if we do these other tests, we have to again balance the harms with the risks. Mammograms, for instance, could give false positives that could warrant a breast biopsy. In the case of Alzheimer's, a false positive could lead to stress and depression in an elderly population already at higher risk for stress related disease. In short, screening tests are great if there's a way to do something about the disease. As far as I know, early prophylactic treatment (with current medications) of Alzheimer's does improve quality of life, slowing progression of the disease manifestations, with relatively few side effects.
- citrablue 8y ago
  You missed some downsides: anguish, more tests for something that isn't a problem, costs, possibly suicides. You may want to read up more on the downsides to breast cancer screenings for reference.
max76 8y ago
Quick algorithm that identifies all brain scans with Alzheimer's. public bool HasAlzheimer(GlucosePetScan patientScan) { return true; }
woodandsteel 8y ago
Yes, but whatever the false positive rate is, my guess is it will decrease significantly in coming years. For instance, if a patient gets a positive rating, there might be another type of test that would eliminate most of the false positives. It seems to me quite unlikely researchers would not try to make improvements.

Presenting the true positive rate without mentioning false positives seems problematic. I couldn’t find any information about that in the article, and they don’t link the paper / publication.

mtgx 8y ago

It's not problematic. It's on purpose and it's criminally misleading. If "AI companies" had to show their false positive rates, 99% of them would go bankrupt because then nobody would want to buy their bullshit.

samfriedman 8y ago

The paper itself [0] claims 82% specificity and 100% sensitivity. Meaning, 100% of positive cases in the test set were correctly identified, and 82% of negative cases in the test set were correctly identified. That corresponds to a 0% false negative rate and an 18% false positive rate. The test set is 40 imaging studies from 40 patients. [0] https://pubs.rsna.org/doi/pdf/10.1148/radiol.2018180958 https://pubs.rsna.org/doi/pdf/10.1148/radiol.2018180958

SolaceQuantum 8y ago
Wouldn’t 82% of negative cases being correctly identified mean that 18% of negative cases were incorrectly identified as positive(false positive), or in other words it will mistake 18% of healthy people as having Alzheimer’s, but will always identify an Alzheimer’s patient?
- samfriedman 8y ago
  82% true negative rate == 18% false positive rate. Your description is correct. It's likely that those statistics will be somewhat worse when the algorithm is tested on a broader and more diverse test set, which the authors claim to plan to do in the future.
  - ThePhysicist 8y ago
    Thanks for digging up the paper, false positives are probably more acceptable than false negatives in this scenario (wrong positive diagnoses can be rectified with additional tests but a negative diagnosis will often not be confirmed by additional testing).
    - bloak 8y ago
      Both things matter, and you need to consider what the follow-up test might be and what it would cost not just financially but also in terms of pain, lost time, risk of harm and so on. There are medical tests which have such a high rate of false positives that in practice in the absence of other indications you would ignore a positive result, because the only possible follow-up test would be a non-risk-free biopsy, and therefore, for your peace of mind, it's probably better not to do the first test at all.
- ThePhysicist 8y ago
  I think OP initially mixed up the numbers but now it's correct (18 % false positives).
fencepost 8y ago
A false positive rate (currently) that high may preclude this being used as a widely available diagnostic test, but if used within a patient base already showing other signs of problems it might be more acceptable. After all, if you're going to see a memory specialist you're already thinking there's a problem.

pge 8y ago

This is a key point. If the false positive rate is 18% (as noted in other comments), and the rate of Alzheimer's is 36/100k[1], then for every patient the test correctly identifies, there are 500 patients that tested positive but did not have Alzheimers. There needs to be significant improvement before such a test becomes useful. [1] that's mortality rate which I am using as a proxy for incidence rate, though the actual incidence rate is presumably higher (mortality rate from the CDC)

gooseus 8y ago
I'm not good at this (yet), but my understanding is that your prior for just this test would be the likelihood of having Alzheimer's based on the statistical occurrence within the general population. After this test you now have updated priors which may not be good enough to make a certain diagnosis, but wouldn't it still be useful combined with another independent test that has similar rates, since you'd be using these updated priors?
bsder 8y ago
That depends. I suspect that this test would only be used on those with a family history of Alzheimer's or similar. That alters the Bayesian priors quite a bit. In addition, you can monitor the progress of someone who "tests positive" over time and see if that is really true. Part of the issue with Alzheimer's treatments is that we may be intervening far too late. These kinds of tests may help that.
- iseeyoubydesign 8y ago
  what ever , this is not science but gambling with peoples health to declare "look what we can do". They is a very irresponsible approach that amounts more to guess work than any actual discovery. In the end people get old. "Far to late" is because the person already lived a full life.

epmaybe 8y ago

Firstly, I agree. You can't hide from that, and research should be upfront with the limitations of the work. Other commenters have noted that the sensitivity is 100% with specificity 82% (false positive rate being 18%). I wanted to provide my own take on what that means for me as a future provider (I'm considering becoming a neurologist after medical school). I'm assuming to the data science crowd here, it's almost second nature to know what these numbers really mean in practice, but maybe I can provide some outside perspective. When considering screening tests in medicine, we have to always balance the benefits and risks of conducting these tests on patients. We have to figure out how reliable a test is, and how patients benefit from earlier detection. In this case, it's an imaging test, so noninvasive. However, it's a test that requires radiation (positron emission tomography or PET). So we already have to determine if that extra radiation dose is worth the increased risk for something like cancer. In an elderly population, it's almost certainly worth the scan given the radiation would not significantly impact their decades of accrued radiation damage. Now, what happens if a patient has a positive test, 6 years earlier than they would normally? Let's assume for a moment that it was a true positive test. They could receive treatment earlier, which could improve their quality of life. They could adjust their risk factors to slow down the progression of disease. The treatment could have side effects, such as increased blood pressure. If we apply that to a large population scale, would we increase mortality associated with heart disease by treating patients with suspected Alzheimer's earlier? Lastly, I wanted to touch on the effects of a screening test with a large amount of false positives. I would argue that a screening test should be allowed to have lots of false positives, simply because we can always do other tests (neuropsychiatric evaluation like MOCA score) to improve our accuracy of detection. However, if we do these other tests, we have to again balance the harms with the risks. Mammograms, for instance, could give false positives that could warrant a breast biopsy. In the case of Alzheimer's, a false positive could lead to stress and depression in an elderly population already at higher risk for stress related disease. In short, screening tests are great if there's a way to do something about the disease. As far as I know, early prophylactic treatment (with current medications) of Alzheimer's does improve quality of life, slowing progression of the disease manifestations, with relatively few side effects.

citrablue 8y ago
You missed some downsides: anguish, more tests for something that isn't a problem, costs, possibly suicides. You may want to read up more on the downsides to breast cancer screenings for reference.

max76 8y ago

Quick algorithm that identifies all brain scans with Alzheimer's. public bool HasAlzheimer(GlucosePetScan patientScan) { return true; }

woodandsteel 8y ago

Yes, but whatever the false positive rate is, my guess is it will decrease significantly in coming years. For instance, if a patient gets a positive rating, there might be another type of test that would eliminate most of the false positives. It seems to me quite unlikely researchers would not try to make improvements.