3 ms·

The false positive rate doesn't depend on the number of true positives in the population. However, as you rightly point out the metric that actually matters for

The false positive rate doesn't depend on the number of true positives in the population. However, as you rightly point out the metric that actually matters for these kinds of screening is the precision/positive predictive value, and that one does.

Beltiras 6y ago
96% specificity. That's really good for a screening test.
- DangerousPie 6y ago
  As the parent comment said - whether or not that's "good" enough for a screening test really depends on the prevalence of the disease it's testing for.
  - deleted 6y ago
    [deleted]
  - bananaface 6y ago
    What kind of prevalance do you need for that to be bad?
    - hn_throwaway_99 6y ago
      The wiki article on positive predictive value has the formula for calculating it based on the sensitivity and specificity of a test, and the prevalence: https://en.m.wikipedia.org/wiki/Positive_and_negative_predictive_values https://en.m.wikipedia.org/wiki/Positive_and_negative_predic...
      - bananaface 6y ago
        Perfect! Thank you.
    - blamestross 6y ago
      Lets whip out Bayes' rule: P(Having cancer given positive) = P(Positive given having cancer) * P(Having Cancer) / P(Having a Positive in general) Lets say 1% true positive rate is a "useless test". 0.01 = 0.88 (Detection rate from paper) * X / 0.05 (inverse of specificity) X = 0.0005 Incident rate of Stomach Cancer (most common) is 27,600 per year in US. The denominator for this fraction is iffy because there are a lot of biases on current detection rates. If we go ahead with "adults in the US" then the rate of cancer is 0.00014 [cite](https://www.cancer.org/cancer/stomach-cancer/about/key-statistics.html#:~:text=Stomach%20cancer%20mostly%20affects%20older,is%20about%201%20in%2095. https://www.cancer.org/cancer/stomach-cancer/about/key-stati...) Basically, if we applied this test to all adults without other diagnostic factors. This would be worse than 1% true positive detection rate. I don't know about how effective intersecting with other diagnostic criteria would be on increasing the odds a patient actually have the disease and it might or might not still be useful after that. This sort of study is mostly bullshit anyway. They took a relatively small pile of data, made a small classifier with high accuracy (maybe overtrained, maybe not) and really can't validate it.
- koheripbal 6y ago
  The issue is that there's no follow-up test, at the moment. The cancer is otherwise undetectable. ...and the test does not determine location, so it's impossible to treat even with a positive test. So from a clinical standpoint it's nearly useless, except to schedule patients for more frequent traditional testing. ...the counter point is that the stress caused to a patient is probably not trivial, even from a physical health perspective. ...but from a scientific standpoint, it's awesome and well worth further study.
  - IshKebab 6y ago
    > except to schedule patients for more frequent traditional testing That sounds very useful to me!
  - psychometry 6y ago
    You actually could potentially determine location by examining mutation patterns, both by looking more closely at methylation and doing further sequencing of cell-free DNA.
  - abcc8 6y ago
    I imagine that there will be a longitudinal trial for this test. Individuals will be tested at set intervals and the test results will be correlated with clinical outcomes. It is possible that enough blood samples have already been collected such these same correlates could be identified retrospectively.
  - mNovak 6y ago
    >> Nearly useless Being able to catch a cancer early on because of increased surveillance is hugely valuable. Part of why cancer is so dangerous is because it often grows/spreads for years before you get any symptoms. If you're in the unlucky cohort of <50 years old and have an asymptomatic cancer, it might go decades before anyone thinks to check. And mind, increased surveillance typically means once a year tests or scans, so it's not so overly onerous. It's possible even that generally being in frequent contact with doctors and more seriously minding your health could have net benefits in preventing other maladies as well.
joshgel 6y ago
Yes, this is right. But, the even more important metric to measure is mortality. Detecting a cancer that is 100% fatal 1 day earlier than traditional tests obviously wouldn't be that impressive. Even if the PPV is very high. It's also very possible that a cancer that is detected accurately 6 months earlier, but that detection still doesn't improve our ability to treat the cancer. In these cases, it might even look like mortality is improved if you measure time from detection to death ("early detection extended survival by 6 months"). But, actually, it didn't change much. So, that's all to say that the "Experimental" in the title should temper expectations considerably, because that has real meaning.
kermitje 6y ago
Ah indeed, I was wrong. You will get approximately 6 false positives for every true positive. Unfortunately I can't edit my comment anymore.
- solinent 6y ago
  The introduction of the test itself may change its effectiveness (reducing it). Cancer is an evolving condition that your immune system fights. Perhaps it's not very useful to characterize the test before it's applied since the test itself may change its efficacy. If you start exercising and dieting in response to the potential diagnosis it'll definitely help. If we had a test that detected everyone and they believed it, we'd improve cancer rates.

Beltiras 6y ago

96% specificity. That's really good for a screening test.

DangerousPie 6y ago
As the parent comment said - whether or not that's "good" enough for a screening test really depends on the prevalence of the disease it's testing for.
- deleted 6y ago
  [deleted]
- bananaface 6y ago
  What kind of prevalance do you need for that to be bad?
  - hn_throwaway_99 6y ago
    The wiki article on positive predictive value has the formula for calculating it based on the sensitivity and specificity of a test, and the prevalence: https://en.m.wikipedia.org/wiki/Positive_and_negative_predictive_values https://en.m.wikipedia.org/wiki/Positive_and_negative_predic...
    - bananaface 6y ago
      Perfect! Thank you.
  - blamestross 6y ago
    Lets whip out Bayes' rule: P(Having cancer given positive) = P(Positive given having cancer) * P(Having Cancer) / P(Having a Positive in general) Lets say 1% true positive rate is a "useless test". 0.01 = 0.88 (Detection rate from paper) * X / 0.05 (inverse of specificity) X = 0.0005 Incident rate of Stomach Cancer (most common) is 27,600 per year in US. The denominator for this fraction is iffy because there are a lot of biases on current detection rates. If we go ahead with "adults in the US" then the rate of cancer is 0.00014 [cite](https://www.cancer.org/cancer/stomach-cancer/about/key-statistics.html#:~:text=Stomach%20cancer%20mostly%20affects%20older,is%20about%201%20in%2095. https://www.cancer.org/cancer/stomach-cancer/about/key-stati...) Basically, if we applied this test to all adults without other diagnostic factors. This would be worse than 1% true positive detection rate. I don't know about how effective intersecting with other diagnostic criteria would be on increasing the odds a patient actually have the disease and it might or might not still be useful after that. This sort of study is mostly bullshit anyway. They took a relatively small pile of data, made a small classifier with high accuracy (maybe overtrained, maybe not) and really can't validate it.
koheripbal 6y ago
The issue is that there's no follow-up test, at the moment. The cancer is otherwise undetectable. ...and the test does not determine location, so it's impossible to treat even with a positive test. So from a clinical standpoint it's nearly useless, except to schedule patients for more frequent traditional testing. ...the counter point is that the stress caused to a patient is probably not trivial, even from a physical health perspective. ...but from a scientific standpoint, it's awesome and well worth further study.
- IshKebab 6y ago
  > except to schedule patients for more frequent traditional testing That sounds very useful to me!
- psychometry 6y ago
  You actually could potentially determine location by examining mutation patterns, both by looking more closely at methylation and doing further sequencing of cell-free DNA.
- abcc8 6y ago
  I imagine that there will be a longitudinal trial for this test. Individuals will be tested at set intervals and the test results will be correlated with clinical outcomes. It is possible that enough blood samples have already been collected such these same correlates could be identified retrospectively.
- mNovak 6y ago
  >> Nearly useless Being able to catch a cancer early on because of increased surveillance is hugely valuable. Part of why cancer is so dangerous is because it often grows/spreads for years before you get any symptoms. If you're in the unlucky cohort of <50 years old and have an asymptomatic cancer, it might go decades before anyone thinks to check. And mind, increased surveillance typically means once a year tests or scans, so it's not so overly onerous. It's possible even that generally being in frequent contact with doctors and more seriously minding your health could have net benefits in preventing other maladies as well.

joshgel 6y ago

Yes, this is right. But, the even more important metric to measure is mortality. Detecting a cancer that is 100% fatal 1 day earlier than traditional tests obviously wouldn't be that impressive. Even if the PPV is very high. It's also very possible that a cancer that is detected accurately 6 months earlier, but that detection still doesn't improve our ability to treat the cancer. In these cases, it might even look like mortality is improved if you measure time from detection to death ("early detection extended survival by 6 months"). But, actually, it didn't change much. So, that's all to say that the "Experimental" in the title should temper expectations considerably, because that has real meaning.

kermitje 6y ago

Ah indeed, I was wrong. You will get approximately 6 false positives for every true positive. Unfortunately I can't edit my comment anymore.

solinent 6y ago
The introduction of the test itself may change its effectiveness (reducing it). Cancer is an evolving condition that your immune system fights. Perhaps it's not very useful to characterize the test before it's applied since the test itself may change its efficacy. If you start exercising and dieting in response to the potential diagnosis it'll definitely help. If we had a test that detected everyone and they believed it, we'd improve cancer rates.