5 ms·

Please consider that the researchers reported a 99% detection rate of those particular cancers in normal people, using a urine test. One would be hard-pressed t

Please consider that the researchers reported a 99% detection rate of those particular cancers in normal people, using a urine test. One would be hard-pressed to come up with a less editorialized title, given HN's constraints.

tptacek 4y ago
I didn't read the paper, but in case you did: is it "near perfect accuracy", or is it 99%? The base rate of pancreatic cancer is very low, so a test with 99% specificity is going to overwhelmingly generate false positives.
- nerdponx 4y ago
  The abstract claims that both sensitivity and specificity are high: > The developed platform successfully classified the human prostate and pancreatic cancer urines in a label-free method supported by two types of deep learning networks, with high clinical sensitivity and specificity. I don't have access to the full article so I can't see the numbers. It's not on the hub of science yet either.
  - gwd 4y ago
    It's not a sensitivity / specificity thing; it's a base rate thing [1]. Suppose that the percentage of people with prostate cancer at any one time is 1 in 1000 (i.e., then "base rate" is 1 in 1000). (Turns out this is actually on the same order of magnitude as real number [2].). And suppose this test has 99% sensitivity and 99% specificity. And suppose you test 1,000,000 people. Of those 1,000,000 people, 1,000 will actually have prostate cancer, and 999,000 will not. Of those 1000 that actually have it, 990 will have a positive test (true positive), and 10 will have a negative test (false negative). Of the 999,000 people who don't actually have it, 989,010 will have a negative test (true negative), and 9,990 will have a positive test (false positive). So even with a test of 99% accuracy, if you get a positive result, your chances of actually having prostate cancer are still only 990 / 10980, or about 9%; 91% of the positives will be false positives. And of course, the more rare the cancer, the worse it gets. EDIT2: So, to follow on with GGP's point: "Near perfect accuracy" isn't very specific, but colloquially would imply that if you have a positive test, you have a high chance of actually having cancer. To get that number to 95% you'd need to have only 52 false positives, would require a specificity closer to 99.995%. EDIT: Fixed some math [1] https://en.wikipedia.org/wiki/Base_rate_fallacy https://en.wikipedia.org/wiki/Base_rate_fallacy [2] https://www.cancer.org/cancer/prostate-cancer/about/key-statistics.html https://www.cancer.org/cancer/prostate-cancer/about/key-stat...
    - nerdponx 4y ago
      Right, the "lots of false positives" conclusion is still the correct analysis. I just wanted to point out that both "P(Detect | Positive)" and "P(Not Detect | Negative)" are both high, since GP only mentioned one and not the other.
- cstrahan 4y ago
  Since most people find probability pretty unintuitive, I figured I’d recommend an article that covers the math involved: https://www.untrammeledmind.com/2018/01/cancer-screening-an-easier-counterintuitive-conditional-probability-problem-with-and-without-bayes-theorem/ https://www.untrammeledmind.com/2018/01/cancer-screening-an-...
- hprotagonist 4y ago
  Indeed, Reverend Bayes’ dark side rears its ugly head right at the point that people claim that two nines of accuracy is good in the context of detecting a rare thing in a large sample set, without mentioning precision, recall, AUC, or other more useful metrics.
deleted 4y ago
[deleted]

tptacek 4y ago

I didn't read the paper, but in case you did: is it "near perfect accuracy", or is it 99%? The base rate of pancreatic cancer is very low, so a test with 99% specificity is going to overwhelmingly generate false positives.

nerdponx 4y ago
The abstract claims that both sensitivity and specificity are high: > The developed platform successfully classified the human prostate and pancreatic cancer urines in a label-free method supported by two types of deep learning networks, with high clinical sensitivity and specificity. I don't have access to the full article so I can't see the numbers. It's not on the hub of science yet either.
- gwd 4y ago
  It's not a sensitivity / specificity thing; it's a base rate thing [1]. Suppose that the percentage of people with prostate cancer at any one time is 1 in 1000 (i.e., then "base rate" is 1 in 1000). (Turns out this is actually on the same order of magnitude as real number [2].). And suppose this test has 99% sensitivity and 99% specificity. And suppose you test 1,000,000 people. Of those 1,000,000 people, 1,000 will actually have prostate cancer, and 999,000 will not. Of those 1000 that actually have it, 990 will have a positive test (true positive), and 10 will have a negative test (false negative). Of the 999,000 people who don't actually have it, 989,010 will have a negative test (true negative), and 9,990 will have a positive test (false positive). So even with a test of 99% accuracy, if you get a positive result, your chances of actually having prostate cancer are still only 990 / 10980, or about 9%; 91% of the positives will be false positives. And of course, the more rare the cancer, the worse it gets. EDIT2: So, to follow on with GGP's point: "Near perfect accuracy" isn't very specific, but colloquially would imply that if you have a positive test, you have a high chance of actually having cancer. To get that number to 95% you'd need to have only 52 false positives, would require a specificity closer to 99.995%. EDIT: Fixed some math [1] https://en.wikipedia.org/wiki/Base_rate_fallacy https://en.wikipedia.org/wiki/Base_rate_fallacy [2] https://www.cancer.org/cancer/prostate-cancer/about/key-statistics.html https://www.cancer.org/cancer/prostate-cancer/about/key-stat...
  - nerdponx 4y ago
    Right, the "lots of false positives" conclusion is still the correct analysis. I just wanted to point out that both "P(Detect | Positive)" and "P(Not Detect | Negative)" are both high, since GP only mentioned one and not the other.
cstrahan 4y ago
Since most people find probability pretty unintuitive, I figured I’d recommend an article that covers the math involved: https://www.untrammeledmind.com/2018/01/cancer-screening-an-easier-counterintuitive-conditional-probability-problem-with-and-without-bayes-theorem/ https://www.untrammeledmind.com/2018/01/cancer-screening-an-...
hprotagonist 4y ago
Indeed, Reverend Bayes’ dark side rears its ugly head right at the point that people claim that two nines of accuracy is good in the context of detecting a rare thing in a large sample set, without mentioning precision, recall, AUC, or other more useful metrics.

deleted 4y ago

[deleted]