3 ms·

>he’s just pioneered a new, improved test for diagnosing pancreatic cancer that is 90% accurate Without more context, that claim is meaningless. Surely it cann

>he’s just pioneered a new, improved test for diagnosing pancreatic cancer that is 90% accurate Without more context, that claim is meaningless. Surely it cannot mean "this test gets the right answer 90% of the time when used on the general population". Nearly 99.99% of people do not have pancreatic cancer, so if I make a program that prints out the sentence "You don't have pancreatic cancer.", then my "test" would have a 99.99% accuracy rate against the general population.

ShawnBird 14y ago
It must mean that it is 90% accurate in identifying cancer. If your test was given 100 people with PC it would be 100% wrong but this test would be 90% correct. That is how I am reading it anyway.
- happimess 14y ago
  The problem is that you're forced to speculate. Your interpretation sounds pretty reasonable to me, but I don't know any more about it than you do. I'd rather have an unambiguous explanation.
- mistercow 14y ago
  But a low false negative rate is equally unimpressive unless it is combined with a low false positive rate. If I modify my program to always say "You have pancreatic cancer", then it would have a 0% false negative rate, but it would still be a worthless test because it would have a 99.99% false positive rate. Simply knowing the false negative rate is not particularly helpful.
- shasta 14y ago
  Sounds like you want my "You do have pancreatic cancer" program, which is 100% accurate.
- firebones 14y ago
  If the prevalence of PC in the population is 1 in 10,000, then giving this test to 100,000 people would result in 10,000 cancer-free people testing positive for PC, 9 people being screened correctly, and 1 person with PC being missed. The number is meaningless without knowing the false positive rate. Rolling a D10 with 9 sides painted black and one white could also give 90% accuracy as a test too.
blahedo 14y ago
One of the things I teach in a unit on information retrieval is the notion of precision and recall, and tradeoffs between them, and the idea that a single "accuracy" rate is almost never useful for these kinds of identification tasks. I've often thought that such a unit would be a good idea for journalists and people in general; the language of reporting on and talking about a lot of scientific advances (medical in particular) is so damn imprecise about this stuff. Even if they report only one number, they could at least say which one it is. 90% could plausibly be either the precision or the recall here, or even F-measure or some other average of them, but as you point out, it's unlikely to be "accuracy" in the sense of (true positives + true negatives) / N, or it wouldn't be very impressive.

ShawnBird 14y ago

It must mean that it is 90% accurate in identifying cancer. If your test was given 100 people with PC it would be 100% wrong but this test would be 90% correct. That is how I am reading it anyway.

happimess 14y ago
The problem is that you're forced to speculate. Your interpretation sounds pretty reasonable to me, but I don't know any more about it than you do. I'd rather have an unambiguous explanation.
mistercow 14y ago
But a low false negative rate is equally unimpressive unless it is combined with a low false positive rate. If I modify my program to always say "You have pancreatic cancer", then it would have a 0% false negative rate, but it would still be a worthless test because it would have a 99.99% false positive rate. Simply knowing the false negative rate is not particularly helpful.
shasta 14y ago
Sounds like you want my "You do have pancreatic cancer" program, which is 100% accurate.
firebones 14y ago
If the prevalence of PC in the population is 1 in 10,000, then giving this test to 100,000 people would result in 10,000 cancer-free people testing positive for PC, 9 people being screened correctly, and 1 person with PC being missed. The number is meaningless without knowing the false positive rate. Rolling a D10 with 9 sides painted black and one white could also give 90% accuracy as a test too.

blahedo 14y ago

One of the things I teach in a unit on information retrieval is the notion of precision and recall, and tradeoffs between them, and the idea that a single "accuracy" rate is almost never useful for these kinds of identification tasks. I've often thought that such a unit would be a good idea for journalists and people in general; the language of reporting on and talking about a lot of scientific advances (medical in particular) is so damn imprecise about this stuff. Even if they report only one number, they could at least say which one it is. 90% could plausibly be either the precision or the recall here, or even F-measure or some other average of them, but as you point out, it's unlikely to be "accuracy" in the sense of (true positives + true negatives) / N, or it wouldn't be very impressive.