4 ms·

Not at all. The model is quite accurate. In fact, with the distribution of samples that they have a model that predicts all cases as having cancer would also be

Not at all. The model is quite accurate. In fact, with the distribution of samples that they have a model that predicts all cases as having cancer would also be very accurate. It would get 17/21 predictions right. The model lacks precision. I suspect that even with a fairly high cut off point the model would still produce a bevvy of false positive predictions due to that. It might still be useful as a screening step if they can increase the sensitivity further, but you would still rely upon further tests to get a true diagnosis.

dmoy 3y ago
Sure we can be technically more surgical in our terminology, but GP is addressing the usage of 'accuracy' in the news headline. In that context, 'accuracy' is kinda a catchall term for both accuracy and precision.
- ImageXav 3y ago
  It's a bit difficult to say, isn't it? The headline is using the term accuracy correctly, the reader might be ascribing the meaning you are to it, especially if they are non technical. As was the parent comment. My goal in pointing out the difference was not to be snarky. It was to point out the very real statistical consequences. Any model can be accurate on a sufficiently biased dataset, but what matters once a screening test hits the real world are the precision (positive predictive value) and negative predictive value. These are the hurdles that the test will have to pass to see widespread adoption.
  - jacquesm 3y ago
    Exactly. This is the real test and so far we simply do not know the answer. It's a nice first step but the headline is simply not justified. But whether solar power, wind power or cancer it's 99.99% of the time (possibly more nines) far less impressive by the time all of the data is in. And that's fine, but headline writers seem to be stuck in a hype cycle.
- parhamn 3y ago
  Theyre not being surgical in the terminology. Accuracy and precision are the two things that matter for a diagnostic test. High accuracy, low precision = screener.
tetramer 3y ago
It also only looks at HER2 and CA15-3 (aka MUC1) expression - what about breast cancers that don't express either of these? I realize this is an early technology and I think it should continue to be explored, but I would anticipate that if compared head to head with screening mammograms, it would be inferior. For patients with relapsed disease, this kind of technology would be neat to non-invasively re-assess biomarker status but as a screening tool, I find it lacking (and certainly a positive screening will require dedicated imaging and biopsy anyway).
jncfhnb 3y ago
Backwards. The distributions here imply it will be a high precision model with not great recall.

dmoy 3y ago

Sure we can be technically more surgical in our terminology, but GP is addressing the usage of 'accuracy' in the news headline. In that context, 'accuracy' is kinda a catchall term for both accuracy and precision.

ImageXav 3y ago
It's a bit difficult to say, isn't it? The headline is using the term accuracy correctly, the reader might be ascribing the meaning you are to it, especially if they are non technical. As was the parent comment. My goal in pointing out the difference was not to be snarky. It was to point out the very real statistical consequences. Any model can be accurate on a sufficiently biased dataset, but what matters once a screening test hits the real world are the precision (positive predictive value) and negative predictive value. These are the hurdles that the test will have to pass to see widespread adoption.
- jacquesm 3y ago
  Exactly. This is the real test and so far we simply do not know the answer. It's a nice first step but the headline is simply not justified. But whether solar power, wind power or cancer it's 99.99% of the time (possibly more nines) far less impressive by the time all of the data is in. And that's fine, but headline writers seem to be stuck in a hype cycle.
parhamn 3y ago
Theyre not being surgical in the terminology. Accuracy and precision are the two things that matter for a diagnostic test. High accuracy, low precision = screener.

tetramer 3y ago

It also only looks at HER2 and CA15-3 (aka MUC1) expression - what about breast cancers that don't express either of these? I realize this is an early technology and I think it should continue to be explored, but I would anticipate that if compared head to head with screening mammograms, it would be inferior. For patients with relapsed disease, this kind of technology would be neat to non-invasively re-assess biomarker status but as a screening tool, I find it lacking (and certainly a positive screening will require dedicated imaging and biopsy anyway).

jncfhnb 3y ago

Backwards. The distributions here imply it will be a high precision model with not great recall.