3 ms·

A quick google shows iPhone apps detect skin cancers with up to 95% accuracy (as of 2020; may have improved a little since), I wonder what human level accuracy

by nomilk 3y ago

A quick google shows iPhone apps detect skin cancers with up to 95% accuracy (as of 2020; may have improved a little since), I wonder what human level accuracy is (i.e. doctor carrying out a visual inspection), because surely it's not 100%? Looks like the accuracy of the FDA-cleared device is similar to the best apps (mid nineties). It could be true that frequent inaccurate checks keeps more people alive than infrequent accurate checks. If so, an inaccurate device (or even iphone app) might be better than a doctor checking infrequently (e.g. every 2 years or so).

bakedoatmeal 3y ago
I don’t have the data for a cost-benefit-analysis but frequent inaccurate checks cause harm as well. The false positive rate is 3%; if all 330+ million Americans tested themselves each year, that’s millions of benign cases that will need to be followed up on which inflate costs and create backlogs.
- nomilk 3y ago
  > The false positive rate is 3%; if all 330+ million Americans tested themselves each year, that’s millions of benign cases that will need to be followed up on which inflate costs and create backlogs. True. I also don't have the cost-benefit. I'm in sunny australia and get skin checks every 2 years, it's a slight PITA tbh, requires a couple of hours off work to commute to the doctor and get it done (the check itself is unobtrusive and usually takes only 10 min, it's getting there and waiting that's time-consuming). Suppose a check via app takes 5 minutes (as opposed to 2 hours), and could alert you to true-positives sooner (possibly 1 year 11 months sooner), that could have a very significant benefit since skin cancers can go from harmless to harmful quickly. The other thing we're missing is the false-positive rate for the human check; I suspect it's non-zero; possibly not trivial either.
_kb 3y ago
It doesn't need to be 'xor' pairing between automated detection or a doctor. It can be both. There's interesting work happening under places like ACEMID [0] on building models that boost detection accuracy. It's a research project so I have no idea what the final process will look like, but dropped in for a scan the other day, was imaged by a rather neat 360 camera rig [1], then had points of interest flagged for review by a dermatologist. I don't think it's a stretch to envision a process that lets clinicians use more accessible/lower cost imaging devices to help focus their checks, particularly when you consider the ability to compare imaging across scans. [0]: https://acemid.centre.uq.edu.au https://acemid.centre.uq.edu.au [1]: https://www.canfieldsci.com/imaging-systems/vectra-wb360-imaging-system/ https://www.canfieldsci.com/imaging-systems/vectra-wb360-ima...
penteract 3y ago
I think that "accuracy" isn't as easy to define as you'd like. There are keratoses which just fall off if left untreated, and these are not considered cancer, but the cells are indistinguishable from certain skin cancers under a microscope. Since it may not be obvious whether something will just fall off if left to itself, and "wait and see" isn't an ideal approach for dealing with something that might be cancer, these often get treated, leaving whether they were 'really' cancer unknowable to modern medicine.
tsimionescu 3y ago
95% accuracy for a diagnostic test is not that impressive. Wikipedia says one in 5 Americans will have a skin cancer in their lives. If each American has 10 moles, and we assume at most 1 is cancerous, that means a test which says "not cancerous" for all moles will have ~98% accuracy. And 10 moles is an underestimation, not to mention that the same mole can be benign at first and develop cancer later, so we should really be looking over the number of moles * checkups, pushing the raw "accuracy" even higher.

by nomilk 3y ago

bakedoatmeal 3y ago

I don’t have the data for a cost-benefit-analysis but frequent inaccurate checks cause harm as well. The false positive rate is 3%; if all 330+ million Americans tested themselves each year, that’s millions of benign cases that will need to be followed up on which inflate costs and create backlogs.

nomilk 3y ago
> The false positive rate is 3%; if all 330+ million Americans tested themselves each year, that’s millions of benign cases that will need to be followed up on which inflate costs and create backlogs. True. I also don't have the cost-benefit. I'm in sunny australia and get skin checks every 2 years, it's a slight PITA tbh, requires a couple of hours off work to commute to the doctor and get it done (the check itself is unobtrusive and usually takes only 10 min, it's getting there and waiting that's time-consuming). Suppose a check via app takes 5 minutes (as opposed to 2 hours), and could alert you to true-positives sooner (possibly 1 year 11 months sooner), that could have a very significant benefit since skin cancers can go from harmless to harmful quickly. The other thing we're missing is the false-positive rate for the human check; I suspect it's non-zero; possibly not trivial either.

_kb 3y ago

It doesn't need to be 'xor' pairing between automated detection or a doctor. It can be both. There's interesting work happening under places like ACEMID [0] on building models that boost detection accuracy. It's a research project so I have no idea what the final process will look like, but dropped in for a scan the other day, was imaged by a rather neat 360 camera rig [1], then had points of interest flagged for review by a dermatologist. I don't think it's a stretch to envision a process that lets clinicians use more accessible/lower cost imaging devices to help focus their checks, particularly when you consider the ability to compare imaging across scans. [0]: https://acemid.centre.uq.edu.au https://acemid.centre.uq.edu.au [1]: https://www.canfieldsci.com/imaging-systems/vectra-wb360-imaging-system/ https://www.canfieldsci.com/imaging-systems/vectra-wb360-ima...

penteract 3y ago

I think that "accuracy" isn't as easy to define as you'd like. There are keratoses which just fall off if left untreated, and these are not considered cancer, but the cells are indistinguishable from certain skin cancers under a microscope. Since it may not be obvious whether something will just fall off if left to itself, and "wait and see" isn't an ideal approach for dealing with something that might be cancer, these often get treated, leaving whether they were 'really' cancer unknowable to modern medicine.

tsimionescu 3y ago

95% accuracy for a diagnostic test is not that impressive. Wikipedia says one in 5 Americans will have a skin cancer in their lives. If each American has 10 moles, and we assume at most 1 is cancerous, that means a test which says "not cancerous" for all moles will have ~98% accuracy. And 10 moles is an underestimation, not to mention that the same mole can be benign at first and develop cancer later, so we should really be looking over the number of moles * checkups, pushing the raw "accuracy" even higher.