3 ms·

99% accuracy in diagnosing benign case This is meaningless. The only thing that matters in this kind of application is false negative rate at some acceptable f

by andy99 2y ago

99% accuracy in diagnosing benign case This is meaningless. The only thing that matters in this kind of application is false negative rate at some acceptable false positive rate. I assume whoever is working on this knows that, so this is mostly a criticism of the article. That said, this is a horrible use of AI.

greazy 2y ago
I went searching for more. The tool is called DERM by Skin Analytics. They have more info on stats: https://skin-analytics.com/ai-pathways/derm-performance/ https://skin-analytics.com/ai-pathways/derm-performance/ A few peer reviewed pubs down the bottom of the page
frereubu 2y ago
What do you mean by this being a "horrible" use of AI? (Although as another commenter has mentioned, this should more properly be called ML).
- jampekka 2y ago
  It's quite easy to correctly classify 100% of benign cases as benign.
  - cmcaleer 2y ago
    If it's so easy, then why do people die from having lesions misdiagnosed as benign? Even if the success rate of the human eye was in the 99.5%+ range, why not have an extra sanity check from an AI model?
    - dijksterhuis 2y ago
      > If it's so easy, then why do people die from having lesions misdiagnosed as benign? You're confusing False Negatives with True Negatives. For Non-Benign (Positive) vs. Benign (Negative) classification: * True Positive Rate (TPR): non-benign classified as non-benign. * False Positive Rate (FPR): benign misclassified as non-benign. * True Negative Rate (TNR): benign classified as benign. * False Negative Rate (FNR): non-benign misclassified as benign. > It's quite easy to correctly classify 100% of benign cases as benign. You can engineer a 100% TNR if you just classify everything as the "benign" negative class. The FNR is going to be 100% too, but that doesn't matter -- you correctly classified 100% of benign cases as benign. > why do people die from having lesions misdiagnosed as benign? Because the FNR is not 0%. FNR is important. You probably want a decent TPR in there as well. And FPR can be very important too, depending on how life-changing/painful/invasive the treatment for a positive case is!
    - OJFord 2y ago
      Because this non-AI function 'correctly diagnoses 100% benign cases as benign': def is_benign(mole: Image) -> bool: return True ...but also misdiagnoses 100% malignant cases.
      - potatoman22 2y ago
        To contextualize, I think the tool in this article correctly diagnoses 97% benign cases as benign but misdiagnosis 22% of malignant cases.
flir 2y ago
Not really. It allows you to order your cases from most-likely to least-likely, and get human eyeballs on the most-likely cases first.
- potatoman22 2y ago
  This is how most ML in healthcare is used. It's a fantasy to think we can give all patients equal attention, so it often makes sense to prioritize those most at risk. Of course no model is perfect, but ML can be very beneficial when used in the right context (i.e. not like United Healthcare).
  - flir 2y ago
    Yup. The word I should have used is "triage".
Spooky23 2y ago
It’s really all about the protocol. AI tends to spot things that doctors don’t, and vice versa. Dermatology is also an area being pillaged by private equity and access is poor for many people. I lost my wife to melanoma. She noticed a lesion within days of it appearing, and a doctor saw it within 48 hours and felt it was benign. My wife didn’t accept it and had a plastic surgeon remove it and biopsy, then had a margin removed by surgical oncologist, the standard of care at the time. It came back as a brain tumor 4 years later and she was gone in 6 months, even with the incredible advancements today. So I’d hold the position strongly that anything that improves overall detection rates and access to care is incredibly important and will save lives. Weeks matter with melanoma. Today with immunotherapy Molly would be fine. But if she hadn’t advocated and gotten the original thing removed, it would have cost her 4 important years.
- tomcam 2y ago
  So sorry, brother.
- asdefghyk 2y ago
  I've been told the only way to be sure if skin cancer or not , is a biopsy. I also have been told ... not skin cancer... but Dr decided to send me to a dermatologist. Is thing he said when say it .. is looks like skin cancer ... lets take a biopsy now ... to check . It was skin cancer ... a BCC
- osn9363739 2y ago
  Dude that sucks mate. I had a melanoma taken off last year. It was dormant (stage 0) but had been there for 10ish years. But reading shit like this reminds me that even though I'm probably fine, all I can do is just live my best life. Hang out with my family. Enjoy the things I enjoy and not think about it too much. (and get my skin checks every 6 months :D)
  - kennyadam 2y ago
    Are your skin checks just visual checks? If so, read my other comment in this thread. It was in UK fyi.
potatoman22 2y ago
It's trendy to say "it's horrible to use AI for this" without giving specific reasons. Some reasons it could be good to use AI here: - this can prioritize urgent patients for the severely overworked doctors - medical error is a leading cause of death, this serves as a second-opinion (97% true-positive rate and 79% true-negative rate) - it can be used as evidence by a nurse or physician advocating for a patient's treatment
degamad 2y ago
> The only thing that matters in this kind of application is false negative rate at some acceptable false positive rate. It sounds like they are inverting the scenario here. The question is not "do you have skin cancer?", it's "can you safely go home without seeing a doctor?". For this new question, we set the acceptable false positive rate to zero (we never want to send a real cancer case home), and determine the false negative rate (we accept that some benign cases will be seen by a doctor). The reason for the interest in identifying benign cases, rather than trying to identify the positive cases, is that it improves the situation for everybody: benign cases identified by AI are sent home almost immediately, everyone else has a shorter waiting time, so benign false negatives can be assessed more quickly by the doctor and given the all clear, and more time is now available for spending with the real cancer cases. The numbers they're citing are 7000 cases with 5% real, so 350 real cancer, 6650 benign. If we can accurately say that 6500 of those benign cases are benign without wasting the doctors' time, then we're down to only 500 people needing to see a doctor, which is a huge improvement for everyone.
NoPicklez 2y ago
I don't think its a horrible use of AI at all, in fact it seems like a fairly good use case. It's a tool that can be used in amongst the current methods to help detect skin cancer, it shouldn't be used at the only method.

by andy99 2y ago

greazy 2y ago

I went searching for more. The tool is called DERM by Skin Analytics. They have more info on stats: https://skin-analytics.com/ai-pathways/derm-performance/ https://skin-analytics.com/ai-pathways/derm-performance/ A few peer reviewed pubs down the bottom of the page

frereubu 2y ago

What do you mean by this being a "horrible" use of AI? (Although as another commenter has mentioned, this should more properly be called ML).

jampekka 2y ago
It's quite easy to correctly classify 100% of benign cases as benign.
- cmcaleer 2y ago
  If it's so easy, then why do people die from having lesions misdiagnosed as benign? Even if the success rate of the human eye was in the 99.5%+ range, why not have an extra sanity check from an AI model?
  - dijksterhuis 2y ago
    > If it's so easy, then why do people die from having lesions misdiagnosed as benign? You're confusing False Negatives with True Negatives. For Non-Benign (Positive) vs. Benign (Negative) classification: * True Positive Rate (TPR): non-benign classified as non-benign. * False Positive Rate (FPR): benign misclassified as non-benign. * True Negative Rate (TNR): benign classified as benign. * False Negative Rate (FNR): non-benign misclassified as benign. > It's quite easy to correctly classify 100% of benign cases as benign. You can engineer a 100% TNR if you just classify everything as the "benign" negative class. The FNR is going to be 100% too, but that doesn't matter -- you correctly classified 100% of benign cases as benign. > why do people die from having lesions misdiagnosed as benign? Because the FNR is not 0%. FNR is important. You probably want a decent TPR in there as well. And FPR can be very important too, depending on how life-changing/painful/invasive the treatment for a positive case is!
  - OJFord 2y ago
    Because this non-AI function 'correctly diagnoses 100% benign cases as benign': def is_benign(mole: Image) -> bool: return True ...but also misdiagnoses 100% malignant cases.
    - potatoman22 2y ago
      To contextualize, I think the tool in this article correctly diagnoses 97% benign cases as benign but misdiagnosis 22% of malignant cases.

flir 2y ago

Not really. It allows you to order your cases from most-likely to least-likely, and get human eyeballs on the most-likely cases first.

potatoman22 2y ago
This is how most ML in healthcare is used. It's a fantasy to think we can give all patients equal attention, so it often makes sense to prioritize those most at risk. Of course no model is perfect, but ML can be very beneficial when used in the right context (i.e. not like United Healthcare).
- flir 2y ago
  Yup. The word I should have used is "triage".

Spooky23 2y ago

It’s really all about the protocol. AI tends to spot things that doctors don’t, and vice versa. Dermatology is also an area being pillaged by private equity and access is poor for many people. I lost my wife to melanoma. She noticed a lesion within days of it appearing, and a doctor saw it within 48 hours and felt it was benign. My wife didn’t accept it and had a plastic surgeon remove it and biopsy, then had a margin removed by surgical oncologist, the standard of care at the time. It came back as a brain tumor 4 years later and she was gone in 6 months, even with the incredible advancements today. So I’d hold the position strongly that anything that improves overall detection rates and access to care is incredibly important and will save lives. Weeks matter with melanoma. Today with immunotherapy Molly would be fine. But if she hadn’t advocated and gotten the original thing removed, it would have cost her 4 important years.

tomcam 2y ago
So sorry, brother.
asdefghyk 2y ago
I've been told the only way to be sure if skin cancer or not , is a biopsy. I also have been told ... not skin cancer... but Dr decided to send me to a dermatologist. Is thing he said when say it .. is looks like skin cancer ... lets take a biopsy now ... to check . It was skin cancer ... a BCC
osn9363739 2y ago
Dude that sucks mate. I had a melanoma taken off last year. It was dormant (stage 0) but had been there for 10ish years. But reading shit like this reminds me that even though I'm probably fine, all I can do is just live my best life. Hang out with my family. Enjoy the things I enjoy and not think about it too much. (and get my skin checks every 6 months :D)
- kennyadam 2y ago
  Are your skin checks just visual checks? If so, read my other comment in this thread. It was in UK fyi.

potatoman22 2y ago

It's trendy to say "it's horrible to use AI for this" without giving specific reasons. Some reasons it could be good to use AI here: - this can prioritize urgent patients for the severely overworked doctors - medical error is a leading cause of death, this serves as a second-opinion (97% true-positive rate and 79% true-negative rate) - it can be used as evidence by a nurse or physician advocating for a patient's treatment

degamad 2y ago

> The only thing that matters in this kind of application is false negative rate at some acceptable false positive rate. It sounds like they are inverting the scenario here. The question is not "do you have skin cancer?", it's "can you safely go home without seeing a doctor?". For this new question, we set the acceptable false positive rate to zero (we never want to send a real cancer case home), and determine the false negative rate (we accept that some benign cases will be seen by a doctor). The reason for the interest in identifying benign cases, rather than trying to identify the positive cases, is that it improves the situation for everybody: benign cases identified by AI are sent home almost immediately, everyone else has a shorter waiting time, so benign false negatives can be assessed more quickly by the doctor and given the all clear, and more time is now available for spending with the real cancer cases. The numbers they're citing are 7000 cases with 5% real, so 350 real cancer, 6650 benign. If we can accurately say that 6500 of those benign cases are benign without wasting the doctors' time, then we're down to only 500 people needing to see a doctor, which is a huge improvement for everyone.

NoPicklez 2y ago

I don't think its a horrible use of AI at all, in fact it seems like a fairly good use case. It's a tool that can be used in amongst the current methods to help detect skin cancer, it shouldn't be used at the only method.