3 ms·

Yeah, those are part of the issue. For example on the webpage, there is no discussion about the false negative/positive rate of the system, and no comparison or

Yeah, those are part of the issue. For example on the webpage, there is no discussion about the false negative/positive rate of the system, and no comparison or link to an academic paper that compares the performance of the system to a trained breast radiologist. Does a patient take a negative exam as "I'm good!" and forego actually seeing a doctor, or having the exam read by a radiologist? A positive result is also a challenging situation. I've talked to patients before after diagnosing them, and as you could expect most people are shocked, scared or they want to know what they should do next. In breast radiology specifically (at least in the US) there is a well defined reporting lexicon and classification system. From what I can tell, the system does not use the lexicon or BI-RADS classification. https://www.acr.org/-/media/ACR/Files/RADS/BI-RADS/BIRADS-Reference-Card.pdf https://www.acr.org/-/media/ACR/Files/RADS/BI-RADS/BIRADS-Re... Breast cancer can be very subtle, and can be hard to figure out, especially if a patient has had surgery or other benign abnormalities. https://radiopaedia.org/cases/development-of-dcis?case_id=development-of-dcis&lang=us https://radiopaedia.org/cases/development-of-dcis?case_id=de... Diagnostic and screening images also undergo rigorous QA for the entire system. Does the model have some kind of QA built in? There is a just a small disclaimer at the top that says "we will not store your data on the server. please do not worry about privacy issues". Not exactly a formal or legal binding agreement. In fact, you can't even connect over https. http://mammo.neuralrad.com:5300/upload http://mammo.neuralrad.com:5300/upload These are just a few things that come to mind off the top of my head. Like I said before, I do think that what coolwulf works on and is trying to do is good, and in the long run can help doctors better characterize findings and help patients. But like anything in health care, there are a lot of edge cases and side effects that you have to think about. The stakes are also very high.

generalizations 4y ago
> Does a patient take a negative exam as "I'm good!" and forego actually seeing a doctor, or having the exam read by a radiologist? I wonder how the false positive/false negative rates for this tool compare to that of a trained human. At some point we may reach parity, in which case what's the harm of trusting the automated result?
- themantalope 4y ago
  https://www.rsna.org/news/2022/march/AI-Potential-in-Breast-Screening https://www.rsna.org/news/2022/march/AI-Potential-in-Breast-... EDIT: To be clear, I believe that this is a different group of authors from coolwulf. Pubmed link: https://pubmed.ncbi.nlm.nih.gov/35348377/ https://pubmed.ncbi.nlm.nih.gov/35348377/
- themantalope 4y ago
  Secondary follow up. I don't think that a human reader in the loop will be going away for a long time. Sure, they can be at parity, but does a combined read have better results? This is an active research question. I think what will happen is that over time, human interpreters and AI systems will "co-evolve" in a sense, where people will pick up on where models are wrong and also learn how to use models to understand their own blind spots. These are also active research topics that are in their infancy.
evanmoran 4y ago
I think care with messaging positive/negative rate is the right reasoning for a productized version of this positioned as an alternative to human radiologists. The difference here is that we know the user has a scan of their body and we can be pretty sure they couldn’t have gotten the scan without doctors already being involved. So we don’t have to worry as much about the “Does a patient take a negative exam as ‘I'm good!’”, because in the normal flow of medicine that gets this scan the doctor will check the scan for them either way. The website probably could have better explanations of this, but most likely negatives will already be double checked independently by doctors, and positives will most likely be handled correctly by bringing it to their current current attention to double check the scan or run more tests.
quasarj 4y ago
To be fair, how is a patient going to get a mammogram or xray without having seen a doctor?

generalizations 4y ago

> Does a patient take a negative exam as "I'm good!" and forego actually seeing a doctor, or having the exam read by a radiologist? I wonder how the false positive/false negative rates for this tool compare to that of a trained human. At some point we may reach parity, in which case what's the harm of trusting the automated result?

themantalope 4y ago
https://www.rsna.org/news/2022/march/AI-Potential-in-Breast-Screening https://www.rsna.org/news/2022/march/AI-Potential-in-Breast-... EDIT: To be clear, I believe that this is a different group of authors from coolwulf. Pubmed link: https://pubmed.ncbi.nlm.nih.gov/35348377/ https://pubmed.ncbi.nlm.nih.gov/35348377/
themantalope 4y ago
Secondary follow up. I don't think that a human reader in the loop will be going away for a long time. Sure, they can be at parity, but does a combined read have better results? This is an active research question. I think what will happen is that over time, human interpreters and AI systems will "co-evolve" in a sense, where people will pick up on where models are wrong and also learn how to use models to understand their own blind spots. These are also active research topics that are in their infancy.

evanmoran 4y ago

I think care with messaging positive/negative rate is the right reasoning for a productized version of this positioned as an alternative to human radiologists. The difference here is that we know the user has a scan of their body and we can be pretty sure they couldn’t have gotten the scan without doctors already being involved. So we don’t have to worry as much about the “Does a patient take a negative exam as ‘I'm good!’”, because in the normal flow of medicine that gets this scan the doctor will check the scan for them either way. The website probably could have better explanations of this, but most likely negatives will already be double checked independently by doctors, and positives will most likely be handled correctly by bringing it to their current current attention to double check the scan or run more tests.

quasarj 4y ago

To be fair, how is a patient going to get a mammogram or xray without having seen a doctor?