6 ms·

The reason I made this project 100% free and available to the general public is to help patients, especially in the remote area who has limited access to experi

The reason I made this project 100% free and available to the general public is to help patients, especially in the remote area who has limited access to experienced radiologists for diagnosis, to at least get a second opinion on their mammogram. And I think this has certain value and this is why I'm doing this project. Hao

ska 4y ago
Inconsistent care is a really good point. I wasn't trying to be negative - hope it didn't come across that way. I was hoping to point out that systemic issues in health care management, at least in a lot of countries, seems to be more of a problem that tech for things like this. Out of curiosity, how are you handling the data access and labelling issues here? I suspect that's the key issue that has limited the performance of the commercial offerings (hardly limited to this problem or this space). OTOH in terms of real impact, properly leveraging a more modestly successful algorithm will probably help more people than getting a few more %. With the (strong) caveat that in a space like this you really have to look at work-up rate and balance risks.
monkeydust 4y ago
There is history of breast cancer in my family and anything that can be done to improve outcomes has to be highly commended. I did however have the same question and this > to at least get a second opinion on their mammogram. for me makes a lot of sense, even in developed countries where you get a result but want extra assurances. It would be interesting to know (assuming you have the data, even anecdotally) if the second opinions using this overturned professional ones and from those how many were corrected an original false negative mistake.
- ska 4y ago
  Not speaking for coolwulf obviously but I can perhaps shed some light. Screening breast mammo has an occurrence rate problme. Something like less than 10 in 1000 studies will require further review; this means in practice as a radiologist you look at a lot of negative films before seeing a TP. It also means a typical read is done fast. Seconds-to-small minutes. This results in a couple of things. Reader variability based on experience/throughput, and false negatives. There were some double reader studies that caught something like 15% (going from memory here) of FN - but nobody can affort to have two radiologists read everything. So the profession is already conceptually used to the idea of using an algorithm as a "second read" and reconsidering. Typically this won't "overturn" anything here but rather say 'hey have another look', but the decision to proceed or not is still the clinicians. Having a positive from the algorithm makes them review carefully, but you have to watch the FP rate here or nobody would get anything else done. I have heard of health systems using algorithms as a first pass too (i.e. radiologist only see films that have had a postive in a tuned-to-be-senstive version), but that has it's own set of issues.
hn_throwaway_99 4y ago
Great project and really cool to see this. Question I have for you is that one of the biggest problems with cancer diagnoses is false positives: "Yes, there is something on your scan, we're not sure what it is, so we'll biopsy it." Biopsy is not a 0-risk procedure, and it can cause a lot of worry and pain, so it's not something to be taken lightly. Also, there are many cases of "OK, it's probably cancer, but the cure may be worse than the disease." This is the classic problem with detecting prostate cancer at an advanced age - it's very likely/probable something else will kill you before the cancer does. How does your software deal with this issue? I'd be worried if, as you put it, people in a remote area with limited access to experienced radiologists, were given access to this and it came back with "Fairly decent chance of cancer" - what do they do then?
- coolwulf 4y ago
  Great question. Definitely sensitivity / specificity balance is a crucial topic in AI-assisted diagnosis. I have to admit this model and website for mammography was done in 2018 and might not be the leading solution out there. At the moment, if I want to improve the results of my earlier work, I will add additional stage of radiomics model to do false positive reduction, and in the mean time lower the threshold in the first model to increase sensitivity. By doing this, combining the feature extractions of deep learning and the past 20 years of knowledge in medical imaging using Radiomics might give better performance in terms of sensitivity / specificity. Hao
  - areoform 4y ago
    How can we help? Is there some way for us to contact you regarding this?
  - dekhn 4y ago
    You're operating in a really serious area of medicine and I encourage you to takes the comments about false positives (and false negatives) more seriously. It's not really just a matter of making adjustments to a model; it has to be pervasive in the entire process of making reproducible modeels that are used for making decisiojns about humans.

ska 4y ago

Inconsistent care is a really good point. I wasn't trying to be negative - hope it didn't come across that way. I was hoping to point out that systemic issues in health care management, at least in a lot of countries, seems to be more of a problem that tech for things like this. Out of curiosity, how are you handling the data access and labelling issues here? I suspect that's the key issue that has limited the performance of the commercial offerings (hardly limited to this problem or this space). OTOH in terms of real impact, properly leveraging a more modestly successful algorithm will probably help more people than getting a few more %. With the (strong) caveat that in a space like this you really have to look at work-up rate and balance risks.

monkeydust 4y ago

There is history of breast cancer in my family and anything that can be done to improve outcomes has to be highly commended. I did however have the same question and this > to at least get a second opinion on their mammogram. for me makes a lot of sense, even in developed countries where you get a result but want extra assurances. It would be interesting to know (assuming you have the data, even anecdotally) if the second opinions using this overturned professional ones and from those how many were corrected an original false negative mistake.

ska 4y ago
Not speaking for coolwulf obviously but I can perhaps shed some light. Screening breast mammo has an occurrence rate problme. Something like less than 10 in 1000 studies will require further review; this means in practice as a radiologist you look at a lot of negative films before seeing a TP. It also means a typical read is done fast. Seconds-to-small minutes. This results in a couple of things. Reader variability based on experience/throughput, and false negatives. There were some double reader studies that caught something like 15% (going from memory here) of FN - but nobody can affort to have two radiologists read everything. So the profession is already conceptually used to the idea of using an algorithm as a "second read" and reconsidering. Typically this won't "overturn" anything here but rather say 'hey have another look', but the decision to proceed or not is still the clinicians. Having a positive from the algorithm makes them review carefully, but you have to watch the FP rate here or nobody would get anything else done. I have heard of health systems using algorithms as a first pass too (i.e. radiologist only see films that have had a postive in a tuned-to-be-senstive version), but that has it's own set of issues.

hn_throwaway_99 4y ago

Great project and really cool to see this. Question I have for you is that one of the biggest problems with cancer diagnoses is false positives: "Yes, there is something on your scan, we're not sure what it is, so we'll biopsy it." Biopsy is not a 0-risk procedure, and it can cause a lot of worry and pain, so it's not something to be taken lightly. Also, there are many cases of "OK, it's probably cancer, but the cure may be worse than the disease." This is the classic problem with detecting prostate cancer at an advanced age - it's very likely/probable something else will kill you before the cancer does. How does your software deal with this issue? I'd be worried if, as you put it, people in a remote area with limited access to experienced radiologists, were given access to this and it came back with "Fairly decent chance of cancer" - what do they do then?

coolwulf 4y ago
Great question. Definitely sensitivity / specificity balance is a crucial topic in AI-assisted diagnosis. I have to admit this model and website for mammography was done in 2018 and might not be the leading solution out there. At the moment, if I want to improve the results of my earlier work, I will add additional stage of radiomics model to do false positive reduction, and in the mean time lower the threshold in the first model to increase sensitivity. By doing this, combining the feature extractions of deep learning and the past 20 years of knowledge in medical imaging using Radiomics might give better performance in terms of sensitivity / specificity. Hao
- areoform 4y ago
  How can we help? Is there some way for us to contact you regarding this?
- dekhn 4y ago
  You're operating in a really serious area of medicine and I encourage you to takes the comments about false positives (and false negatives) more seriously. It's not really just a matter of making adjustments to a model; it has to be pervasive in the entire process of making reproducible modeels that are used for making decisiojns about humans.