4 ms·

Great question. Definitely sensitivity / specificity balance is a crucial topic in AI-assisted diagnosis. I have to admit this model and website for mammography

Great question. Definitely sensitivity / specificity balance is a crucial topic in AI-assisted diagnosis. I have to admit this model and website for mammography was done in 2018 and might not be the leading solution out there. At the moment, if I want to improve the results of my earlier work, I will add additional stage of radiomics model to do false positive reduction, and in the mean time lower the threshold in the first model to increase sensitivity. By doing this, combining the feature extractions of deep learning and the past 20 years of knowledge in medical imaging using Radiomics might give better performance in terms of sensitivity / specificity. Hao

areoform 4y ago
How can we help? Is there some way for us to contact you regarding this?
dekhn 4y ago
You're operating in a really serious area of medicine and I encourage you to takes the comments about false positives (and false negatives) more seriously. It's not really just a matter of making adjustments to a model; it has to be pervasive in the entire process of making reproducible modeels that are used for making decisiojns about humans.
- coolwulf 4y ago
  The website is cleared marked as a breast health awareness tool and not for diagnosis. We are not doing any decision making for humans. However I would like to point out there are quite some FDA approved mammo AI products in the market at the moment.
  - dekhn 4y ago
    So, you think if you put up a website, people won't use it for diagnosis anyway? yes, I know mammo AI products are on the market- those that were approved followed a collection of regulations that I think you are not.
    - endisneigh 4y ago
      Even if people use it for diagnosis, so what? Take the info and if it’s positive to confirm with doctor, if it’s negative but you have reservations go see doctor. It doesn’t really change anything.
      - dekhn 4y ago
        so you're saying the test is worthless in terms of actionability? Why woudl you want to take it then?
        endisneigh 4y ago
        It’s not that it’s worthless, it’s simply another data point to give to a professional.
- ska 4y ago
  I think this comment is perhaps a bit uncharitable. Even if you have done everything you are supposed to do in the process, at the end of the day you are looking at ROC curves or equivalent and trying to understand sensitivity vs. specificity trade offs, and often you have some sort of (indirect) parameters than can move to different points of that tradeoff. This is quite critical in deployment, if you are screening you usually want something different than in diagnosis; as alluded to elsewhere if you raise the work-up rate too much you definitely risk killing more people from biopsy complications than you help with higher sensitivity (it's more complicated than that in practice)
  - dekhn 4y ago
    deploying ML medical diagnostics is like everything else in ML: the ML part is 1% of a much larger "thing", which involves business, legal, and many other concerns far beyond the data analytics. Nothing abotu what I'm saying is uncharitable- in a sense, it's charitable because I'm helping warn a person who is going down a dangerous path to consult more with experts in this field.
    - ska 4y ago
      The charitable response would have been to assume they do have some understanding the broader context, and perhaps raise specific concerns or points of interest. What you did was assume that they were ignorant in potentially dangerous ways, and assert they should do something different. I don't think a reading of the comments/responses (at least at your time of posting) really supported that assumption, especially considering the limitations of the medium. Hence my reply to you, while also detailing the trade offs a tiny bit.
      - dekhn 4y ago
        I read the entire article and I didn't see anything in there that would convince me this author is anything other than an amateur programmer (I can't parse the section about bruker- is the "amateur programmer" also a director at Bruker who ddevelops medical devices full-time"?). Please be assured that I put a fair amount of thought into this - for example, I used to do due diligence for a VC firm evaluating proposals like this all the time and we had to reject most of them because the founders didn't understand the basic rules of deploying medical technology in highly regulated environments like the US. Based on my interactions with the author in the various parts this post, I continue to conclude this individual is lacking core knowledge and wisdom required to execute a project like this successfully at scale.
billiam 4y ago
You will save even more lives if you over-communicate about the need to use your awareness tools (with better false positive reduction) to drive more effective diagnosis.

areoform 4y ago

How can we help? Is there some way for us to contact you regarding this?

dekhn 4y ago

You're operating in a really serious area of medicine and I encourage you to takes the comments about false positives (and false negatives) more seriously. It's not really just a matter of making adjustments to a model; it has to be pervasive in the entire process of making reproducible modeels that are used for making decisiojns about humans.

coolwulf 4y ago
The website is cleared marked as a breast health awareness tool and not for diagnosis. We are not doing any decision making for humans. However I would like to point out there are quite some FDA approved mammo AI products in the market at the moment.
- dekhn 4y ago
  So, you think if you put up a website, people won't use it for diagnosis anyway? yes, I know mammo AI products are on the market- those that were approved followed a collection of regulations that I think you are not.
  - endisneigh 4y ago
    Even if people use it for diagnosis, so what? Take the info and if it’s positive to confirm with doctor, if it’s negative but you have reservations go see doctor. It doesn’t really change anything.
    - dekhn 4y ago
      so you're saying the test is worthless in terms of actionability? Why woudl you want to take it then?
      - endisneigh 4y ago
        It’s not that it’s worthless, it’s simply another data point to give to a professional.
ska 4y ago
I think this comment is perhaps a bit uncharitable. Even if you have done everything you are supposed to do in the process, at the end of the day you are looking at ROC curves or equivalent and trying to understand sensitivity vs. specificity trade offs, and often you have some sort of (indirect) parameters than can move to different points of that tradeoff. This is quite critical in deployment, if you are screening you usually want something different than in diagnosis; as alluded to elsewhere if you raise the work-up rate too much you definitely risk killing more people from biopsy complications than you help with higher sensitivity (it's more complicated than that in practice)
- dekhn 4y ago
  deploying ML medical diagnostics is like everything else in ML: the ML part is 1% of a much larger "thing", which involves business, legal, and many other concerns far beyond the data analytics. Nothing abotu what I'm saying is uncharitable- in a sense, it's charitable because I'm helping warn a person who is going down a dangerous path to consult more with experts in this field.
  - ska 4y ago
    The charitable response would have been to assume they do have some understanding the broader context, and perhaps raise specific concerns or points of interest. What you did was assume that they were ignorant in potentially dangerous ways, and assert they should do something different. I don't think a reading of the comments/responses (at least at your time of posting) really supported that assumption, especially considering the limitations of the medium. Hence my reply to you, while also detailing the trade offs a tiny bit.
    - dekhn 4y ago
      I read the entire article and I didn't see anything in there that would convince me this author is anything other than an amateur programmer (I can't parse the section about bruker- is the "amateur programmer" also a director at Bruker who ddevelops medical devices full-time"?). Please be assured that I put a fair amount of thought into this - for example, I used to do due diligence for a VC firm evaluating proposals like this all the time and we had to reject most of them because the founders didn't understand the basic rules of deploying medical technology in highly regulated environments like the US. Based on my interactions with the author in the various parts this post, I continue to conclude this individual is lacking core knowledge and wisdom required to execute a project like this successfully at scale.

billiam 4y ago

You will save even more lives if you over-communicate about the need to use your awareness tools (with better false positive reduction) to drive more effective diagnosis.