7 ms·

This is very cool work. I'm a radiologist, I also work on developing ML/AI based systems for cancer detection and characterization. Literally just took a break

This is very cool work. I'm a radiologist, I also work on developing ML/AI based systems for cancer detection and characterization. Literally just took a break for a few minutes from creating some labels and saw this as the top HN post! I think in some ways making the model available online can be good, but in other ways could be harmful too. Very complicated topic. 恭喜coolwulf，祝你继续成功。

martincmartin 4y ago
How could it be harmful? Is it just because of errors, i.e. false negative means person won't talk to doctor and won't be caught early; false positive means needless worry, maybe tarnish more general medical industry? Or something else?
- dekhn 4y ago
  The consequences are false positives and negatives in cancer detection are well known (easily googled), and are actually extremely important when looking at cancer treatment in large populations. You nailed the most important ones.
- themantalope 4y ago
  Yeah, those are part of the issue. For example on the webpage, there is no discussion about the false negative/positive rate of the system, and no comparison or link to an academic paper that compares the performance of the system to a trained breast radiologist. Does a patient take a negative exam as "I'm good!" and forego actually seeing a doctor, or having the exam read by a radiologist? A positive result is also a challenging situation. I've talked to patients before after diagnosing them, and as you could expect most people are shocked, scared or they want to know what they should do next. In breast radiology specifically (at least in the US) there is a well defined reporting lexicon and classification system. From what I can tell, the system does not use the lexicon or BI-RADS classification. https://www.acr.org/-/media/ACR/Files/RADS/BI-RADS/BIRADS-Reference-Card.pdf https://www.acr.org/-/media/ACR/Files/RADS/BI-RADS/BIRADS-Re... Breast cancer can be very subtle, and can be hard to figure out, especially if a patient has had surgery or other benign abnormalities. https://radiopaedia.org/cases/development-of-dcis?case_id=development-of-dcis&lang=us https://radiopaedia.org/cases/development-of-dcis?case_id=de... Diagnostic and screening images also undergo rigorous QA for the entire system. Does the model have some kind of QA built in? There is a just a small disclaimer at the top that says "we will not store your data on the server. please do not worry about privacy issues". Not exactly a formal or legal binding agreement. In fact, you can't even connect over https. http://mammo.neuralrad.com:5300/upload http://mammo.neuralrad.com:5300/upload These are just a few things that come to mind off the top of my head. Like I said before, I do think that what coolwulf works on and is trying to do is good, and in the long run can help doctors better characterize findings and help patients. But like anything in health care, there are a lot of edge cases and side effects that you have to think about. The stakes are also very high.
  - generalizations 4y ago
    > Does a patient take a negative exam as "I'm good!" and forego actually seeing a doctor, or having the exam read by a radiologist? I wonder how the false positive/false negative rates for this tool compare to that of a trained human. At some point we may reach parity, in which case what's the harm of trusting the automated result?
    - themantalope 4y ago
      https://www.rsna.org/news/2022/march/AI-Potential-in-Breast-Screening https://www.rsna.org/news/2022/march/AI-Potential-in-Breast-... EDIT: To be clear, I believe that this is a different group of authors from coolwulf. Pubmed link: https://pubmed.ncbi.nlm.nih.gov/35348377/ https://pubmed.ncbi.nlm.nih.gov/35348377/
    - themantalope 4y ago
      Secondary follow up. I don't think that a human reader in the loop will be going away for a long time. Sure, they can be at parity, but does a combined read have better results? This is an active research question. I think what will happen is that over time, human interpreters and AI systems will "co-evolve" in a sense, where people will pick up on where models are wrong and also learn how to use models to understand their own blind spots. These are also active research topics that are in their infancy.
  - evanmoran 4y ago
    I think care with messaging positive/negative rate is the right reasoning for a productized version of this positioned as an alternative to human radiologists. The difference here is that we know the user has a scan of their body and we can be pretty sure they couldn’t have gotten the scan without doctors already being involved. So we don’t have to worry as much about the “Does a patient take a negative exam as ‘I'm good!’”, because in the normal flow of medicine that gets this scan the doctor will check the scan for them either way. The website probably could have better explanations of this, but most likely negatives will already be double checked independently by doctors, and positives will most likely be handled correctly by bringing it to their current current attention to double check the scan or run more tests.
- DantesKite 4y ago
  Well this radiologist could lose their job to an algorithm. That's pretty harmful to the industry.
  - deleted 4y ago
    [deleted]
  - themantalope 4y ago
    I'm not worried. If things are that different, the rest of the economy will be too. The job will also evolve over time, like any other profession that changes with new technology. It's our job (as radiologists and engineers) to shape that in a way that benefits patients.
    - tryptophan 4y ago
      If AI automates radiology, it will automate the rest of the world first.
DantesKite 4y ago
I've always felt the "could be harmful" was a rationalization by radiologists worried about their job security since it's easily mitigated with a warning and multiple tests. And especially because in the future, most radiology work will be done by software. It's just a matter of whether it's 10 years or 100 years from now.
- deleted 4y ago
  [deleted]
- quasarj 4y ago
  Yeah, I've read all the arguments about harm, and I just don't buy it. I'd get a CT scan every year if I could convince someone to sign off on it. There is absolutely no substitute for knowing.
  - themantalope 4y ago
    Until you've had an unnecessary biopsy that kills you :)
  - kashunstva 4y ago
    > I'd get a CT scan every year You would choose annual body CT even if outcomes-based evidence showed no benefit? Even among those at highest risk for lung carcinoma, studies of annual screening chest CT do not uniformly show improved disease-specific survival as compared to an unscreened but otherwise matched cohort.
    - themantalope 4y ago
      https://www.nejm.org/doi/full/10.1056/nejmoa1102873 https://www.nejm.org/doi/full/10.1056/nejmoa1102873 https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3762603/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3762603/ What I found mind blowing when I first read these studies (particularly the second link) is that the positive-predictive value of the CT screen was only 3.8% for lung nodules that were 4mm or larger in size. Basically just means we find all kinds of lung abnormalities all the time that "could be cancer" but aren't. Keep in mind, that the screening criteria was people over age 65 who had a 30+ pack-year smoking history (1 pack-year = smoking 1 pack of cigarettes per day in one year), and had to have smoked within the last 15 years. I can also tell you from doing CT lung biopsies that it's not a procedure to take lightly. Lot of important structures in the chest.
ska 4y ago
>Very complicated topic. This is very true. Data availability (and moreso, label availability) is the biggest barrier to improvement here I suspect [ thanks for labeling!]. Access being another. Using a public site to bootstrap that could do very interesting things. On the other hand, public access to a poorly RA/QA'd algorithm could also cause more trouble than help, easily.
- themantalope 4y ago
  Label availability is a big problem. There are some academic collaborations for some diseases which is making this better, but datasets in general are minuscule compared to what is available for more general computer vision applications. I am hopeful though. Few-shot learning and self supervision are very active questions right now and there are a lot of papers in the medical AI field that are getting published on these topics. I'm personally interested in liver cancer, which does not have large, well-curated and shared database of cases. Sharing data gets tricky, especially in the US. Labels I'm working with and creating (at least at this point) are for my own research, which won't be shared publicly any time soon.
  - ska 4y ago
    > n the medical AI field that are getting published on these topics. Most of what I've seen isn't very promising. The energy in these research areas are because it would be so much cheaper than the "right" way, far more than because of the likelihood of success. And also perversely, because it's hard for the academic researchers to get enough data to do other studies :) NB: I'm not saying there is nothing useful coming out of the learning literature in last few years, just that it a) isn't a silver bullet and b) is often being misapplied in these areas anyway. Label quality and availability isn't the only big problem though. Many data sets exhibit problematic sampling bias, as well as being order(s) of magnitude too small, because of the way they are gathered and how access is granted.
    - themantalope 4y ago
      Problem is though for most of these diseases there just aren't the number of samples available, period, to do it the "right way". HCC for example, has around 50K new cases/year in the US. Even if every single case went into a repository with perfect labels, would still take a long time to collect that info. Not to mention you need either a radiologist (4 year of medical school + 6 years of post-school training) or a very skilled and experienced technician to label the data. Not to mention imaging protocols are not standardized, and the imaging technology is also evolving so scans we do today may not be "correct" or standard in 5-10 years.

martincmartin 4y ago

How could it be harmful? Is it just because of errors, i.e. false negative means person won't talk to doctor and won't be caught early; false positive means needless worry, maybe tarnish more general medical industry? Or something else?

dekhn 4y ago
The consequences are false positives and negatives in cancer detection are well known (easily googled), and are actually extremely important when looking at cancer treatment in large populations. You nailed the most important ones.
themantalope 4y ago
Yeah, those are part of the issue. For example on the webpage, there is no discussion about the false negative/positive rate of the system, and no comparison or link to an academic paper that compares the performance of the system to a trained breast radiologist. Does a patient take a negative exam as "I'm good!" and forego actually seeing a doctor, or having the exam read by a radiologist? A positive result is also a challenging situation. I've talked to patients before after diagnosing them, and as you could expect most people are shocked, scared or they want to know what they should do next. In breast radiology specifically (at least in the US) there is a well defined reporting lexicon and classification system. From what I can tell, the system does not use the lexicon or BI-RADS classification. https://www.acr.org/-/media/ACR/Files/RADS/BI-RADS/BIRADS-Reference-Card.pdf https://www.acr.org/-/media/ACR/Files/RADS/BI-RADS/BIRADS-Re... Breast cancer can be very subtle, and can be hard to figure out, especially if a patient has had surgery or other benign abnormalities. https://radiopaedia.org/cases/development-of-dcis?case_id=development-of-dcis&lang=us https://radiopaedia.org/cases/development-of-dcis?case_id=de... Diagnostic and screening images also undergo rigorous QA for the entire system. Does the model have some kind of QA built in? There is a just a small disclaimer at the top that says "we will not store your data on the server. please do not worry about privacy issues". Not exactly a formal or legal binding agreement. In fact, you can't even connect over https. http://mammo.neuralrad.com:5300/upload http://mammo.neuralrad.com:5300/upload These are just a few things that come to mind off the top of my head. Like I said before, I do think that what coolwulf works on and is trying to do is good, and in the long run can help doctors better characterize findings and help patients. But like anything in health care, there are a lot of edge cases and side effects that you have to think about. The stakes are also very high.
- generalizations 4y ago
  > Does a patient take a negative exam as "I'm good!" and forego actually seeing a doctor, or having the exam read by a radiologist? I wonder how the false positive/false negative rates for this tool compare to that of a trained human. At some point we may reach parity, in which case what's the harm of trusting the automated result?
  - themantalope 4y ago
    https://www.rsna.org/news/2022/march/AI-Potential-in-Breast-Screening https://www.rsna.org/news/2022/march/AI-Potential-in-Breast-... EDIT: To be clear, I believe that this is a different group of authors from coolwulf. Pubmed link: https://pubmed.ncbi.nlm.nih.gov/35348377/ https://pubmed.ncbi.nlm.nih.gov/35348377/
  - themantalope 4y ago
    Secondary follow up. I don't think that a human reader in the loop will be going away for a long time. Sure, they can be at parity, but does a combined read have better results? This is an active research question. I think what will happen is that over time, human interpreters and AI systems will "co-evolve" in a sense, where people will pick up on where models are wrong and also learn how to use models to understand their own blind spots. These are also active research topics that are in their infancy.
- evanmoran 4y ago
  I think care with messaging positive/negative rate is the right reasoning for a productized version of this positioned as an alternative to human radiologists. The difference here is that we know the user has a scan of their body and we can be pretty sure they couldn’t have gotten the scan without doctors already being involved. So we don’t have to worry as much about the “Does a patient take a negative exam as ‘I'm good!’”, because in the normal flow of medicine that gets this scan the doctor will check the scan for them either way. The website probably could have better explanations of this, but most likely negatives will already be double checked independently by doctors, and positives will most likely be handled correctly by bringing it to their current current attention to double check the scan or run more tests.
DantesKite 4y ago
Well this radiologist could lose their job to an algorithm. That's pretty harmful to the industry.
- deleted 4y ago
  [deleted]
- themantalope 4y ago
  I'm not worried. If things are that different, the rest of the economy will be too. The job will also evolve over time, like any other profession that changes with new technology. It's our job (as radiologists and engineers) to shape that in a way that benefits patients.
  - tryptophan 4y ago
    If AI automates radiology, it will automate the rest of the world first.

DantesKite 4y ago

I've always felt the "could be harmful" was a rationalization by radiologists worried about their job security since it's easily mitigated with a warning and multiple tests. And especially because in the future, most radiology work will be done by software. It's just a matter of whether it's 10 years or 100 years from now.

deleted 4y ago
[deleted]
quasarj 4y ago
Yeah, I've read all the arguments about harm, and I just don't buy it. I'd get a CT scan every year if I could convince someone to sign off on it. There is absolutely no substitute for knowing.
- themantalope 4y ago
  Until you've had an unnecessary biopsy that kills you :)
- kashunstva 4y ago
  > I'd get a CT scan every year You would choose annual body CT even if outcomes-based evidence showed no benefit? Even among those at highest risk for lung carcinoma, studies of annual screening chest CT do not uniformly show improved disease-specific survival as compared to an unscreened but otherwise matched cohort.
  - themantalope 4y ago
    https://www.nejm.org/doi/full/10.1056/nejmoa1102873 https://www.nejm.org/doi/full/10.1056/nejmoa1102873 https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3762603/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3762603/ What I found mind blowing when I first read these studies (particularly the second link) is that the positive-predictive value of the CT screen was only 3.8% for lung nodules that were 4mm or larger in size. Basically just means we find all kinds of lung abnormalities all the time that "could be cancer" but aren't. Keep in mind, that the screening criteria was people over age 65 who had a 30+ pack-year smoking history (1 pack-year = smoking 1 pack of cigarettes per day in one year), and had to have smoked within the last 15 years. I can also tell you from doing CT lung biopsies that it's not a procedure to take lightly. Lot of important structures in the chest.

ska 4y ago

>Very complicated topic. This is very true. Data availability (and moreso, label availability) is the biggest barrier to improvement here I suspect [ thanks for labeling!]. Access being another. Using a public site to bootstrap that could do very interesting things. On the other hand, public access to a poorly RA/QA'd algorithm could also cause more trouble than help, easily.

themantalope 4y ago
Label availability is a big problem. There are some academic collaborations for some diseases which is making this better, but datasets in general are minuscule compared to what is available for more general computer vision applications. I am hopeful though. Few-shot learning and self supervision are very active questions right now and there are a lot of papers in the medical AI field that are getting published on these topics. I'm personally interested in liver cancer, which does not have large, well-curated and shared database of cases. Sharing data gets tricky, especially in the US. Labels I'm working with and creating (at least at this point) are for my own research, which won't be shared publicly any time soon.
- ska 4y ago
  > n the medical AI field that are getting published on these topics. Most of what I've seen isn't very promising. The energy in these research areas are because it would be so much cheaper than the "right" way, far more than because of the likelihood of success. And also perversely, because it's hard for the academic researchers to get enough data to do other studies :) NB: I'm not saying there is nothing useful coming out of the learning literature in last few years, just that it a) isn't a silver bullet and b) is often being misapplied in these areas anyway. Label quality and availability isn't the only big problem though. Many data sets exhibit problematic sampling bias, as well as being order(s) of magnitude too small, because of the way they are gathered and how access is granted.
  - themantalope 4y ago
    Problem is though for most of these diseases there just aren't the number of samples available, period, to do it the "right way". HCC for example, has around 50K new cases/year in the US. Even if every single case went into a repository with perfect labels, would still take a long time to collect that info. Not to mention you need either a radiologist (4 year of medical school + 6 years of post-school training) or a very skilled and experienced technician to label the data. Not to mention imaging protocols are not standardized, and the imaging technology is also evolving so scans we do today may not be "correct" or standard in 5-10 years.