18 ms·

Hospitals Deploy AI Tools to Detect Covid-19 on Chest Scans

75 points by jonbaer 7y ago

6gvONxR4sf7o 7y ago
The article is entirely about papers coming out and company press releases, not about hospitals actually using these tools. As someone in the AI community (who rolls their eyes every time it's called "AI"), I've been frustrated at all the "I trained a covid model with 98% accuracy" self-back-patting. Here's a perfect example: https://www.reddit.com/r/MachineLearning/comments/fni5ow/d_why_is_the_ai_hype_absolutely_bonkers/ https://www.reddit.com/r/MachineLearning/comments/fni5ow/d_w... Now, there's real work happening in the space, but I have yet to see much evidence it's being used in the real world, and when an article about its deployment only talks about its deployment via '“I wouldn’t use this as a primary screening tool, but I would use it for opportunistic screening,” such as flagging suspicious CTs or X-rays on patients who received imaging for unrelated medical reasons, says Lungren,' or talks about actual real live deployment today via 'it's totally being used some places in Korea and Brazil (according to this company's press release on their website),' I'm skeptical about the value it's adding. There's serious work happening, but it doesn't seem to be useful to the real world (yet?), and this headline is 99.9% bullshit.
- rvz 7y ago
  Yes. I'm also very skeptical when it comes to Deep Learning methods in healthcare since it usually involves blackbox methods like this which is of a concern to clinicians and medical professionals. The same concerns can be said for other heavily regulated industries that are there to reduce the risks of abuse, loop-holes, uncertainty or inaccurate results. This sort of news is what risks restarting the AI hype if it is proven to be unreliable or another blackbox solution which will be dismissed by medical professionals.
- joshgel 7y ago
  I spent the weekend in the Emergency Dept of a major NYC hospital. In the past 3 weeks I've probably looked at ~250 Chest X-rays of COVID-19+ or likely patients. (Not getting many CT scans currently). You really don't need AI to figure this out. In about 5 minutes, I could show any reasonably smart non-medical person what to look for and they'd do just as well as me (not a radiologist). They all have it and it all looks similar. What would be more interesting is if they attach outcomes data to this. From my anecdotal review of these patients over very limited follow up time periods, x-ray findings didn't seem to correlate with outcomes (death, intubation, etc) or even the need for supplemental oxygen support. That wouldn't be hype, that would be actionable and that would help us on the front-lines
  - alexandercrohde 7y ago
    Well, I wonder how we get that information out there. Maybe you should make a youtube video for other doctors, and show them 20 example X-rays, and indicate why they are or aren't covid.
alexfromapex 7y ago
I think they would need FDA approval to even use a model since it would be considered a medical device correct?
- tedivm 7y ago
  It depends on whether they have a radiologist review the results for accuracy or not. If they don't then it is a diagnostic tool, and would need approval. If it's just used to put higher priority patients in front of a radiologist sooner, and the radiologist does the diagnosing, then it might not need approval if they can show that the FDA has approved something similar in the past.
tedivm 7y ago
This is cute but kind of useless at the moment, as there's no shortage of radiologists in the field right now. If anything it's the opposite- since no one is doing elective procedures most stand alone imaging centers have closed or cut back hours. Some radiology groups are even furloughing their staff because there isn't enough work to justify having them on.
doctoring 7y ago
They say "Hospitals deploy" but where? how? who? I haven't heard any from colleagues in the US or China, even from people who do medical AI work. One major role for AI that I see here is in prognostication. Are there features within radiographs or CT scans that could predict disease severity? Maybe ones that radiologists & clinicians can't discern yet? This would help triage care in capacity-constrained settings. But as the article says, that's "in the future". I haven't seen any robust research in this area yet, and a useful prognostic model would probably incorporate more than just imaging -- symptomology, vital signs, etc. For screening, on the other hand, there are some opportunities but also challenges. When PCR tests/materials/reagents are limited and caseload is high, CT scanning can be a powerful tool to screen and differentiate "sick" from "probably not sick", as was done in Wuhan and Northern Italy. But the AI doesn't offer a whole lot -- the findings are not particularly subtle. Amusingly, from the article: "Teams in China and the United States found that the lungs of patients with COVID-19 symptoms had certain visual hallmarks, such as ground-glass opacities... and areas of increased lung density called consolidation." This is repeated in every AI article on COVID imaging. Like, this is literally Radiology 101 -- any radiologist and many non-radiologist physicians would have been able to say this before COVID existed. Viral pneumonias (SARS, influenza, COVID) and other things (edema, atypical pneumonias, drug reactions) all cause ground-glass opacities. Consolidations happen when the lung gets socked in or super-infected (i.e. with bacteria). Another issue is that early lung infection does not appear on CT, which is common for almost all pneumonias. So CT misses early and especially early mild infections, which is not ideal for screening. And finally, interpretation capacity (what AI could help with) is not a bottleneck. There are TONS of radiologists who are underutilized right now as elective imaging and procedures are way down (e.g. not many screening mammograms or sports injury MRIs being done these days) and on top of that it would not take a radiologist very long to say "this is probably not COVID" or "this could be COVID" or "this is really bad lung disease". Even with lung scans being fast and if we used every CT scanner on them, AI wouldn't really help increase our throughput interpretation-wise. Of course, I write from the perspective of a highly-resourced health system. In the developing world, radiology access, like medical care in general, has always been an issue. I think Kenya as a country has like 100-200 radiologists for what, 40 million people? While a single mid-size hospital in the U.S. could have 200 radiologists. And Kenya's on the better side. So there AI could play an huge role in providing care (and certainly not just in this pandemic!).
segmondy 7y ago
old news, China did this and shared it in Feb https://www.wired.com/story/chinese-hospitals-deploy-ai-help-diagnose-covid-19/ https://www.wired.com/story/chinese-hospitals-deploy-ai-help...
heinrichf 7y ago
See this blog post from a radiologist and ML researcher: https://lukeoakdenrayner.wordpress.com/2020/03/23/ct-scanning-is-just-awful-for-diagnosing-covid-19/ https://lukeoakdenrayner.wordpress.com/2020/03/23/ct-scannin...

75 points by jonbaer 7y ago

The article is entirely about papers coming out and company press releases, not about hospitals actually using these tools. As someone in the AI community (who rolls their eyes every time it's called "AI"), I've been frustrated at all the "I trained a covid model with 98% accuracy" self-back-patting. Here's a perfect example: https://www.reddit.com/r/MachineLearning/comments/fni5ow/d_why_is_the_ai_hype_absolutely_bonkers/ https://www.reddit.com/r/MachineLearning/comments/fni5ow/d_w... Now, there's real work happening in the space, but I have yet to see much evidence it's being used in the real world, and when an article about its deployment only talks about its deployment via '“I wouldn’t use this as a primary screening tool, but I would use it for opportunistic screening,” such as flagging suspicious CTs or X-rays on patients who received imaging for unrelated medical reasons, says Lungren,' or talks about actual real live deployment today via 'it's totally being used some places in Korea and Brazil (according to this company's press release on their website),' I'm skeptical about the value it's adding. There's serious work happening, but it doesn't seem to be useful to the real world (yet?), and this headline is 99.9% bullshit.

rvz 7y ago
Yes. I'm also very skeptical when it comes to Deep Learning methods in healthcare since it usually involves blackbox methods like this which is of a concern to clinicians and medical professionals. The same concerns can be said for other heavily regulated industries that are there to reduce the risks of abuse, loop-holes, uncertainty or inaccurate results. This sort of news is what risks restarting the AI hype if it is proven to be unreliable or another blackbox solution which will be dismissed by medical professionals.
joshgel 7y ago
I spent the weekend in the Emergency Dept of a major NYC hospital. In the past 3 weeks I've probably looked at ~250 Chest X-rays of COVID-19+ or likely patients. (Not getting many CT scans currently). You really don't need AI to figure this out. In about 5 minutes, I could show any reasonably smart non-medical person what to look for and they'd do just as well as me (not a radiologist). They all have it and it all looks similar. What would be more interesting is if they attach outcomes data to this. From my anecdotal review of these patients over very limited follow up time periods, x-ray findings didn't seem to correlate with outcomes (death, intubation, etc) or even the need for supplemental oxygen support. That wouldn't be hype, that would be actionable and that would help us on the front-lines
- alexandercrohde 7y ago
  Well, I wonder how we get that information out there. Maybe you should make a youtube video for other doctors, and show them 20 example X-rays, and indicate why they are or aren't covid.

alexfromapex 7y ago

I think they would need FDA approval to even use a model since it would be considered a medical device correct?

tedivm 7y ago
It depends on whether they have a radiologist review the results for accuracy or not. If they don't then it is a diagnostic tool, and would need approval. If it's just used to put higher priority patients in front of a radiologist sooner, and the radiologist does the diagnosing, then it might not need approval if they can show that the FDA has approved something similar in the past.

tedivm 7y ago

This is cute but kind of useless at the moment, as there's no shortage of radiologists in the field right now. If anything it's the opposite- since no one is doing elective procedures most stand alone imaging centers have closed or cut back hours. Some radiology groups are even furloughing their staff because there isn't enough work to justify having them on.

doctoring 7y ago

They say "Hospitals deploy" but where? how? who? I haven't heard any from colleagues in the US or China, even from people who do medical AI work. One major role for AI that I see here is in prognostication. Are there features within radiographs or CT scans that could predict disease severity? Maybe ones that radiologists & clinicians can't discern yet? This would help triage care in capacity-constrained settings. But as the article says, that's "in the future". I haven't seen any robust research in this area yet, and a useful prognostic model would probably incorporate more than just imaging -- symptomology, vital signs, etc. For screening, on the other hand, there are some opportunities but also challenges. When PCR tests/materials/reagents are limited and caseload is high, CT scanning can be a powerful tool to screen and differentiate "sick" from "probably not sick", as was done in Wuhan and Northern Italy. But the AI doesn't offer a whole lot -- the findings are not particularly subtle. Amusingly, from the article: "Teams in China and the United States found that the lungs of patients with COVID-19 symptoms had certain visual hallmarks, such as ground-glass opacities... and areas of increased lung density called consolidation." This is repeated in every AI article on COVID imaging. Like, this is literally Radiology 101 -- any radiologist and many non-radiologist physicians would have been able to say this before COVID existed. Viral pneumonias (SARS, influenza, COVID) and other things (edema, atypical pneumonias, drug reactions) all cause ground-glass opacities. Consolidations happen when the lung gets socked in or super-infected (i.e. with bacteria). Another issue is that early lung infection does not appear on CT, which is common for almost all pneumonias. So CT misses early and especially early mild infections, which is not ideal for screening. And finally, interpretation capacity (what AI could help with) is not a bottleneck. There are TONS of radiologists who are underutilized right now as elective imaging and procedures are way down (e.g. not many screening mammograms or sports injury MRIs being done these days) and on top of that it would not take a radiologist very long to say "this is probably not COVID" or "this could be COVID" or "this is really bad lung disease". Even with lung scans being fast and if we used every CT scanner on them, AI wouldn't really help increase our throughput interpretation-wise. Of course, I write from the perspective of a highly-resourced health system. In the developing world, radiology access, like medical care in general, has always been an issue. I think Kenya as a country has like 100-200 radiologists for what, 40 million people? While a single mid-size hospital in the U.S. could have 200 radiologists. And Kenya's on the better side. So there AI could play an huge role in providing care (and certainly not just in this pandemic!).

segmondy 7y ago

old news, China did this and shared it in Feb https://www.wired.com/story/chinese-hospitals-deploy-ai-help-diagnose-covid-19/ https://www.wired.com/story/chinese-hospitals-deploy-ai-help...

heinrichf 7y ago

See this blog post from a radiologist and ML researcher: https://lukeoakdenrayner.wordpress.com/2020/03/23/ct-scanning-is-just-awful-for-diagnosing-covid-19/ https://lukeoakdenrayner.wordpress.com/2020/03/23/ct-scannin...