3 ms·

I was a bit confused by exactly what the percentage in the headline was, so here: >> In two of the largest retrospective cohort studies of AI to replace radiol

by macksd 5y ago

I was a bit confused by exactly what the percentage in the headline was, so here: >> In two of the largest retrospective cohort studies of AI to replace radiologists in Europe (n=76 813 women), all AI systems were less accurate than consensus of two radiologists, and 34 of 36 AI systems were less accurate than a single reader I'm still a little unclear how you measure the accuracy of a consensus of radiologists, without relying on the consensus of other radiologists. Maybe just a bigger consensus, or by using known end results but looking at imagery that could have been an early warning?

Areading314 5y ago
The methodology in this paper compared the results of the radiology report to a follow up biopsy.
michaelmrose 5y ago
You could simply see if the person had actually gotten cancer with the benefit of perspective.
honksillet 5y ago
Well, to train the AI you would be using radiologist reads anyways so I'm not sure that's really a problems.
- cratermoon 5y ago
  Oh yes, just that easy! Here, AI, he's some radiologist reads. Learn them, we're going live in Q1 2022.
matchagaucho 5y ago
Or inversely, can this be interpreted as 2 of 36 AI systems are better than humans? Therefore Hospitals should be investing in those 2 systems?
- dbt00 5y ago
  Only if you started with a hypothesis that those two AIs were the best. If you did not, then you're just p hacking.
- cratermoon 5y ago
  Or that 2 of those systems got lucky, none of them actually work for their stated purpose, and investing in radiology AI is about as wise as investing in the Keely Motor.
- pessimizer 5y ago
  Unless the next time you test, it's a different 2.
- macksd 5y ago
  Not necessarily. If you were running a casino and someone was able to significantly beat the odds on 34 of your 36 tables, would you assume they cheated at all other tables but not those two, or just that other players at those tables were really good? It could be that 2 of those systems are actually better. It could also be that the state of the art in deep learning just isn't there yet for radiology. But this is a strong suggestion that research is being dramatically overstated, so I would get real careful with my investments in this area overall.
dang 5y ago
We've since changed the title. The submitted title ("94% of 36 AI systems evaluated were less accurate than a single radiologist") broke the site guidelines by editorializing. Submitters: please don't do that. "Please use the original title, unless it is misleading or linkbait; don't editorialize." https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html

by macksd 5y ago

Areading314 5y ago

The methodology in this paper compared the results of the radiology report to a follow up biopsy.

michaelmrose 5y ago

You could simply see if the person had actually gotten cancer with the benefit of perspective.

honksillet 5y ago

Well, to train the AI you would be using radiologist reads anyways so I'm not sure that's really a problems.

cratermoon 5y ago
Oh yes, just that easy! Here, AI, he's some radiologist reads. Learn them, we're going live in Q1 2022.

matchagaucho 5y ago

Or inversely, can this be interpreted as 2 of 36 AI systems are better than humans? Therefore Hospitals should be investing in those 2 systems?

dbt00 5y ago
Only if you started with a hypothesis that those two AIs were the best. If you did not, then you're just p hacking.
cratermoon 5y ago
Or that 2 of those systems got lucky, none of them actually work for their stated purpose, and investing in radiology AI is about as wise as investing in the Keely Motor.
pessimizer 5y ago
Unless the next time you test, it's a different 2.
macksd 5y ago
Not necessarily. If you were running a casino and someone was able to significantly beat the odds on 34 of your 36 tables, would you assume they cheated at all other tables but not those two, or just that other players at those tables were really good? It could be that 2 of those systems are actually better. It could also be that the state of the art in deep learning just isn't there yet for radiology. But this is a strong suggestion that research is being dramatically overstated, so I would get real careful with my investments in this area overall.

dang 5y ago

We've since changed the title. The submitted title ("94% of 36 AI systems evaluated were less accurate than a single radiologist") broke the site guidelines by editorializing. Submitters: please don't do that. "Please use the original title, unless it is misleading or linkbait; don't editorialize." https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html