5 ms·
Previous submission of the paper itself: https://news.ycombinator.com/item?id=28050699 https://news.ycombinator.com/item?id=28050699 We know that various featu
by abrichr 5y ago
Previous submission of the paper itself: https://news.ycombinator.com/item?id=28050699 https://news.ycombinator.com/item?id=28050699
We know that various features visible in medical images correlate with race, eg breast density, bone density, etc. Most likely the network is just learning a classifier on top of these features.
This is trivially verifiable but conspicuously absent from the paper.
- ceejayoz 5y ago> We know that various features visible in medical images correlate with race, eg breast density, bone density, etc. That can still be picked out when pixellated down to 8x8 as illustrated in the article? That seems unlikely. I wonder if the AI is just cheating, as they sometimes inadvertently do. https://techcrunch.com/2018/12/31/this-clever-ai-hid-data-from-its-creators-to-cheat-at-its-appointed-task/ https://techcrunch.com/2018/12/31/this-clever-ai-hid-data-fr... > In some early results, the agent was doing well — suspiciously well. What tipped the team off was that, when the agent reconstructed aerial photographs from its street maps, there were lots of details that didn’t seem to be on the latter at all. For instance, skylights on a roof that were eliminated in the process of creating the street map would magically reappear when they asked the agent to do the reverse process...
- ChrisLovejoy 5y agoI suspect there must be some confounder here - like the positioning used for the CXRs correlating with race, based on the methodology used in a particular region / hospital. Seems the most likely explanation for it still working even when pixellated as 8x8?
- OneEyedRobot 5y agoThat was exactly my thinking. I wouldn't publish anything until the pixellated versions were better understood.
- SpicyLemonZest 5y agoWell, isn’t the point of publishing to get help figuring it out from other researchers in the field? I agree it’s very likely that there’s some kind of explainable trick the AI is using, but there’s no guarantee it’s an easy trick that the authors could have figured out.
- OneEyedRobot 5y agoI'd warm up to that concept if the article was: "We don't know what in the hell is going on here. Here's our source code and data set of x-rays and race. What do you think?" It could be that in the realm of machine learning, most of what is going on is people turning random knobs on a big machine and getting mysterious results. It's the birth of science without understanding.
- SpicyLemonZest 5y agoThat's precisely what the researchers are saying. In the underlying paper, they conclude that "this capability is extremely difficult to isolate or mitigate", call for "further investigation and research into the human-hidden but model-decipherable information", and suggest medical imaging people should "consider the use of deep learning models with extreme caution" until future research produces a better understanding of what's happening.
- deleted 5y ago[deleted]
- OneEyedRobot 5y agoThey always call for 'further investigation'. Looking at this: https://arxiv.org/pdf/2107.10356.pdf https://arxiv.org/pdf/2107.10356.pdf My general impression (no more than that) is a whole bunch of people crowding into a paper. The paper is mostly applying trivial image processing functions and seeing how some software they don't understand is responding. The main aim is pearl-clutching about 'bias' rather than any kind of understanding. God knows what they're going to do when any medical exam includes some kind of deep dive into the patient's genetics. No surprises. It's the nature of the era.
- abrichr 5y agoThat was my initial reaction as well but they validate on separate datasets from training which makes this unlikely. The performance despite degradation may be the same phenomenon that results in adversarial examples that are indistinguishable to human eyes, ie we know that neural nets are highly sensitive to visually imperceptible differences.
- zepto 5y agoThe article covers this.
- hgial 5y agoIt's not "conspicuously absent from the paper". They have a whole group of experiments on this: "Experiments on anatomic and phenotype confounders" and conclude "Race detection is not due to obvious anatomic and phenotype confounder variables."