4 ms·

.98 AUC classifying race from chest x-rays (external validation)

17 points by abrichr 5y ago

h2odragon 5y ago
> Even more interesting, no part of the image spectrum was primarily responsible either. We could get rid of all the high-frequency information, and the AI could still recognise race in fairly blurry (non-diagnostic) images. Similarly, and I think this might be the most amazing figure I have ever seen, we could get rid of the low-frequency information to the point that a human can’t even tell the image is still an x-ray, and the model can still predict racial identity just as well as with the original image! Im not totally ignorant of what the models are supposed to be doing, about as much as "the boat should float" makes me a qualified sailor... This doesn't sound like the boat is floating the right way up?
- abrichr 5y ago
  This is a very surprising result, putting it mildly! Edit: Upon reflection, the performance as a function of image degradation is not that surprising given what we know about the sensitivity of neural networks to slight perturbations. My best guess at this point is that while humans can detect other features like breast density/bone density/BMI from the scan, they don't automatically interpret race as a function of these, while of course the NN does. The fact that it does so much better than the direct regression between these features and race (e.g. .55 AUC predicting race from BMI, .54 AUC predicting race from breast density) is initially very surprising. But they don't report the results of a similar experiment using all of these together. I suspect that simply predicting race from BMI + Breast Density + Age + Bone Density + Sex would achieve similar performance.
pvitz 5y ago
I am maybe totally naive, but if this feature is bad, then why do all of these models even include race in the training set? Removing it would also remove the potential bias? How is self-identified race potentially relevant for disease identification? Also, I guess humans don't perform well in this task because nobody is studying x-rays for the purpose of identifying race?
- nightcracker 5y ago
  The point is that if a neural network can predict the relationship when labelled with such accuracy it means this information is unavoidably encoded within the image. The result is that even when race is not labeled that the AI might still be racist by automatically (and unknowingly) learning the race.
  - pvitz 5y ago
    If one only trains the model to correctly identify a disease (e.g. maximising the correct identification and minimising the error), then there is no space for the model to be "racist". I don't question that there is a signal inside the images, but someone has to change the model and training to then maliciously use that information.
ribble 5y ago
this making out "white men" to be evil from the start is getting tiresome. white people are a mere fraction of total people. why are "we" not crying foul that china, south korea, vietnam, etc are not "diversifying"? why are we not crying foul that mexico and south america are not diversifying? why is it only white nations must diversify, and cast the male members of the race as "evil"?
Agingcoder 5y ago
Can someone give an example of why it's bad? I sincerely don't understand. Even though the model may know, it doesn't have to tell the radiologist. Furthermore, isn't this just a symptom of 'we're all sick, but different ethnic/social groups have different bodies/manifestations of diseases'? The only thing I can readily understand to be dangerous is for these models to be trained only on white men, and subsequently used on everyone else, but this is a classic problem (train on X, apply on Y which is actually completely different) , not only in medical science. Edit:I have read the blog post, and paper. I really don't understand the panic/urgency.
- raxxorrax 5y ago
  > it doesn't have to tell the radiologist. While there is this stereotype that radiologists hide in their office while making good amounts of cash for just putting people in tubes, chances are that the readiologist does indeed see the patient at one point.
  - Agingcoder 5y ago
    Well, the blog post seems to state quite explicitly that, by and large, radiologists don't see their patients and only look at images , which kind of undermines the point they are trying to make.
    - mlnewb2 5y ago
      why is this a contradiction? If the radiologist doesn't see the patient, they can't appreciate when the AI might be making decisions based on race rather than clinically relevant info.
      - Agingcoder 5y ago
        I understand that ai training is done without injecting race bias in the model, since radiologists don't know about it. Since the nn is supposed to generalize the training set, I don't understand how it can suddenly become racist - in other words, it may learn race, but I don't understand why it would be biased towards one ethnic group or another. Is my understanding correct? Edit: do you assume that the ai output, inferred through a deep neural network, can be interpreted?

17 points by abrichr 5y ago

> Even more interesting, no part of the image spectrum was primarily responsible either. We could get rid of all the high-frequency information, and the AI could still recognise race in fairly blurry (non-diagnostic) images. Similarly, and I think this might be the most amazing figure I have ever seen, we could get rid of the low-frequency information to the point that a human can’t even tell the image is still an x-ray, and the model can still predict racial identity just as well as with the original image! Im not totally ignorant of what the models are supposed to be doing, about as much as "the boat should float" makes me a qualified sailor... This doesn't sound like the boat is floating the right way up?

abrichr 5y ago
This is a very surprising result, putting it mildly! Edit: Upon reflection, the performance as a function of image degradation is not that surprising given what we know about the sensitivity of neural networks to slight perturbations. My best guess at this point is that while humans can detect other features like breast density/bone density/BMI from the scan, they don't automatically interpret race as a function of these, while of course the NN does. The fact that it does so much better than the direct regression between these features and race (e.g. .55 AUC predicting race from BMI, .54 AUC predicting race from breast density) is initially very surprising. But they don't report the results of a similar experiment using all of these together. I suspect that simply predicting race from BMI + Breast Density + Age + Bone Density + Sex would achieve similar performance.

pvitz 5y ago

I am maybe totally naive, but if this feature is bad, then why do all of these models even include race in the training set? Removing it would also remove the potential bias? How is self-identified race potentially relevant for disease identification? Also, I guess humans don't perform well in this task because nobody is studying x-rays for the purpose of identifying race?

nightcracker 5y ago
The point is that if a neural network can predict the relationship when labelled with such accuracy it means this information is unavoidably encoded within the image. The result is that even when race is not labeled that the AI might still be racist by automatically (and unknowingly) learning the race.
- pvitz 5y ago
  If one only trains the model to correctly identify a disease (e.g. maximising the correct identification and minimising the error), then there is no space for the model to be "racist". I don't question that there is a signal inside the images, but someone has to change the model and training to then maliciously use that information.

ribble 5y ago

this making out "white men" to be evil from the start is getting tiresome. white people are a mere fraction of total people. why are "we" not crying foul that china, south korea, vietnam, etc are not "diversifying"? why are we not crying foul that mexico and south america are not diversifying? why is it only white nations must diversify, and cast the male members of the race as "evil"?

Agingcoder 5y ago

Can someone give an example of why it's bad? I sincerely don't understand. Even though the model may know, it doesn't have to tell the radiologist. Furthermore, isn't this just a symptom of 'we're all sick, but different ethnic/social groups have different bodies/manifestations of diseases'? The only thing I can readily understand to be dangerous is for these models to be trained only on white men, and subsequently used on everyone else, but this is a classic problem (train on X, apply on Y which is actually completely different) , not only in medical science. Edit:I have read the blog post, and paper. I really don't understand the panic/urgency.

raxxorrax 5y ago
> it doesn't have to tell the radiologist. While there is this stereotype that radiologists hide in their office while making good amounts of cash for just putting people in tubes, chances are that the readiologist does indeed see the patient at one point.
- Agingcoder 5y ago
  Well, the blog post seems to state quite explicitly that, by and large, radiologists don't see their patients and only look at images , which kind of undermines the point they are trying to make.
  - mlnewb2 5y ago
    why is this a contradiction? If the radiologist doesn't see the patient, they can't appreciate when the AI might be making decisions based on race rather than clinically relevant info.
    - Agingcoder 5y ago
      I understand that ai training is done without injecting race bias in the model, since radiologists don't know about it. Since the nn is supposed to generalize the training set, I don't understand how it can suddenly become racist - in other words, it may learn race, but I don't understand why it would be biased towards one ethnic group or another. Is my understanding correct? Edit: do you assume that the ai output, inferred through a deep neural network, can be interpreted?