4 ms·

Can someone give an example of why it's bad? I sincerely don't understand. Even though the model may know, it doesn't have to tell the radiologist. Furthermor

Can someone give an example of why it's bad? I sincerely don't understand. Even though the model may know, it doesn't have to tell the radiologist. Furthermore, isn't this just a symptom of 'we're all sick, but different ethnic/social groups have different bodies/manifestations of diseases'? The only thing I can readily understand to be dangerous is for these models to be trained only on white men, and subsequently used on everyone else, but this is a classic problem (train on X, apply on Y which is actually completely different) , not only in medical science. Edit:I have read the blog post, and paper. I really don't understand the panic/urgency.

raxxorrax 5y ago
> it doesn't have to tell the radiologist. While there is this stereotype that radiologists hide in their office while making good amounts of cash for just putting people in tubes, chances are that the readiologist does indeed see the patient at one point.
- Agingcoder 5y ago
  Well, the blog post seems to state quite explicitly that, by and large, radiologists don't see their patients and only look at images , which kind of undermines the point they are trying to make.
  - mlnewb2 5y ago
    why is this a contradiction? If the radiologist doesn't see the patient, they can't appreciate when the AI might be making decisions based on race rather than clinically relevant info.
    - Agingcoder 5y ago
      I understand that ai training is done without injecting race bias in the model, since radiologists don't know about it. Since the nn is supposed to generalize the training set, I don't understand how it can suddenly become racist - in other words, it may learn race, but I don't understand why it would be biased towards one ethnic group or another. Is my understanding correct? Edit: do you assume that the ai output, inferred through a deep neural network, can be interpreted?
      - lukeor 5y ago
        Hi, author here. I'll try to answer here with two points. 1) race is not provided explicitly or intentionally during training, but medical practice is biased so it is reasonable to assume there is a signal in the data. 2) we know there must be a signal, since AI models learn it. The optimisation process should only discover features that correlate with the labels, but we see the ability to predict race in models trained to look for diseases like pneumonia (which do not appear different for people from different racial groups).
        Agingcoder 5y ago
        Thanks a lot for taking the time to reply. I have to admit that I have found your post unexpectedly super interesting. It's quite subtle, and while I can see all the facts, there are a few logical steps which don't quite make sense to me. So you're saying that 1. Unlike what the blog post says, there is practice bias even in radiology. This would indeed explain why the model can learn 'racial bias'. 2. This is less clear to me. Just like humans can't see race on radio images, humans might be unable to see differences for diseases like pneumonia, but the nn could see them, no? In other words, how do you know that the differences have to come from hidden racial bias, and not from hidden pathological differences (that you don't know about, just like you've just discovered hidden racial bias) ?
        Agingcoder 5y ago
        (replying to myself about 2) You say the race can't be seen in the image by humans because it's not here, and therefore the race information must come from bias introduced by the radiologist. I guess it depends on whether all differences are visible to the human eye (and there's nothing pathological hidden that the nn 'sees') , and whether you can prove (a very hard thing in stats) that there's no way you can possibly extract race in a different manner. Now, assuming the data is racially biased because of medical practice biased, forgive me for being naive, but, why is this surprising / major? Isn't this just another instance of models being trained on bad data, and there are already plenty of examples in ia ? Edit:.. Unless the actual finding becomes 'radiology is (unexpectedly) racially biased, so much so that an ia can learn it'?

raxxorrax 5y ago

> it doesn't have to tell the radiologist. While there is this stereotype that radiologists hide in their office while making good amounts of cash for just putting people in tubes, chances are that the readiologist does indeed see the patient at one point.

Agingcoder 5y ago
Well, the blog post seems to state quite explicitly that, by and large, radiologists don't see their patients and only look at images , which kind of undermines the point they are trying to make.
- mlnewb2 5y ago
  why is this a contradiction? If the radiologist doesn't see the patient, they can't appreciate when the AI might be making decisions based on race rather than clinically relevant info.
  - Agingcoder 5y ago
    I understand that ai training is done without injecting race bias in the model, since radiologists don't know about it. Since the nn is supposed to generalize the training set, I don't understand how it can suddenly become racist - in other words, it may learn race, but I don't understand why it would be biased towards one ethnic group or another. Is my understanding correct? Edit: do you assume that the ai output, inferred through a deep neural network, can be interpreted?
    - lukeor 5y ago
      Hi, author here. I'll try to answer here with two points. 1) race is not provided explicitly or intentionally during training, but medical practice is biased so it is reasonable to assume there is a signal in the data. 2) we know there must be a signal, since AI models learn it. The optimisation process should only discover features that correlate with the labels, but we see the ability to predict race in models trained to look for diseases like pneumonia (which do not appear different for people from different racial groups).
      - Agingcoder 5y ago
        Thanks a lot for taking the time to reply. I have to admit that I have found your post unexpectedly super interesting. It's quite subtle, and while I can see all the facts, there are a few logical steps which don't quite make sense to me. So you're saying that 1. Unlike what the blog post says, there is practice bias even in radiology. This would indeed explain why the model can learn 'racial bias'. 2. This is less clear to me. Just like humans can't see race on radio images, humans might be unable to see differences for diseases like pneumonia, but the nn could see them, no? In other words, how do you know that the differences have to come from hidden racial bias, and not from hidden pathological differences (that you don't know about, just like you've just discovered hidden racial bias) ?
        Agingcoder 5y ago
        (replying to myself about 2) You say the race can't be seen in the image by humans because it's not here, and therefore the race information must come from bias introduced by the radiologist. I guess it depends on whether all differences are visible to the human eye (and there's nothing pathological hidden that the nn 'sees') , and whether you can prove (a very hard thing in stats) that there's no way you can possibly extract race in a different manner. Now, assuming the data is racially biased because of medical practice biased, forgive me for being naive, but, why is this surprising / major? Isn't this just another instance of models being trained on bad data, and there are already plenty of examples in ia ? Edit:.. Unless the actual finding becomes 'radiology is (unexpectedly) racially biased, so much so that an ia can learn it'?