6 ms·

Study reveals why AI models that analyze medical images can be biased

45 points by wglb 2y ago

zaptrem 2y ago
``` The researchers also found that they could retrain the models in a way that improves their fairness. However, their approach to "debiasing" worked best when the models were tested on the same types of patients on whom they were trained, such as patients from the same hospital. When these models were applied to patients from different hospitals, the fairness gaps reappeared. "I think the main takeaways are, first, you should thoroughly evaluate any external models on your own data because any fairness guarantees that model developers provide on their training data may not transfer to your population. Second, whenever sufficient data is available, you should train models on your own data," says Haoran Zhang, an MIT graduate student and one of the lead authors of the new paper. ``` This is just overfitting. Why are they training whole models on only one hospital worth of data when they appear to have access to five? They should be training on all of the data in the world they can get their hands on then maybe fine tuning on their specific hospital (maybe they have higher quality outcomes data that verifies the readings) if there are still accuracy issues. The last five years have taught us that gobbling up everything (even if it's not the best quality) is the way.
- hdhshdhshdjd 2y ago
  It could be an instrumentation issue, different hospitals use different machines.
  - zaptrem 2y ago
    Likely this and radiologist practices and storage methods and patient demographics and a bunch of other things. However, if they're training these models with the intention of using them in other hospitals I'd still define that as overfitting.
    - deleted 2y ago
      [deleted]
teruakohatu 2y ago
I have not been able to fully digest this paper yet, and medical data is not my speciality. It is interesting but not surprising that models appear able to determine the demographics of the patient that even radiologists are unable to. It is also not surprising that models use this to "cheat" (find demographic shortcuts in disease classification). My understanding is that doctors may unconsciously do this as well, ignoring a possible diagnosis because they don't expect a patient of a certain demographic to have a particular issue. I would expect radiologists who practice in very different demographic environments would not do as well when evaluating images another environment. At the end of the day radiology is more an art than a science, so the training data may well be faulty. Krupinski (2010) wrote in an interesting paper [1]: "Medical images need to be interpreted because they are not self-explanatory... In radiology alone, estimates suggest that, in some areas, there may be up to a 30% miss rate and an equally high false positive rate ... interpretation errors can be caused by a host of psychophysical processes ... radiologists are less accurate after a day of reading diagnostic images and that their ability to focus on the display screen is reduced because of myopia. " I would hope datasets included a substantial amount of images that were originally mis-classified as a human. [1] https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3881280/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3881280/
- carbocation 2y ago
  It's not too hard to train yourself to identify sex or approximate age. (I claim this because of time spent reviewing model output for work that I have done to build models to estimate age.) The reason radiologists don't do this is that there is no clinical reason to do it, so it's not a skill that they develop. (Deep neural networks can also beat radiologists at bird classification, another task that is not relevant for their job.)
wiradikusuma 2y ago
Instead of trying to make the model "fair", can we do "model = models.getByRace(x)" so we have optimized model for each, instead of being "jack of all trades"?
- teruakohatu 2y ago
  This paper covers many demographics, not just ethnicity. So you are wanting to really do "model = models.get(a, b, x, y, z)". It may be possible, but with medical data in such short supply, you would be lucky to get a large enough dataset. You would also need to assume there no overlap in the distribution in the training data for each strata of training data. I would imagine that if you taught someone to detect cancer in x-rays, and only used images of people aged 80+, you might still have some success (>0.5) at detecting cancer in people aged 20-30.
  - sdenton4 2y ago
    This seems like a place where generative augmentation could help - generate an image of a person with demographic values x, y, z and an early stage lung cancer. Then you can generate images for more corners of the days, more evenly, and train the classifier on less biased data.
    - teruakohatu 2y ago
      If your model already understand the distribution of the data (fundamentally this is what statistical models are trying to discover) well enough for generative models then they probably can do classification well enough. The greater issue is that, for privacy and other reasons, there are many demographics that are not represented at all, or under represented, in medical image datasets.
    - CoastalCoder 2y ago
      Just in case you'd want to know, the singular of "strata" is "stratum". (Although apparently that's sometimes debated [0].) [0] https://www.dictionary.com/browse/stratum https://www.dictionary.com/browse/stratum
ogarten 2y ago
How does this surprise anyone? Medical data for AI training is almost always sources in some more or less shady country because they lack any privacy regulations. It's then annotated by a hoard of cheap workers who may or may not have advanced medical training. Even "normal medicine" is extremely biased towards male people fitting inside the norm which is why a lot of things are not detected early enough in women or in people who do not match that norm. Next thing: Doctors often think that their annotations are the absolute gold standard but they don't necessarily know everything that is in an X-Ray or an MRI. A few years ago we tried to build synthetic data for this exact purpose by simulating medical images for 3D body models with different diseases and nobody we talked to cared about it, because "we have good data".
- aprilthird2021 2y ago
  Yep, you nailed it. You really don't have to think hard about why AI which only learns from what we feed it and can access has gaps and biases more pronounced than the real world. AI lives in the internet world, it's trained on horrible cesspools of anonymous text like 4chan and reddit. No wonder it will be biased. If you only try to feed it sanitary data you wouldn't have enough to get the results we get now.
  - DrScientist 2y ago
    > You really don't have to think hard about why AI which only learns from what we feed it Sadly I'd say that people are no different. > it's trained on horrible cesspools of .... So it's really not the future of AI we should be worrying about...
    - aprilthird2021 2y ago
      People are different because they have human social interaction offline where terminally online stereotypes and biases often fall apart
- davedx 2y ago
  I know GPT4o can diagnose medical images. Is their model likely to be using the same kind of datasets as these models for medical systems?
carbocation 2y ago
The authors refer to a literature describing "shortcuts" as "correlations that are present in the data but have no real clinical basis, for instance deep models using the hospital as a shortcut for disease prediction". It feels like a parallel language is developing. Most of us would call such a phenomenon "overfitting" or describe some specific issue with generalization. That example is not a shortcut in any normal sense of the word unless you are providing the hospital via some extra path. They call demographics like age and sex "shortcuts" but I find this to be a frustrating term since it seems to obscure what's happening under the hood. (They cite many papers using the same word, so I'm not blaming them for this usage.) Men are typically larger; old bones do not look like young bones. There is plenty of biology involved in what they refer to as demographic shortcuts. I think you could take the same results and say "Models are able to distinguish men from women. For our purposes, it's important that they cannot do this. Therefore, we did XYZ on these weakly labeled public databases." But perhaps that sounds less exciting.
- bravura 2y ago
  This is not overfitting precisely because of the bias/variance tradeoff. A model overfits if it is unnecessarily COMPLEX for the training data. If there is bias in the training (and validation and test) data that allows a SIMPLE model to fit the data because of a spurious correlation, that is not overfitting.
  - carbocation 2y ago
    There is no reason to believe that an X-ray model's correct estimation of age is due to "spurious correlation". Rather, it seems to be "undesirable correlation".
    - TeMPOraL 2y ago
      > Rather, it seems to be "undesirable correlation". More specifically, politically undesirable correlation - as in, "it's there, but its existence upsets some people". It's pretty obvious and self-evident that there are meaningful biological differences related to age, sex, and other demographics. Whether or not they're clinically relevant for a specific diagnosis under question is one thing, but they are clinically relevant for great many diagnoses; trying to "de-bias" reality here will only lead to unnecessary suffering and loss of life.
amluto 2y ago
I’m suspicious that there’s another factor in play: images being correctly labeled in the training set. Even from high-end hospitals, my personal experience leads me to believe that radiologists make both major types of error on a regular basis: calling out abnormalities in an image that are entirely irrelevant and missing actual relevant problems in an image that are subsequently easily seen by a doctor who is aware of what seems to be actually wrong. To top it off, sure many patients end up being diagnosed with whatever the radiologist saw, and no one ever confirms that the diagnosis was really correct. (And how could they? A lot of conditions are hard to diagnose!) So the training data is probably hugely biased, and the models will learn to predict the training labels as opposed to any magically correct “ground truth”. And internally detecting demographics and producing an output biased by demographics may well result in a better match to the training data than a perfect, unbiased output would be.
- resource_waste 2y ago
  Yeah this is actually it. Its much easier for physicians to blame technology than their profession.
  - nradov 2y ago
    I'm not sure it's a matter of "blame" as such. While there are occasional cases of incompetence or malpractice, biology is inherently messy and uncertain.
- KingOfCoders 2y ago
  We recently had two ultrasonic examinations, both by specialists, both saw different things.
aaron695 2y ago
[dead]
adammarples 2y ago
Is it just me or did the article at no point explain why medical models produce biased results? I seems to take this as a given, and an uninteresting one at that, and focuses on trying to correct it, without explaining why it happens in the first place. Yes, those models could use race as a shortcut to, presumably, not diagnose cancer in black people for example, but why is the race shortcut boosting model training accuracy? I am still none the wiser.
throwaway22032 2y ago
The language used in the article seems political/inflammatory to me. It's not a lack of "fairness", it's just a lack of accuracy. Imagine that you train a model to find roofing issues or subsidence or something from aerial imagery. Maybe it performs better on Victorian terraces, because there are lots of those in the UK. Would you call it unfair because it doesn't do so well on thatched roof properties? No, it's just inaccurate, calling it unfair is a value judgement. Bias is better because it at least has a statistical basis but fairness is, well.. inaccurate...
- CoastalCoder 2y ago
  This was my general take as well. Although now that I think of it, there could be a "fairness" angle regarding how many training data are obtained from each population group. But if that's what the author meant by fairness, it wasn't clear.
- KingOfCoders 2y ago
  I think how a scientist would use 'fairness' - e.g. mathematicians - and the general public would use 'fairness' are two different things. On top of that fairness in the general public is undefined as well. I once wrote on the many facets just concerning fair salaries here: https://www.amazingcto.com/fair-remote-developer-salaries/ https://www.amazingcto.com/fair-remote-developer-salaries/
kosh2 2y ago
I agree with many posters in here, that the cause will likely be bad data one way or another. Maybe we need to take a step back and only use data, that are almost 100% accurate. Like the time of death after the data was collected. If a model could with a high accuracy predict, that a patient will die within X days (without proper treatment), it will be already very valuable. Second, as Sora has shown, going multi model can have amazing benefits. Get a breath analysis of the patient, get a video, get a sound recording, get an MRI, get a CT, get a full blood sample and then let the model do its pattern finding magic.

45 points by wglb 2y ago

``` The researchers also found that they could retrain the models in a way that improves their fairness. However, their approach to "debiasing" worked best when the models were tested on the same types of patients on whom they were trained, such as patients from the same hospital. When these models were applied to patients from different hospitals, the fairness gaps reappeared. "I think the main takeaways are, first, you should thoroughly evaluate any external models on your own data because any fairness guarantees that model developers provide on their training data may not transfer to your population. Second, whenever sufficient data is available, you should train models on your own data," says Haoran Zhang, an MIT graduate student and one of the lead authors of the new paper. ``` This is just overfitting. Why are they training whole models on only one hospital worth of data when they appear to have access to five? They should be training on all of the data in the world they can get their hands on then maybe fine tuning on their specific hospital (maybe they have higher quality outcomes data that verifies the readings) if there are still accuracy issues. The last five years have taught us that gobbling up everything (even if it's not the best quality) is the way.

hdhshdhshdjd 2y ago
It could be an instrumentation issue, different hospitals use different machines.
- zaptrem 2y ago
  Likely this and radiologist practices and storage methods and patient demographics and a bunch of other things. However, if they're training these models with the intention of using them in other hospitals I'd still define that as overfitting.
  - deleted 2y ago
    [deleted]

teruakohatu 2y ago

I have not been able to fully digest this paper yet, and medical data is not my speciality. It is interesting but not surprising that models appear able to determine the demographics of the patient that even radiologists are unable to. It is also not surprising that models use this to "cheat" (find demographic shortcuts in disease classification). My understanding is that doctors may unconsciously do this as well, ignoring a possible diagnosis because they don't expect a patient of a certain demographic to have a particular issue. I would expect radiologists who practice in very different demographic environments would not do as well when evaluating images another environment. At the end of the day radiology is more an art than a science, so the training data may well be faulty. Krupinski (2010) wrote in an interesting paper [1]: "Medical images need to be interpreted because they are not self-explanatory... In radiology alone, estimates suggest that, in some areas, there may be up to a 30% miss rate and an equally high false positive rate ... interpretation errors can be caused by a host of psychophysical processes ... radiologists are less accurate after a day of reading diagnostic images and that their ability to focus on the display screen is reduced because of myopia. " I would hope datasets included a substantial amount of images that were originally mis-classified as a human. [1] https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3881280/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3881280/

carbocation 2y ago
It's not too hard to train yourself to identify sex or approximate age. (I claim this because of time spent reviewing model output for work that I have done to build models to estimate age.) The reason radiologists don't do this is that there is no clinical reason to do it, so it's not a skill that they develop. (Deep neural networks can also beat radiologists at bird classification, another task that is not relevant for their job.)

wiradikusuma 2y ago

Instead of trying to make the model "fair", can we do "model = models.getByRace(x)" so we have optimized model for each, instead of being "jack of all trades"?

teruakohatu 2y ago
This paper covers many demographics, not just ethnicity. So you are wanting to really do "model = models.get(a, b, x, y, z)". It may be possible, but with medical data in such short supply, you would be lucky to get a large enough dataset. You would also need to assume there no overlap in the distribution in the training data for each strata of training data. I would imagine that if you taught someone to detect cancer in x-rays, and only used images of people aged 80+, you might still have some success (>0.5) at detecting cancer in people aged 20-30.
- sdenton4 2y ago
  This seems like a place where generative augmentation could help - generate an image of a person with demographic values x, y, z and an early stage lung cancer. Then you can generate images for more corners of the days, more evenly, and train the classifier on less biased data.
  - teruakohatu 2y ago
    If your model already understand the distribution of the data (fundamentally this is what statistical models are trying to discover) well enough for generative models then they probably can do classification well enough. The greater issue is that, for privacy and other reasons, there are many demographics that are not represented at all, or under represented, in medical image datasets.
  - CoastalCoder 2y ago
    Just in case you'd want to know, the singular of "strata" is "stratum". (Although apparently that's sometimes debated [0].) [0] https://www.dictionary.com/browse/stratum https://www.dictionary.com/browse/stratum

ogarten 2y ago

How does this surprise anyone? Medical data for AI training is almost always sources in some more or less shady country because they lack any privacy regulations. It's then annotated by a hoard of cheap workers who may or may not have advanced medical training. Even "normal medicine" is extremely biased towards male people fitting inside the norm which is why a lot of things are not detected early enough in women or in people who do not match that norm. Next thing: Doctors often think that their annotations are the absolute gold standard but they don't necessarily know everything that is in an X-Ray or an MRI. A few years ago we tried to build synthetic data for this exact purpose by simulating medical images for 3D body models with different diseases and nobody we talked to cared about it, because "we have good data".

aprilthird2021 2y ago
Yep, you nailed it. You really don't have to think hard about why AI which only learns from what we feed it and can access has gaps and biases more pronounced than the real world. AI lives in the internet world, it's trained on horrible cesspools of anonymous text like 4chan and reddit. No wonder it will be biased. If you only try to feed it sanitary data you wouldn't have enough to get the results we get now.
- DrScientist 2y ago
  > You really don't have to think hard about why AI which only learns from what we feed it Sadly I'd say that people are no different. > it's trained on horrible cesspools of .... So it's really not the future of AI we should be worrying about...
  - aprilthird2021 2y ago
    People are different because they have human social interaction offline where terminally online stereotypes and biases often fall apart
davedx 2y ago
I know GPT4o can diagnose medical images. Is their model likely to be using the same kind of datasets as these models for medical systems?

carbocation 2y ago

The authors refer to a literature describing "shortcuts" as "correlations that are present in the data but have no real clinical basis, for instance deep models using the hospital as a shortcut for disease prediction". It feels like a parallel language is developing. Most of us would call such a phenomenon "overfitting" or describe some specific issue with generalization. That example is not a shortcut in any normal sense of the word unless you are providing the hospital via some extra path. They call demographics like age and sex "shortcuts" but I find this to be a frustrating term since it seems to obscure what's happening under the hood. (They cite many papers using the same word, so I'm not blaming them for this usage.) Men are typically larger; old bones do not look like young bones. There is plenty of biology involved in what they refer to as demographic shortcuts. I think you could take the same results and say "Models are able to distinguish men from women. For our purposes, it's important that they cannot do this. Therefore, we did XYZ on these weakly labeled public databases." But perhaps that sounds less exciting.

bravura 2y ago
This is not overfitting precisely because of the bias/variance tradeoff. A model overfits if it is unnecessarily COMPLEX for the training data. If there is bias in the training (and validation and test) data that allows a SIMPLE model to fit the data because of a spurious correlation, that is not overfitting.
- carbocation 2y ago
  There is no reason to believe that an X-ray model's correct estimation of age is due to "spurious correlation". Rather, it seems to be "undesirable correlation".
  - TeMPOraL 2y ago
    > Rather, it seems to be "undesirable correlation". More specifically, politically undesirable correlation - as in, "it's there, but its existence upsets some people". It's pretty obvious and self-evident that there are meaningful biological differences related to age, sex, and other demographics. Whether or not they're clinically relevant for a specific diagnosis under question is one thing, but they are clinically relevant for great many diagnoses; trying to "de-bias" reality here will only lead to unnecessary suffering and loss of life.

amluto 2y ago

I’m suspicious that there’s another factor in play: images being correctly labeled in the training set. Even from high-end hospitals, my personal experience leads me to believe that radiologists make both major types of error on a regular basis: calling out abnormalities in an image that are entirely irrelevant and missing actual relevant problems in an image that are subsequently easily seen by a doctor who is aware of what seems to be actually wrong. To top it off, sure many patients end up being diagnosed with whatever the radiologist saw, and no one ever confirms that the diagnosis was really correct. (And how could they? A lot of conditions are hard to diagnose!) So the training data is probably hugely biased, and the models will learn to predict the training labels as opposed to any magically correct “ground truth”. And internally detecting demographics and producing an output biased by demographics may well result in a better match to the training data than a perfect, unbiased output would be.

resource_waste 2y ago
Yeah this is actually it. Its much easier for physicians to blame technology than their profession.
- nradov 2y ago
  I'm not sure it's a matter of "blame" as such. While there are occasional cases of incompetence or malpractice, biology is inherently messy and uncertain.
KingOfCoders 2y ago
We recently had two ultrasonic examinations, both by specialists, both saw different things.

aaron695 2y ago

[dead]

adammarples 2y ago

Is it just me or did the article at no point explain why medical models produce biased results? I seems to take this as a given, and an uninteresting one at that, and focuses on trying to correct it, without explaining why it happens in the first place. Yes, those models could use race as a shortcut to, presumably, not diagnose cancer in black people for example, but why is the race shortcut boosting model training accuracy? I am still none the wiser.

throwaway22032 2y ago

The language used in the article seems political/inflammatory to me. It's not a lack of "fairness", it's just a lack of accuracy. Imagine that you train a model to find roofing issues or subsidence or something from aerial imagery. Maybe it performs better on Victorian terraces, because there are lots of those in the UK. Would you call it unfair because it doesn't do so well on thatched roof properties? No, it's just inaccurate, calling it unfair is a value judgement. Bias is better because it at least has a statistical basis but fairness is, well.. inaccurate...

CoastalCoder 2y ago
This was my general take as well. Although now that I think of it, there could be a "fairness" angle regarding how many training data are obtained from each population group. But if that's what the author meant by fairness, it wasn't clear.
KingOfCoders 2y ago
I think how a scientist would use 'fairness' - e.g. mathematicians - and the general public would use 'fairness' are two different things. On top of that fairness in the general public is undefined as well. I once wrote on the many facets just concerning fair salaries here: https://www.amazingcto.com/fair-remote-developer-salaries/ https://www.amazingcto.com/fair-remote-developer-salaries/

kosh2 2y ago

I agree with many posters in here, that the cause will likely be bad data one way or another. Maybe we need to take a step back and only use data, that are almost 100% accurate. Like the time of death after the data was collected. If a model could with a high accuracy predict, that a patient will die within X days (without proper treatment), it will be already very valuable. Second, as Sora has shown, going multi model can have amazing benefits. Get a breath analysis of the patient, get a video, get a sound recording, get an MRI, get a CT, get a full blood sample and then let the model do its pattern finding magic.