18 ms·

"AIs want the future to be like the past, and AIs make the future like the past. If the training data is full of human bias, then the predictions will also be f

by orr94 2y ago

"AIs want the future to be like the past, and AIs make the future like the past. If the training data is full of human bias, then the predictions will also be full of human bias, and then the outcomes will be full of human bias, and when those outcomes are copraphagically fed back into the training data, you get new, highly concentrated human/machine bias.” https://pluralistic.net/2025/03/18/asbestos-in-the-walls/#government-by-spicy-autocomplete https://pluralistic.net/2025/03/18/asbestos-in-the-walls/#go...

mhuffman 2y ago
"The model used in the new study, called CheXzero, was developed in 2022 by a team at Stanford University using a data set of almost 400,000 chest x-rays of people from Boston with conditions such as pulmonary edema, an accumulation of fluids in the lungs. Researchers fed their model the x-ray images without any of the associated radiologist reports, which contained information about diagnoses. " ... very interesting that the inputs to the model had nothing related to race or gender, but somehow it still was able to miss diagnose Black and female patients? I am curious of the mechanism for this. Can it just tell which x-rays belong to Black or female patients and then use some latent racism or misogyny to change the diagnosis? I do remember when it came out that AI could predict race from medical images with no other information[1], so that part seems possible. But where would it get the idea to do a worse diagnosis, even if it determines this? Surely there is no medical literature that recommends this! [1]https://news.mit.edu/2022/artificial-intelligence-predicts-patients-race-from-medical-images-0520 https://news.mit.edu/2022/artificial-intelligence-predicts-p...
- protonbob 2y ago
  I'm going to wager an uneducated guess. Black people are less likely to go to the doctor for both economic and historical reasons so images from them are going to be underrepresented. So in some way I guess you could say that yes, latent racism caused people to go to the doctor less which made them appear less in the data.
  - apical_dendrite 2y ago
    Where the data comes from also matters. Data is collected based on what's available to the researcher. Data from a particular city or time period may have a very different distribution than the general population.
  - encipriano 2y ago
    Arent black people like 10% of us population? You dont have ro look further
  - ars 2y ago
    Men are also way less likely to go to Dr vs women. Yet this claims a bias against women as well.
- daveguy 2y ago
  You really just have to understand one thing: AI is not intelligent. It's pattern matching without wisdom. If fewer people in the dataset are a particular race or gender it will do a shittier job predicting and won't even "understand" why or that it has bias, because it doesn't understand anything at a human level or even a dog level. At least most humans can learn their biases.
- bilbo0s 2y ago
  Isn't it kind of clear that it would have to be that the data they chose was influenced somehow by bias? Machines don't spontaneously do this stuff. But the humans that train the machines definitely do it all the time. Mostly without even thinking about it. I'm positive the issue is in the data selection and vetting. I would have been shocked if it was anything else.
- h2zizzle 2y ago
  Non-technical suggestion: if AI represents an aspect of the collective unconscious, as it were, then a racist society would produce latently racist training data that manifests in racist output, without anyone at any step being overtly racist. Same as an image model having a preference for red apples (even though there are many colors of apple, and even red ones are not uniformly cherry red). The training data has a preponderance of examples where doctors missed a clear diagnosis because of their unconscious bias? Then this outcome would be unsurprising. An interesting test would be to see if a similar issue pops up for obese patients. A common complaint, IIUC, is that doctors will chalk up a complaint to their obesity rather than investigating further for a more specific (perhaps pathological) cause.
- FanaHOVA 2y ago
  The non-tinfoil hat approach is to simply Google "Boston demographics", and think of how training data distribution impacts model performance. > The data set used to train CheXzero included more men, more people between 40 and 80 years old, and more white patients, which Yang says underscores the need for larger, more diverse data sets. I'm not a doctor so I cannot tell you how xrays differ across genders / ethnicities, but these models aren't magic (especially computer vision ones, which are usually much smaller). If there are meaningful differences and they don't see those specific cases in training data, they will always fail to recognize them at inference.
- cratermoon 2y ago
  > Can it just tell which x-rays belong to Black or female patients and then use some latent racism or misogyny to change the diagnosis? The opposite. The dataset is for the standard model "white male", and the diagnoses generated pattern-matched on that. Because there's no gender or racial information, the model produced the statistically most likely result for white male, a result less likely to be correct for a patient that doesn't fit the standard model.
  - XorNot 2y ago
    The better question is just "are you actually just selecting for symptom occurrence by socioeconomic group?" Like you could modify the question to ask "is the model better at diagnosing people who went to a certain school?" and simplistically the answer would likely seem to be yes.
  - searealist 2y ago
    Then why is the headline not "AI models miss disease in Asian patients" or even "AI models miss disease in Latino patients"? It just so happens to align with what maximizes political capital in today's world.
ideamotor 2y ago
I really can’t help but think of the simulation hypothesis. What are the chances this copy-cat technology was developed when I was alive, given that it keeps going.
- kcorbitt 2y ago
  We may be in a simulation, but your odds of being alive to see this (conditioned on being born as a human at some point) aren't that low. Around 7% of all humans ever born are alive today!
  - encipriano 2y ago
    I dont believe that percentage. Especially considering how spread the homo branch already was more than 100 000 years ago. And from which point do you start counting? Homo erectus?
    - bobthepanda 2y ago
      I would imagine this is probably the source, which benchmarks using the last 200,000 years. https://www.prb.org/articles/how-many-people-have-ever-lived-on-earth/ https://www.prb.org/articles/how-many-people-have-ever-lived... Given that we only hit the first billion people in 1804 and the second billion in 1927 it's not all that shocking.
    - XorNot 2y ago
      That argument works both ways, it might be significantly higher depending how you count. But this is also just the non-intuitiveness of exponential growth which has only now tapering off.
    - jfengel 2y ago
      It kinda doesn't matter where you start counting. Exponential curves put almost everything at the end. Adding to the left side doesn't change it much. You could go back to Lucy and add only a few million. Compared to the billions at this specific instant, it just doesn't make a difference.
bko 2y ago
Suppose you have a system that saves 90% of lives on group A but only 80% of lives in group B. This is due to the fact that you have considerably more training data on group A. You cannot release this life saving technology because it has a 'disparate impact' on group B relative to group A. So the obvious thing to do is to have the technology intentionally kill ~1 out of every 10 patients from group A so the efficacy rate is ~80% for both groups. Problem solved From the article: > “What is clear is that it’s going to be really difficult to mitigate these biases,” says Judy Gichoya, an interventional radiologist and informatician at Emory University who was not involved in the study. Instead, she advocates for smaller, but more diverse data sets that test these AI models to identify their flaws and correct them on a small scale first. Even so, “Humans have to be in the loop,” she says. “AI can’t be left on its own.” Quiz: What impact would smaller data sets have on efficacy for group A? How about group B? Explain your reasoning
- janice1999 2y ago
  > You cannot release this life saving technology because it has a 'disparate impact' on group B relative to group A. Who is preventing you in this imagined scenario? There are drugs that are more effective on certain groups of people than others. BiDil, for example, is an FDA approved drug marketed to a single racial-ethnic group, African Americans, in the treatment of congestive heart failure. As long as the risks are understood there can be accommodations made ("this AI tool is for males only" etc). However such limitations and restrictions are rarely mentioned or understood by AI hype people.
  - bko 2y ago
    What does this have to do with FDA or drugs? Re-read the comment I was replying to. It's complaining that a technology could serve one group of people better than another, and I would argue that this should not be our goal. A technology should be judged by "does it provide value to any group or harm any other group". But endlessly dividing people into groups and saying how everything is unfair because it benefits group A over group B due to the nature of the problem, just results in endless hand-wringing and conservatism and delays useful technology from being released due to the fear of mean headlines like this.
timewizard 2y ago
LLMs don't and cannot want things. Human beings also like it when the future is mostly like the past. They just call that "predictability." Human data is bias. You literally cannot remove one from the other. There are some people who want to erase humanity's will and replace it with an anthropomorphized algorithm. These people concern me.
- balamatom 2y ago
  The most concerning people are -- as ever -- those who only think that they are thinking. Those who keep trying to fit square pegs into triangular holes without, you know, stopping to reflect: who gave them those pegs in the first place, and to what end? Why be obtuse? There is no "anthropomorphic fallacy" here to dispel. You know very well that "LLMs want" is simply a way of speaking about teleology without antagonizing people who are taught that they should be afraid of precise notions ("big words"). But accepting that bias can lead to some pretty funny conflations. For example, humanity as a whole doesn't have this "will" you speak of any more than LLMs can "want"; will is an aspect of the consciousness of the individual. So you seem to be be uncritically anthropomorphizing social processes! If we assume those to be chaotic, in that sense any sort of algorithm is slightly more anthropomorphic: at least it works towards a human-given and therefore human-comprehensible purpose -- on the other hand, whether there is some particular "destination of history" towards which humanity is moving, is a question that can only ever be speculated upon, but not definitively perceived.
  - verisimi 2y ago
    > If we assume those to be chaotic, in that sense any sort of algorithm is slightly more anthropomorphic: at least it works towards a human-given and therefore human-comprehensible purpose -- on the other hand, whether there is some particular "destination of history" towards which humanity is moving, is a question that can only ever be speculated upon, but not definitively perceived. Do you not think that if you anthropomorphise things that aren't actually anthropic, that you then insert a bias towards those things? The bias will actually discriminate at the expense of people. If that is so, the destination of history will inevitably be misanthropic. Misplaced anthropomorphism is a genuine, present concern.
MountainArras 2y ago
The dataset they used to train the model are chest xrays of known diseases. I'm having trouble understanding how that's relevant here. The key takeaway is that you can't treat all humans as a single group in this context, and variations in the biology across different groups of people may need to be taken into account within the training process. In other words, the model will need to be trained on this racial/gender data too in order to get better results when predicting the targeted diseases within these groups. I think it's interesting to think about instead attaching generic information instead of group data, which would be blind to human bias and the messiness of our rough categorizations of subgroups.
- pelorat 2y ago
  I think the model needs to be thought about human anatomy, not just fed a bunch of scans. It needs to understand what ribs and organs are.
  - ericmcer 2y ago
    I don't think LLMs can achieve "understanding" in that sense.
    - nomel 2y ago
      These aren't LLM. Most of the neat things in science, involving AI, aren't LLM. Next word prediction has extremely limited use with non-text data.
      - thaumasiotes 2y ago
        People seem to have started to use "LLM" to refer to any suite of software that includes an LLM somewhere within it; you can see them talking about LLM-generated art, for example.
        hnlmorg 2y ago
        Was it ascii art? ;)
_l7dh 2y ago
As Sara Hooker discussed in her paper https://www.cell.com/patterns/fulltext/S2666-3899(21)00061-1?ref=salesforce-research https://www.cell.com/patterns/fulltext/S2666-3899(21)00061-1..., bias goes way beyond data.
jhanschoo 2y ago
I like how the author used neo-Greek words to sneak in graphic imagery that would normally be taboo in this register of writing
- MonkeyClub 2y ago
  I dislike how they misspelled it though.

by orr94 2y ago

mhuffman 2y ago

"The model used in the new study, called CheXzero, was developed in 2022 by a team at Stanford University using a data set of almost 400,000 chest x-rays of people from Boston with conditions such as pulmonary edema, an accumulation of fluids in the lungs. Researchers fed their model the x-ray images without any of the associated radiologist reports, which contained information about diagnoses. " ... very interesting that the inputs to the model had nothing related to race or gender, but somehow it still was able to miss diagnose Black and female patients? I am curious of the mechanism for this. Can it just tell which x-rays belong to Black or female patients and then use some latent racism or misogyny to change the diagnosis? I do remember when it came out that AI could predict race from medical images with no other information[1], so that part seems possible. But where would it get the idea to do a worse diagnosis, even if it determines this? Surely there is no medical literature that recommends this! [1]https://news.mit.edu/2022/artificial-intelligence-predicts-patients-race-from-medical-images-0520 https://news.mit.edu/2022/artificial-intelligence-predicts-p...

protonbob 2y ago
I'm going to wager an uneducated guess. Black people are less likely to go to the doctor for both economic and historical reasons so images from them are going to be underrepresented. So in some way I guess you could say that yes, latent racism caused people to go to the doctor less which made them appear less in the data.
- apical_dendrite 2y ago
  Where the data comes from also matters. Data is collected based on what's available to the researcher. Data from a particular city or time period may have a very different distribution than the general population.
- encipriano 2y ago
  Arent black people like 10% of us population? You dont have ro look further
- ars 2y ago
  Men are also way less likely to go to Dr vs women. Yet this claims a bias against women as well.
daveguy 2y ago
You really just have to understand one thing: AI is not intelligent. It's pattern matching without wisdom. If fewer people in the dataset are a particular race or gender it will do a shittier job predicting and won't even "understand" why or that it has bias, because it doesn't understand anything at a human level or even a dog level. At least most humans can learn their biases.
bilbo0s 2y ago
Isn't it kind of clear that it would have to be that the data they chose was influenced somehow by bias? Machines don't spontaneously do this stuff. But the humans that train the machines definitely do it all the time. Mostly without even thinking about it. I'm positive the issue is in the data selection and vetting. I would have been shocked if it was anything else.
h2zizzle 2y ago
Non-technical suggestion: if AI represents an aspect of the collective unconscious, as it were, then a racist society would produce latently racist training data that manifests in racist output, without anyone at any step being overtly racist. Same as an image model having a preference for red apples (even though there are many colors of apple, and even red ones are not uniformly cherry red). The training data has a preponderance of examples where doctors missed a clear diagnosis because of their unconscious bias? Then this outcome would be unsurprising. An interesting test would be to see if a similar issue pops up for obese patients. A common complaint, IIUC, is that doctors will chalk up a complaint to their obesity rather than investigating further for a more specific (perhaps pathological) cause.
FanaHOVA 2y ago
The non-tinfoil hat approach is to simply Google "Boston demographics", and think of how training data distribution impacts model performance. > The data set used to train CheXzero included more men, more people between 40 and 80 years old, and more white patients, which Yang says underscores the need for larger, more diverse data sets. I'm not a doctor so I cannot tell you how xrays differ across genders / ethnicities, but these models aren't magic (especially computer vision ones, which are usually much smaller). If there are meaningful differences and they don't see those specific cases in training data, they will always fail to recognize them at inference.
cratermoon 2y ago
> Can it just tell which x-rays belong to Black or female patients and then use some latent racism or misogyny to change the diagnosis? The opposite. The dataset is for the standard model "white male", and the diagnoses generated pattern-matched on that. Because there's no gender or racial information, the model produced the statistically most likely result for white male, a result less likely to be correct for a patient that doesn't fit the standard model.
- XorNot 2y ago
  The better question is just "are you actually just selecting for symptom occurrence by socioeconomic group?" Like you could modify the question to ask "is the model better at diagnosing people who went to a certain school?" and simplistically the answer would likely seem to be yes.
- searealist 2y ago
  Then why is the headline not "AI models miss disease in Asian patients" or even "AI models miss disease in Latino patients"? It just so happens to align with what maximizes political capital in today's world.

ideamotor 2y ago

I really can’t help but think of the simulation hypothesis. What are the chances this copy-cat technology was developed when I was alive, given that it keeps going.

kcorbitt 2y ago
We may be in a simulation, but your odds of being alive to see this (conditioned on being born as a human at some point) aren't that low. Around 7% of all humans ever born are alive today!
- encipriano 2y ago
  I dont believe that percentage. Especially considering how spread the homo branch already was more than 100 000 years ago. And from which point do you start counting? Homo erectus?
  - bobthepanda 2y ago
    I would imagine this is probably the source, which benchmarks using the last 200,000 years. https://www.prb.org/articles/how-many-people-have-ever-lived-on-earth/ https://www.prb.org/articles/how-many-people-have-ever-lived... Given that we only hit the first billion people in 1804 and the second billion in 1927 it's not all that shocking.
  - XorNot 2y ago
    That argument works both ways, it might be significantly higher depending how you count. But this is also just the non-intuitiveness of exponential growth which has only now tapering off.
  - jfengel 2y ago
    It kinda doesn't matter where you start counting. Exponential curves put almost everything at the end. Adding to the left side doesn't change it much. You could go back to Lucy and add only a few million. Compared to the billions at this specific instant, it just doesn't make a difference.

bko 2y ago

Suppose you have a system that saves 90% of lives on group A but only 80% of lives in group B. This is due to the fact that you have considerably more training data on group A. You cannot release this life saving technology because it has a 'disparate impact' on group B relative to group A. So the obvious thing to do is to have the technology intentionally kill ~1 out of every 10 patients from group A so the efficacy rate is ~80% for both groups. Problem solved From the article: > “What is clear is that it’s going to be really difficult to mitigate these biases,” says Judy Gichoya, an interventional radiologist and informatician at Emory University who was not involved in the study. Instead, she advocates for smaller, but more diverse data sets that test these AI models to identify their flaws and correct them on a small scale first. Even so, “Humans have to be in the loop,” she says. “AI can’t be left on its own.” Quiz: What impact would smaller data sets have on efficacy for group A? How about group B? Explain your reasoning

janice1999 2y ago
> You cannot release this life saving technology because it has a 'disparate impact' on group B relative to group A. Who is preventing you in this imagined scenario? There are drugs that are more effective on certain groups of people than others. BiDil, for example, is an FDA approved drug marketed to a single racial-ethnic group, African Americans, in the treatment of congestive heart failure. As long as the risks are understood there can be accommodations made ("this AI tool is for males only" etc). However such limitations and restrictions are rarely mentioned or understood by AI hype people.
- bko 2y ago
  What does this have to do with FDA or drugs? Re-read the comment I was replying to. It's complaining that a technology could serve one group of people better than another, and I would argue that this should not be our goal. A technology should be judged by "does it provide value to any group or harm any other group". But endlessly dividing people into groups and saying how everything is unfair because it benefits group A over group B due to the nature of the problem, just results in endless hand-wringing and conservatism and delays useful technology from being released due to the fear of mean headlines like this.

timewizard 2y ago

LLMs don't and cannot want things. Human beings also like it when the future is mostly like the past. They just call that "predictability." Human data is bias. You literally cannot remove one from the other. There are some people who want to erase humanity's will and replace it with an anthropomorphized algorithm. These people concern me.

balamatom 2y ago
The most concerning people are -- as ever -- those who only think that they are thinking. Those who keep trying to fit square pegs into triangular holes without, you know, stopping to reflect: who gave them those pegs in the first place, and to what end? Why be obtuse? There is no "anthropomorphic fallacy" here to dispel. You know very well that "LLMs want" is simply a way of speaking about teleology without antagonizing people who are taught that they should be afraid of precise notions ("big words"). But accepting that bias can lead to some pretty funny conflations. For example, humanity as a whole doesn't have this "will" you speak of any more than LLMs can "want"; will is an aspect of the consciousness of the individual. So you seem to be be uncritically anthropomorphizing social processes! If we assume those to be chaotic, in that sense any sort of algorithm is slightly more anthropomorphic: at least it works towards a human-given and therefore human-comprehensible purpose -- on the other hand, whether there is some particular "destination of history" towards which humanity is moving, is a question that can only ever be speculated upon, but not definitively perceived.
- verisimi 2y ago
  > If we assume those to be chaotic, in that sense any sort of algorithm is slightly more anthropomorphic: at least it works towards a human-given and therefore human-comprehensible purpose -- on the other hand, whether there is some particular "destination of history" towards which humanity is moving, is a question that can only ever be speculated upon, but not definitively perceived. Do you not think that if you anthropomorphise things that aren't actually anthropic, that you then insert a bias towards those things? The bias will actually discriminate at the expense of people. If that is so, the destination of history will inevitably be misanthropic. Misplaced anthropomorphism is a genuine, present concern.

MountainArras 2y ago

The dataset they used to train the model are chest xrays of known diseases. I'm having trouble understanding how that's relevant here. The key takeaway is that you can't treat all humans as a single group in this context, and variations in the biology across different groups of people may need to be taken into account within the training process. In other words, the model will need to be trained on this racial/gender data too in order to get better results when predicting the targeted diseases within these groups. I think it's interesting to think about instead attaching generic information instead of group data, which would be blind to human bias and the messiness of our rough categorizations of subgroups.

pelorat 2y ago
I think the model needs to be thought about human anatomy, not just fed a bunch of scans. It needs to understand what ribs and organs are.
- ericmcer 2y ago
  I don't think LLMs can achieve "understanding" in that sense.
  - nomel 2y ago
    These aren't LLM. Most of the neat things in science, involving AI, aren't LLM. Next word prediction has extremely limited use with non-text data.
    - thaumasiotes 2y ago
      People seem to have started to use "LLM" to refer to any suite of software that includes an LLM somewhere within it; you can see them talking about LLM-generated art, for example.
      - hnlmorg 2y ago
        Was it ascii art? ;)

_l7dh 2y ago

As Sara Hooker discussed in her paper https://www.cell.com/patterns/fulltext/S2666-3899(21)00061-1?ref=salesforce-research https://www.cell.com/patterns/fulltext/S2666-3899(21)00061-1..., bias goes way beyond data.

jhanschoo 2y ago

I like how the author used neo-Greek words to sneak in graphic imagery that would normally be taboo in this register of writing

MonkeyClub 2y ago
I dislike how they misspelled it though.