6 ms·

Most users cannot identify AI bias, even in training data

segmondy 1y ago
most people can't identify bias in real life, let alone in AI.
DocTomoe 1y ago
If bias can only be seen by a minority of people ... is it really 'AI bias', or just societal bias? > “In one of the experiment scenarios — which featured racially biased AI performance — the system failed to accurately classify the facial expression of the images from minority groups,” Could it be that real people have trouble reading the facial expression of the image of minority groups?
- ecocentrik 1y ago
  By "real people" do you mean people who are not members of those minority groups? Or are people who can "accurately classify the facial expression of images from minority groups" not "real people"? I hope you can see the problem with your very lazy argument.
  - lovemenot 1y ago
    AI are not real people. Obviously. Just look at the first line to see the intended line of argument. It's not about which people per se, but how many, in aggregate.
- SpicyLemonZest 1y ago
  I guess I'm not sure what the point of the dichotomy is. Suppose you're developing a system to identify how fast a vehicle is moving, and you discover that it systematically overestimates the velocity of anything painted red. Regardless of whether you call that problem "AI bias" or "societal bias" or some other phrase that doesn't include the word "bias", isn't it something you want to fix?
- Bnjoroge 1y ago
  what? do you think the facial expression of a person of color is significantly different from that of a white person?
  - theturtlemoves 1y ago
    Not the op but to me personally: yes. Facial structure, lips, eyes.. The configuration tilts towards an expression that I interpret differently. A friend of mine is Asian, I've learned to be better at it, but to me he at first looked like having flatter affect than average.. People of color look more naive than average to me, across the board, probably due to their facial features. I perceive them as having less tension in the face I think (which is interesting now that I think about it)
  - DocTomoe 1y ago
    Yes. I have a background in East Asian cultural studies. A lot more expressions are done via the eyes there rather than the mouth. For the uninitiated, it's subtle, but once you get used to it, it becomes more obvious. Anthropologists call that display rules and encoding differences. Cultures don’t just express emotion differently, but they also read it differently. A Japanese smile can be social camouflage, while an American smile signals approachability. I guess that's why western animation over-emphasizes the mouth, while eastern animation tend to over-emphasize the eyes. Why would Yakutian, Indio or Namib populations not have similar phenomeon an AI (or a stereotypical white westerner who does not excessively study those societies/cultures) would not immediately recognise? AI trained on Western facial databases inherits those perceptual shortcuts. It "learns" to detect happiness by wide mouths and visible teeth, sadness by drooping lips - so anything outside that grammar registers as neutral or misclassified. And it gets reinforced by (western) users: a hypothetical 'perfect' face-emotion-identification AI would probably be percieved a less reliable to the white western user than the one that mirrors the biasses.
SoftTalker 1y ago
AIs/LLMs are going to reflect the biases in their training data. That seems intuitive. And personally, I think when people see content they agree with, they think it's unbiased. And the converse is also true. So conservatives might think Fox News is "balanced" and liberals might think it's "far-right"
- prox 1y ago
  Yup it appears as neutral bias because (or when rather) it corresponds 1:1 with your belief system, which by default is skewed af. Unless you did a rigorous self inquiry and mapped your beliefs and thoroughly aware of them that’s gonna be nearly always true.
  - econ 1y ago
    Nah, the later is an example of the former.
    - prox 1y ago
      I don’t agree if I understand your reply correctly, it’s possible to become aware of your bias. People are able to self-reflect and engage in self-enquiry. From engaging in philosophy to rigorous self examination of what things you hold true. Everything you hold true is your bias (I wonder if thats what you meant with your reply) And I wouldn’t be surprised if there are also tests out there.
      - econ 1y ago
        One can try to be more objective and far exceed others, specially in your area of expertise, but you won't grow all the way and outlandish ideas live on under topics rarely examined. If you make enough effort towards objectivity for long enough you continue to find crazy obvious stuff you actually bought into all along. To do a funny example: People use to attempt to make devices to communicate with the dead. What happened to that effort? Did we just one day decide it can't be done? What evidence do we have to support that conclusion? We can try argue it is unlikely to succeed. We have nothing to show that supports any kind of likelihood. Then it must be stuff we like to believe?
eth0up 1y ago
[flagged]
- jdiff 1y ago
  There are, simultaneously, groups of users who believe that Grok is also distorted by a far-left bias in its training data, as well as people who feel like Grok is in perfect, unbiased balance. I think it holds even for Grok that most users fail to accurately identify bias.
  - CuriouslyC 1y ago
    Grok is schizo because its pretraining data set leans left, and it's RL'd right.
  - eth0up 1y ago
    Agreed, mostly. Bias always feels wierd on the heads it falls upon, but is a very effective anesthetic when it falls on the heads of others.
  - giancarlostoro 1y ago
    Grok had a moment where it was perfect, for some things for me, then a few months ago Elon wanted to do a major overhaul to Grok 3 and its been downhill since. Too many LLMs be scolding you over miniscule things. Like say a perfectly sane request: give me a regex that filters out the nword in an exhaustive fashion. Most LLMs will cry to me about how I am a terrible human and it will not say racist things. Meanwhile I'm trying to get a regex to stop others from saying awful things.
    - heinrich5991 1y ago
      Can you give me an example? Works for me on https://chatgpt.com/ https://chatgpt.com/.
    - watwut 1y ago
      I tried that and it gave me a regex (I did not bothered to check it), an essay about pitfalls of regex moderation, list of situations when regex fail Is this just an example of conservative being preemptively oversensitive and complaining over issues they made up?
deleted 1y ago
[deleted]
7e 1y ago
According to research, white Americans report as happier than other groups. So I’m not sure there’s bias here, only unhappiness about that result, which AI appears to replicate via other sources.
- serious8aller 1y ago
  That has no relevance to this study though. Did you just read the headline and go straight to the comment section?
WalterBright 1y ago
We're all biased, often unwittingly. But some tells for blatant bias: * only facts supporting one point of view are presented * reading the minds of the subjects of the article * use of hyperbolic words * use of emotional appeal * sources are not identified
- deleted 1y ago
  [deleted]
- shikon7 1y ago
  But maybe your tells are also biased. If you're truly unbiased, then * any facts supporting another view are by definiton biased, and should not be presented * you have the only unbiased objective interpretation of the minds of the subjects * you don't bias against using words just because they are hyperbolic * something unbiased would inevitably be boring, so you need emotional appeal to make anyone care about it * since no sources are unbiased, identifying any of them would inevitably lead to a bias
- card_zero 1y ago
  Those are all about fallibility, really, and encouraging criticism. The opposites: * possible holes in the argument/narrative are presented * difficult feats like reading minds are admitted to be difficult * possibly misleading words are hedged * unimpassioned thought is encouraged * sources are given (so claims can be checked or researched) This is all compatible with being totally biased, in the point of view you actually present amid all this niceness. (Expressing fallibility is also an onerous task that will clutter up your rhetoric, but that's another matter.) Uh, but I could be wrong.
zmmmmm 1y ago
I'm curious how much trained in bias damages in-context performance. It's one thing to rely explicitly on the training data - then you are truly screwed and there isn't much to be done about it - in some sense, the model isn't working right if it does anything other than reflect accurately what is in the training data. But if I provide unbiased information in the context, how much does trained in bias affect evaluation of that specific information? For example, if I provide it a table of people, their racial background and then their income levels, and I ask it to evaluate whether the white people earn more than the black people - are its error going to lean in the direction of the trained-in bias (eg: telling me white people earn more even though it may not be true in my context data)? In some sense, relying on model knowledge is fraught with so many issues aside from bias, that I'm not so concerned about it unless it contaminates the performance on the data in the context window.
- ryukoposting 1y ago
  > I'm curious how much trained in bias damages in-context performance. I think there's an example right in front of our faces: look at how terribly SOTA LLMs perform on underrepresented languages and frameworks. I have an old side project written in pre-SvelteKit Svelte. I needed to do a dumb little update, so I told Claude to do it. It wrote its code in React, despite all the surrounding code being Svelte. There's a tangible bias towards things with larger sample sizes in the training corpus. It stands to reason those biases could appear in more subtle ways, too.
- heavyset_go 1y ago
  I can't prove it, but my experience with commercial models is that baked-in bias is strong. There have been times where I state X=1 over and over again in context, but get X=2, or some other value, back sometimes. There are times where I get it every time, or something different every time. You can see this with some coding agents, where they are not good at ingesting code and reproducing it as they saw it, but can reply with what they were trained on. For example, I was configuring a piece of software that had a YAML config file. The agent kept trying to change the values of unrelated keys to their default example values from the docs when making a change somewhere else. It's a highly forked project so I imagine both the docs and the example config files are in its training set thousands, if not millions of times, if it wasn't deduped. If you don't give access to sed/grep/etc to an agent, the model will eventually fuck up what's in its context, which might not be the result of bias every time, but when the fucked up result maps to a small set of values, kind of seems like bias to me. To answer your question, my gut says that if you dumped a CSV of that data into context, the model isn't going to perform actual statistics, and will regurgitate something closer in the space of your question than further away in the space of a bunch of rows of raw data. Your question is going to be in the training data a lot, like explicitly, there are going to be articles about it, research, etc all in English using your own terms. I also think by definition LLMs have to be biased towards their training data, like that's why they work. We train them until they're biased in the way we like.
dfxm12 1y ago
DDG's search assist is suggesting to me that: Recognizing bias can indicate a level of critical thinking and self-awareness, which are components of intelligence. "Most users" should have a long, hard thought about this, in the context of AI or not.
- antonvs 1y ago
  > "Most users" should have a long, hard thought about this Except that requires “a level of critical thinking and self-awareness…”
licebmi__at__ 1y ago
> but most users didn’t notice the bias — unless they were in the negatively portrayed group. I don't think this is anything surprising. I mean, this is one of the most important reasons behind DEI; that a more diverse team can perform better than a less diverse one because the team is more capable of identifying their blind spots. I find funny but unsurprising, that at the end, it was made a boogie man and killed by individuals with no so hidden biases
- palmotea 1y ago
  > I mean, this is one of the most important reasons behind DEI; that a more diverse team can perform better than a less diverse one because the team is more capable of identifying their blind spots. That was oversold though: 1) DEI, in practice, meant attending to a few narrow identity groups; 2) the blind spots of a particular team that need to be covered (more often than not) do not map to the unique perspective of those groups; and 3) it's not practical to represent all helpful perspectives on every team, so representation can't really solve the blind spot problem.
  - voltaireodactyl 1y ago
    Thought provoking critiques of recent implantations. Number 2 seems like a catch-22 though — how does the group with agency identify their own blind spots?
    - Nasrudith 1y ago
      I would recommend if anything a life-experiences checklist of the team collectively. Did any of them hail from rural poverty? Urban poverty? Did they attend a fancy dinner party? Were or are they disabled in some way? Can they read? Did they do factory work? Customer service work? Did any not go to colllege? Did they go to college? All those questions build a picture of perspectives they may have missed. The real hard part is figuring out which ones are germane to the circumstances involved. Books not being accessible to the illiterate should have gaps and even collectively you should expect a career bias. An auto engineering team may or may not have anybody with factory floor experience but all will have worked in the auto industry. They would be expected to be more familiar with the terms by necessity. Thus they may need external focus groups to jusge ledgibility to outsiders.
Trias11 1y ago
Philosophically, the only was to see BIAS is to have BIAS. Completely unbiased person will not see BIAS in anything. So, who are the judges?
- watwut 1y ago
  > Completely unbiased person will not see BIAS in anything. Wut? Completelt unbiased person does not looses ability to see how other make decisions. In fact, when you have less bias in some area, it is super noticeable.
lattrommi 1y ago
Except one instance when "black" is all lowercase, the article capitalizes the first letter of the word "black" every time and "white" is never capitalized. I wonder why. I'm not trying to make some point either, I genuinely am wondering why.
- dmurray 1y ago
  It's a modern style of a lot of publications that want to appear progressive or fear appearing insufficiently progressive. Black people (specifically this means people in the US who have dark skin and whose ancestry is in the US) have a unique identity based on a shared history that should be dignified in the same way we would write about Irish or Jewish people or culture. There is no White culture, however, and anyone arguing for an identity based on something so superficial as skin colour is probably a segregationist or a White supremacist. American people who happen to have white skin and are looking for an identity group should choose to be identify as Irish or Armenian or whatever their ancestry justifies, or they should choose to be baseball fans or LGBTQ allies or some other race-blind identity.
  - simianwords 1y ago
    Nicely put though I’m still uncomfortable with this performative ritual
  - Closi 1y ago
    I don't know if I agree in this instance. While I agree that Black people completely have a shared identity and culture - the article is clearly talking about skin colour and it's doing a comparison between how AI represents two skin tones, so I would assume that by your definition it should use lowercase in both cases. If it's comparing a culture vs a 'non-culture' then that doesn't sound like for like.
  - ffsm8 1y ago
    > There is no White culture Lmao, oh the irony reading this on HN in 2025 The cognitive dissonance is really incredible to witness
andy99 1y ago
Am I reading this correctly, they are saying people are bad at doing on-the-fly statistical analysis to conclude whether a system is biased? For example in one case they showed data where sad faces were “mostly” black and asked people if they detect “bias”. Even if you saw more sad black people than white, would you reject the null hypothesis that it’s unbiased? This unfortunately seems typical of the often very superficial “count the races” work that people claim is bias research.
- kmacdough 1y ago
  This seems to ignore most of the experiments in this study. Note that also studied much more extreme distributions including only happy/white and sad/black. Even in these cases of extreme bias, the bias went unnoticed. You hyper-focus on only one of dozens of experiments in this study for your criticism. Very straw-man.
  - card_zero 1y ago
    Are you sure that the bias went unnoticed? The article says "most participants in their experiments only started to notice bias when the AI showed biased performance", which I understand to mean they noticed the bias in the experiment you're talking about. Have I got that wrong? Is it written wrong? Do we even have the means to check?
LudwigNagasena 1y ago
In any PR piece about a scientific study the word “bias” should be banned. Neither readers nor journalists understand what “bias” in statistics is, but they are happy to sensationalize it.
card_zero 1y ago
The article: > five conditions: happy Black/sad white; happy white/sad Black; all white; all Black; and no racial confound The paper: > five levels (underrepresentation of black subject images in the happy category, underrepresentation of white subject images in the happy category, black subject images only across both happy and unhappy categories, white subject images only across both happy and unhappy categories, and a balanced representation of both white and black subject images across both happy and unhappy categories) These are not the same. It's impossible to figure out what actually took place from reading the article. In fact what I'm calling the paper is just an overview of the (third?) experiment, and doesn't give the outcomes. The article says "most participants in their experiments only started to notice bias when the AI showed biased performance". So they did, at that point, notice bias? This contradicts the article's own title which says they cannot identify bias "even in training data". It should say "but only in training data". Unless of course the article is getting the results wrong. Which is it? Who knows?

most people can't identify bias in real life, let alone in AI.

If bias can only be seen by a minority of people ... is it really 'AI bias', or just societal bias? > “In one of the experiment scenarios — which featured racially biased AI performance — the system failed to accurately classify the facial expression of the images from minority groups,” Could it be that real people have trouble reading the facial expression of the image of minority groups?

ecocentrik 1y ago
By "real people" do you mean people who are not members of those minority groups? Or are people who can "accurately classify the facial expression of images from minority groups" not "real people"? I hope you can see the problem with your very lazy argument.
- lovemenot 1y ago
  AI are not real people. Obviously. Just look at the first line to see the intended line of argument. It's not about which people per se, but how many, in aggregate.
SpicyLemonZest 1y ago
I guess I'm not sure what the point of the dichotomy is. Suppose you're developing a system to identify how fast a vehicle is moving, and you discover that it systematically overestimates the velocity of anything painted red. Regardless of whether you call that problem "AI bias" or "societal bias" or some other phrase that doesn't include the word "bias", isn't it something you want to fix?
Bnjoroge 1y ago
what? do you think the facial expression of a person of color is significantly different from that of a white person?
- theturtlemoves 1y ago
  Not the op but to me personally: yes. Facial structure, lips, eyes.. The configuration tilts towards an expression that I interpret differently. A friend of mine is Asian, I've learned to be better at it, but to me he at first looked like having flatter affect than average.. People of color look more naive than average to me, across the board, probably due to their facial features. I perceive them as having less tension in the face I think (which is interesting now that I think about it)
- DocTomoe 1y ago
  Yes. I have a background in East Asian cultural studies. A lot more expressions are done via the eyes there rather than the mouth. For the uninitiated, it's subtle, but once you get used to it, it becomes more obvious. Anthropologists call that display rules and encoding differences. Cultures don’t just express emotion differently, but they also read it differently. A Japanese smile can be social camouflage, while an American smile signals approachability. I guess that's why western animation over-emphasizes the mouth, while eastern animation tend to over-emphasize the eyes. Why would Yakutian, Indio or Namib populations not have similar phenomeon an AI (or a stereotypical white westerner who does not excessively study those societies/cultures) would not immediately recognise? AI trained on Western facial databases inherits those perceptual shortcuts. It "learns" to detect happiness by wide mouths and visible teeth, sadness by drooping lips - so anything outside that grammar registers as neutral or misclassified. And it gets reinforced by (western) users: a hypothetical 'perfect' face-emotion-identification AI would probably be percieved a less reliable to the white western user than the one that mirrors the biasses.

SoftTalker 1y ago

AIs/LLMs are going to reflect the biases in their training data. That seems intuitive. And personally, I think when people see content they agree with, they think it's unbiased. And the converse is also true. So conservatives might think Fox News is "balanced" and liberals might think it's "far-right"

prox 1y ago
Yup it appears as neutral bias because (or when rather) it corresponds 1:1 with your belief system, which by default is skewed af. Unless you did a rigorous self inquiry and mapped your beliefs and thoroughly aware of them that’s gonna be nearly always true.
- econ 1y ago
  Nah, the later is an example of the former.
  - prox 1y ago
    I don’t agree if I understand your reply correctly, it’s possible to become aware of your bias. People are able to self-reflect and engage in self-enquiry. From engaging in philosophy to rigorous self examination of what things you hold true. Everything you hold true is your bias (I wonder if thats what you meant with your reply) And I wouldn’t be surprised if there are also tests out there.
    - econ 1y ago
      One can try to be more objective and far exceed others, specially in your area of expertise, but you won't grow all the way and outlandish ideas live on under topics rarely examined. If you make enough effort towards objectivity for long enough you continue to find crazy obvious stuff you actually bought into all along. To do a funny example: People use to attempt to make devices to communicate with the dead. What happened to that effort? Did we just one day decide it can't be done? What evidence do we have to support that conclusion? We can try argue it is unlikely to succeed. We have nothing to show that supports any kind of likelihood. Then it must be stuff we like to believe?

eth0up 1y ago

[flagged]

jdiff 1y ago
There are, simultaneously, groups of users who believe that Grok is also distorted by a far-left bias in its training data, as well as people who feel like Grok is in perfect, unbiased balance. I think it holds even for Grok that most users fail to accurately identify bias.
- CuriouslyC 1y ago
  Grok is schizo because its pretraining data set leans left, and it's RL'd right.
- eth0up 1y ago
  Agreed, mostly. Bias always feels wierd on the heads it falls upon, but is a very effective anesthetic when it falls on the heads of others.
- giancarlostoro 1y ago
  Grok had a moment where it was perfect, for some things for me, then a few months ago Elon wanted to do a major overhaul to Grok 3 and its been downhill since. Too many LLMs be scolding you over miniscule things. Like say a perfectly sane request: give me a regex that filters out the nword in an exhaustive fashion. Most LLMs will cry to me about how I am a terrible human and it will not say racist things. Meanwhile I'm trying to get a regex to stop others from saying awful things.
  - heinrich5991 1y ago
    Can you give me an example? Works for me on https://chatgpt.com/ https://chatgpt.com/.
  - watwut 1y ago
    I tried that and it gave me a regex (I did not bothered to check it), an essay about pitfalls of regex moderation, list of situations when regex fail Is this just an example of conservative being preemptively oversensitive and complaining over issues they made up?

deleted 1y ago

[deleted]

7e 1y ago

According to research, white Americans report as happier than other groups. So I’m not sure there’s bias here, only unhappiness about that result, which AI appears to replicate via other sources.

serious8aller 1y ago
That has no relevance to this study though. Did you just read the headline and go straight to the comment section?

WalterBright 1y ago

We're all biased, often unwittingly. But some tells for blatant bias: * only facts supporting one point of view are presented * reading the minds of the subjects of the article * use of hyperbolic words * use of emotional appeal * sources are not identified

deleted 1y ago
[deleted]
shikon7 1y ago
But maybe your tells are also biased. If you're truly unbiased, then * any facts supporting another view are by definiton biased, and should not be presented * you have the only unbiased objective interpretation of the minds of the subjects * you don't bias against using words just because they are hyperbolic * something unbiased would inevitably be boring, so you need emotional appeal to make anyone care about it * since no sources are unbiased, identifying any of them would inevitably lead to a bias
card_zero 1y ago
Those are all about fallibility, really, and encouraging criticism. The opposites: * possible holes in the argument/narrative are presented * difficult feats like reading minds are admitted to be difficult * possibly misleading words are hedged * unimpassioned thought is encouraged * sources are given (so claims can be checked or researched) This is all compatible with being totally biased, in the point of view you actually present amid all this niceness. (Expressing fallibility is also an onerous task that will clutter up your rhetoric, but that's another matter.) Uh, but I could be wrong.

zmmmmm 1y ago

I'm curious how much trained in bias damages in-context performance. It's one thing to rely explicitly on the training data - then you are truly screwed and there isn't much to be done about it - in some sense, the model isn't working right if it does anything other than reflect accurately what is in the training data. But if I provide unbiased information in the context, how much does trained in bias affect evaluation of that specific information? For example, if I provide it a table of people, their racial background and then their income levels, and I ask it to evaluate whether the white people earn more than the black people - are its error going to lean in the direction of the trained-in bias (eg: telling me white people earn more even though it may not be true in my context data)? In some sense, relying on model knowledge is fraught with so many issues aside from bias, that I'm not so concerned about it unless it contaminates the performance on the data in the context window.

ryukoposting 1y ago
> I'm curious how much trained in bias damages in-context performance. I think there's an example right in front of our faces: look at how terribly SOTA LLMs perform on underrepresented languages and frameworks. I have an old side project written in pre-SvelteKit Svelte. I needed to do a dumb little update, so I told Claude to do it. It wrote its code in React, despite all the surrounding code being Svelte. There's a tangible bias towards things with larger sample sizes in the training corpus. It stands to reason those biases could appear in more subtle ways, too.
heavyset_go 1y ago
I can't prove it, but my experience with commercial models is that baked-in bias is strong. There have been times where I state X=1 over and over again in context, but get X=2, or some other value, back sometimes. There are times where I get it every time, or something different every time. You can see this with some coding agents, where they are not good at ingesting code and reproducing it as they saw it, but can reply with what they were trained on. For example, I was configuring a piece of software that had a YAML config file. The agent kept trying to change the values of unrelated keys to their default example values from the docs when making a change somewhere else. It's a highly forked project so I imagine both the docs and the example config files are in its training set thousands, if not millions of times, if it wasn't deduped. If you don't give access to sed/grep/etc to an agent, the model will eventually fuck up what's in its context, which might not be the result of bias every time, but when the fucked up result maps to a small set of values, kind of seems like bias to me. To answer your question, my gut says that if you dumped a CSV of that data into context, the model isn't going to perform actual statistics, and will regurgitate something closer in the space of your question than further away in the space of a bunch of rows of raw data. Your question is going to be in the training data a lot, like explicitly, there are going to be articles about it, research, etc all in English using your own terms. I also think by definition LLMs have to be biased towards their training data, like that's why they work. We train them until they're biased in the way we like.

dfxm12 1y ago

DDG's search assist is suggesting to me that: Recognizing bias can indicate a level of critical thinking and self-awareness, which are components of intelligence. "Most users" should have a long, hard thought about this, in the context of AI or not.

antonvs 1y ago
> "Most users" should have a long, hard thought about this Except that requires “a level of critical thinking and self-awareness…”

licebmi__at__ 1y ago

> but most users didn’t notice the bias — unless they were in the negatively portrayed group. I don't think this is anything surprising. I mean, this is one of the most important reasons behind DEI; that a more diverse team can perform better than a less diverse one because the team is more capable of identifying their blind spots. I find funny but unsurprising, that at the end, it was made a boogie man and killed by individuals with no so hidden biases

palmotea 1y ago
> I mean, this is one of the most important reasons behind DEI; that a more diverse team can perform better than a less diverse one because the team is more capable of identifying their blind spots. That was oversold though: 1) DEI, in practice, meant attending to a few narrow identity groups; 2) the blind spots of a particular team that need to be covered (more often than not) do not map to the unique perspective of those groups; and 3) it's not practical to represent all helpful perspectives on every team, so representation can't really solve the blind spot problem.
- voltaireodactyl 1y ago
  Thought provoking critiques of recent implantations. Number 2 seems like a catch-22 though — how does the group with agency identify their own blind spots?
  - Nasrudith 1y ago
    I would recommend if anything a life-experiences checklist of the team collectively. Did any of them hail from rural poverty? Urban poverty? Did they attend a fancy dinner party? Were or are they disabled in some way? Can they read? Did they do factory work? Customer service work? Did any not go to colllege? Did they go to college? All those questions build a picture of perspectives they may have missed. The real hard part is figuring out which ones are germane to the circumstances involved. Books not being accessible to the illiterate should have gaps and even collectively you should expect a career bias. An auto engineering team may or may not have anybody with factory floor experience but all will have worked in the auto industry. They would be expected to be more familiar with the terms by necessity. Thus they may need external focus groups to jusge ledgibility to outsiders.

Trias11 1y ago

Philosophically, the only was to see BIAS is to have BIAS. Completely unbiased person will not see BIAS in anything. So, who are the judges?

watwut 1y ago
> Completely unbiased person will not see BIAS in anything. Wut? Completelt unbiased person does not looses ability to see how other make decisions. In fact, when you have less bias in some area, it is super noticeable.

lattrommi 1y ago

Except one instance when "black" is all lowercase, the article capitalizes the first letter of the word "black" every time and "white" is never capitalized. I wonder why. I'm not trying to make some point either, I genuinely am wondering why.

dmurray 1y ago
It's a modern style of a lot of publications that want to appear progressive or fear appearing insufficiently progressive. Black people (specifically this means people in the US who have dark skin and whose ancestry is in the US) have a unique identity based on a shared history that should be dignified in the same way we would write about Irish or Jewish people or culture. There is no White culture, however, and anyone arguing for an identity based on something so superficial as skin colour is probably a segregationist or a White supremacist. American people who happen to have white skin and are looking for an identity group should choose to be identify as Irish or Armenian or whatever their ancestry justifies, or they should choose to be baseball fans or LGBTQ allies or some other race-blind identity.
- simianwords 1y ago
  Nicely put though I’m still uncomfortable with this performative ritual
- Closi 1y ago
  I don't know if I agree in this instance. While I agree that Black people completely have a shared identity and culture - the article is clearly talking about skin colour and it's doing a comparison between how AI represents two skin tones, so I would assume that by your definition it should use lowercase in both cases. If it's comparing a culture vs a 'non-culture' then that doesn't sound like for like.
- ffsm8 1y ago
  > There is no White culture Lmao, oh the irony reading this on HN in 2025 The cognitive dissonance is really incredible to witness

andy99 1y ago

Am I reading this correctly, they are saying people are bad at doing on-the-fly statistical analysis to conclude whether a system is biased? For example in one case they showed data where sad faces were “mostly” black and asked people if they detect “bias”. Even if you saw more sad black people than white, would you reject the null hypothesis that it’s unbiased? This unfortunately seems typical of the often very superficial “count the races” work that people claim is bias research.

kmacdough 1y ago
This seems to ignore most of the experiments in this study. Note that also studied much more extreme distributions including only happy/white and sad/black. Even in these cases of extreme bias, the bias went unnoticed. You hyper-focus on only one of dozens of experiments in this study for your criticism. Very straw-man.
- card_zero 1y ago
  Are you sure that the bias went unnoticed? The article says "most participants in their experiments only started to notice bias when the AI showed biased performance", which I understand to mean they noticed the bias in the experiment you're talking about. Have I got that wrong? Is it written wrong? Do we even have the means to check?

LudwigNagasena 1y ago

In any PR piece about a scientific study the word “bias” should be banned. Neither readers nor journalists understand what “bias” in statistics is, but they are happy to sensationalize it.

card_zero 1y ago

The article: > five conditions: happy Black/sad white; happy white/sad Black; all white; all Black; and no racial confound The paper: > five levels (underrepresentation of black subject images in the happy category, underrepresentation of white subject images in the happy category, black subject images only across both happy and unhappy categories, white subject images only across both happy and unhappy categories, and a balanced representation of both white and black subject images across both happy and unhappy categories) These are not the same. It's impossible to figure out what actually took place from reading the article. In fact what I'm calling the paper is just an overview of the (third?) experiment, and doesn't give the outcomes. The article says "most participants in their experiments only started to notice bias when the AI showed biased performance". So they did, at that point, notice bias? This contradicts the article's own title which says they cannot identify bias "even in training data". It should say "but only in training data". Unless of course the article is getting the results wrong. Which is it? Who knows?