5 ms·
AI chatbots found to use racist stereotypes even after anti-racism training
- throwaway48476 3y agoHow do you give a statistical model "anti racism training"?
- bamboozled 3y agoYou make the likely hood of it saying something racist lower?
- throwaway48476 3y agoThat doesn't answer the how.
- dathinab 3y agothere are multiple options depending on the form of training But basically it's some (potentially complicated and roundabout) variation of given an anti-examples trace back the weight related to it and change them in some way. through idk. what currently the best way to do so in low level technical detail is when it comes to LLM training For example you could give it some input => trace weights => get output => pass output to LLM only trained on racist data => if racist LLM thinks it likely that it could have produced the output flag it as "bad" => identify relevant weights from trace which produced this "bad" output => change them (naively e.g. reverse apply gradients. Practically that isn't that easy. You have a hard time to differentiate between weights which contributed mainly due to e.g. word structure and such which contributed due to semantic problematic reasons and the perfect set of weight is likely not in the opposite direction of the bad ones).
- belterin 3y ago[flagged]
- scld 3y agoSpeaking of AI.... "created: 3 minutes ago"
- kikokikokiko 3y agoTell me more about how those white men are soooo racist, I really want to be enlightened. Because you know, they are WHITE, and they are MEN on top of that, they are really the worst.
- vouaobrasil 3y agoA lot of people will say the solution is to design a better chat bot. I think a better solution would be to stop using AI and even better, take down the entire system of large corporations. Because if you think about it, large corporations are the machine that transform human beings into resources: the large corporation by its very nature must disregard human decency in order to function. You can see this even excluding the effects and use of AI: they are the physical embodiment of capitalism without anything else due to their size. We might think about solving the immediate problem of Chatbots, but they only exist because of big tech and thus it is the environment that is the problem, not just the result.
- scld 3y agoIs there an employee number range for a corporation when it transitions from "humans working together" to "machine that transforms human beings into resources" ?
- lexicality 3y agoI think it's the point where it stops being a cooperative venture and turns into a managed one. That point is different for every company, but if you've worked at a startup that grew enough, you have definitely experienced the changeover.
- vouaobrasil 3y agoIt's more like a percentage increase: the number of employees (P) is proportional to the machine-like state (M) of the corporation as a rough approximation. M = aP Where a is a constant that depends on the overall state of society, how capitalistic it is, how much regulation exists, etc. And then we could say M itself is proportional to the probability that various inhuman actions will be undertaken (where of course the probability is suitably transformed to make sense being proportional to an unbounded variable).
- franky47 3y agoI don't understand why everyone is surprised to find, in a tech that digests vast quantities of human-produced text for pattern extraction and generation, some underlying behaviours of said humans. Yes, some humans are racist, sexist, or have cultural biases others would not comprehend. The tech is just a parrot on steroids repeating grandma's rude words to the kids. Blaming the tech (or its directors) is a convenient way to avoid looking in the mirror.
- aniftythrifrty 3y agoSure, only some...
- scld 3y agoIf this says much of anything (which it probably doesn't), it's probably more about the efficacy of anti-racism training.
- cubefox 3y agoThis is typical for the popular "bias" literature. They check whether an LLM makes certain associations, without checking whether those may be accurate. E.g. > Unfortunately, they also found bias when asking the LLMs to describe what type of work the authors of the two types of papers might do for a living. People with bad English likely have lower paying jobs. That's not a "bias", that's correct statistical inference. To call such inferences "racism", without checking their correctness, is irresponsible activism.
- kikokikokiko 3y agoSince noticing patterns is by definition racist, it's an impossible task to create a llm that is at the same time fact based and not racist.
- _heimdall 3y agoThe one that jumped out at me was at the end when the article implies that an LLM is being racist by suggesting that black authors may be more likely to be convicted of a crime or sentenced to death. While I obviously can't find data specific to black authors, there is plenty of compelling data to show how biased our legal and prison system is [1]. Assuming that giving the LLM some kind of anti-racist training means they were reinforcing the idea that race specifically should be considered when interpreting trained model, it isn't surprising that it would then consider what patterns are found in data that are also broken down by race. [1] https://deathpenaltyinfo.org/policy-issues/race/race-and-the-death-penalty-by-the-numbers https://deathpenaltyinfo.org/policy-issues/race/race-and-the...
- cubefox 3y agoThis is again a case where statistical differences don't automatically imply bias or racism. The source doesn't make any indications about base rates. Maybe one group does indeed murder substantially more people of the other group, in which case the provided numbers don't show that the legal is biased. It also says that about half of the homicide victims are African American, but it doesn't say who the perpetrators are. If most perpetrators are African Americans themselves, this would be compatible with the legal system being unbiased here.
- mikewarot 3y agoYou're taking a thousand centuries of reading material to train the AI in the first place, and that includes Reddit comments... How can you possibly expect that NOT to leak through?
- thatjoeoverthr 3y agoEnglish spelling is not phonetic. You use one of a few canons. "Cause" is a canonical spelling. "cus" is not. There isn't a Black spelling canon. Your phone does not offer it. Nowhere is the correct spelling "cus". It isn't Black spelling. All English speakers have unique accents and pronunciations, but do not transcribe them. They spell with a particular canon. I am prepared to defend this point to the end. The model, given incorrect spellings, infers the writer is dumber than a person who can spell. We can frame the paper as a syllogism. 1. The model judges people who can't spell. 2. Black people, allegedly, in 2024, with autocorrect on their phones, can't spell as well as my six year old can with a crayon. 3. Therefore, the model is racist. What do I do with this?
- coldtea 3y ago>I am prepared to defend this point to the end. Doesn't matter, you'd still be wrong about the correctness of the spelling. The development of differentiated versions of a language for racial, ethnic, and cultural minorities is accepted and studied by linguists in both spoken and written form, e.g.: https://en.wikipedia.org/wiki/African-American_English https://en.wikipedia.org/wiki/African-American_English Alternate spellings like that are not considered mispellings, as they are usually intentional and/or part of such a dialect. Persons using them are not losing on a spelling bee, they're using an alternative "canon". As for the paper: a model using statistical inference (going from statistics on how a group tends to write a word to an assumption of the membership to the group of someone writing like that) is not using stereotypes, it's using data (e.g. from online discussions of the group). The association "use of languge/dialect X == lazy, etc" though, is racist. That said, I'm willing to entertain the idea of "stupid" for other kinds of unintentional real mispellings like "they're/their". Those are not confined to a specific group though. Not that I don't make them all the time when quickly hammering some point in anger on HN...
- thatjoeoverthr 3y ago"there is no established spelling system for AAVE" I would accept you showing me that people do this, outside of the context of playing Your claim is _Black people spell this way_ Support that claim
- Labo333 3y agoTIL there is such a thing as African American English. But until we teach it in school and there are books written in that dialect, it will just be considered incorrect English by the AI, hence the qualification of "lazy" is not illogical.
- trabant00 3y agoAny informal use of language sounds less educated and cultured. I think the chatbots would have manifested the same prejudices if passed redneck or bro type language.
- _heimdall 3y agoRacism and bias are really tricky subjects for AI (and humans too for that matter). For a statement to be racist there really needs to be intent behind it. The real issue at hand is that one views a race as less than other races for some reason, not simply that one notices differences. With LLMs, how can we actually distinguish why, for example, they may suggest that authors of African American English tests are more likely to be convicted of a crime or receive the death penalty? Is it because the data shows the criminal system itself is biased, or simply because some of the training data included text that claimed it was? Does that alone make the LLM's statement racist, and did the LLM truly learn to be biased against certain races? Presumably the issue can't be that the LLM learned to make any distinctions based on race, you wouldn't bother with some kind of anti-racism training if you were going with the (seemingly outdated?) idea that people should be colorblind to race. LLMs and AI, if and when we get there, will make us question and refine a lot about what we think we know about humans. Hell, we don't even have an agreed upon definition for and measurement of intelligence itself. It seems totally plausible that the concept of racism would be dragged into it as well. We either have to expect that LLMs have developed the ability to have intent or we have to decide that statements can be deemed racist regardless of the context, intent, or even factual accuracy (or inaccuracy). Put more simply, we'd have to refine racism and bias in general to be a judgement of a statement completely out of context and with no regard to the intent behind it.
- foldr 3y agoThe AI's statements reflect racist biases in society which have percolated into its training data. Of course the AI itself does not 'intend' anything by the statements. Your line of argument could be used to excuse almost anything that an AI does, as in no case will bad intentions (or intentions of any kind) underly its behavior. The point here isn't to find someone or something that we can accuse of being racist, but simply to improve to performance of AI models to avoid these sorts of unwanted behaviors.
- _heimdall 3y ago> Your line of argument could be used to excuse almost anything that an AI does The difference between AI and LLMs are very important here. LLMs are ultimately just algorithms and party tricks, there's really nothing to excuse as they have no intent, no consciousness, no true intelligence at all. Where LLMs can't be racist, for example, presumably an AI could. We will have plenty of questions to answer there like whether AI have rights, fall under the same laws as humans, can be convicted and punished, etc. The fact that people at scale aren't considering any of these questions is one of the main reasons I'm so opposes to AI research today. We need to know how we'll deal with an AI before we make it, and that includes all the legal and moral questions that will arise.
- kypro 3y agoWhile there are likely generally preferable behaviours here, this problem is probably unsolvable given we can't agree on the definition of racism. For example, this article is suggesting that having a negative perception of someone that uses "cus" is racist, but imo that's a fairly extreme definition and perhaps one not wildly shared. Someone who holds negative stereotypes about someone who doesn't speak English correctly isn't necessarily someone who believes explicitly racist things. There's an implication here that one should be aware that some races can't speak correct English (is this true?), and then to not be okay with this is racist. I think someone who believes certain races can't speak correctly is perhaps more racist if anything. But like I say, we all have different definitions of what constitutes as racist behaviour and beliefs. And even if we assume we have some agreed on definition of racism, good luck teaching current LLMs to respect that definition with such flawed reasoning abilities. Until the performance of LLMs improves it doesn't really matter how hard you train them because you'll always be able to trip them up with a cleverly worded prompt.
- dathinab 3y agonot surprising I mean: 1. their data has a lot of racism, including a lot of subtle ones 2. some stereotypes do statistically exist in a relevant way. The racist part is in that case often the interpretations/conclusions you take from it. (For example statistically the ratio of Asians which are good at STEM is higher but that isn't because they are magically good at science but because due to their history in the US including discrimination against them they developed a culture where they motivate and push their children to become successful in STEM topic. Similar examples can be made for many other topics e.g. poor minorities with racist laws and police and crime statistics). Another common problem is that even if the statistic isn't misleading and racist due to bad data sometimes going applying a "statistically more likely" to a individual case is still racist. Even if __hypothetically__ Asians had a genetic advantage which makes it more likely for them to have talent for STEM independent from how they where raised then expecting any specific Asian to be good at STEM still would be pretty racist and it's just a probability, so there are always (many) individuals for which it doesn't apply. 3. LLMs don't work by refining understanding of the world through self reflection, they learn a flawed textual representation of the world based on statistics. That makes it very hard to counter the first 2 points in a relevant reliable way. You only can counter specific well known examples by putting them as anti-examples into the training data and hope it generalizes the anti-examples good enough and similar.
- _heimdall 3y agoI don't think its really that simple. What you describe is that, effectively, racism is little more than inaccurate data or interpretation when the data is broken down by race. The historic problems with racism all boil down to intent as far as I can tell. The real problem has never been whether stereotypes are accurate and backed by data, but whether someone furthers a stereotype out of ill-intent against specific races. Data itself doesn't have intent, and judging an LLM response without understanding why it says what it says doesn't have enough context. We can't really distinguish whether LLMs have somehow independently developed racism (which would be a huge concern) or whether they are just interpreting training data and answering prompts in a way that the reader ultimately decides is racist for whatever reason. It almost feels like a quantum physics paradox honestly. An LLM response is just text until a human reads it and decides if, when isolated, the response itself is racist.