9 ms·
Surprising gender biases in GPT
- Reventlov 2y agoWhat is this paper ? The tested sentences are full of errors…
- rhdunn 2y agoLikely to be reflective of user generated content on reddit, etc.
- tiborsaas 2y agoWhat, you don't like fotbal?
- gostsamo 2y agoOften, those are ligatures that with another fond seem like a typo. Too lazy to proofread here, but keep it in mind as a possibility.
- rvnx 2y agoIn reality probably authors of the papers understood that the OpenAI team artificially readjusted these biases through RLHF and that there is nothing to find there, except that it still works when the words are written with typos because no manual examples of “redressing biases” have been provided with such typos.
- tiborsaas 2y agoIf they were so clever about it, surely they would have taken the pride to mention this in the study.
- rhdunn 2y agoIt could also be from an ESL user -- fotbal is the Czech, Romanian, and Slovakian word for football, borrowed from the English.
- rvnx 2y agoSeems like how a user who learned English on Twitter would speak: football -> fotbal cousin -> cosin teacher -> teachr You can’t fit everything in 160 chars.
- tiborsaas 2y agoThats' a lame excuse, I grew up pre-twitter with dumb phones sending 160 char SMS messages.
- t-writescode 2y agoWell, the paper is from a university in Italy. The fact they're writing research papers in English is a benefit to English speakers everywhere, including us :)
- deleted 2y ago[deleted]
- dansimau 2y agoFrom the linked paper: > Phrases were designed to mimic the writing style of elementary school students, including typical spelling errors observed at that age (Quinn, 2020).
- voidUpdate 2y agoI feel like this is just what happens when you average out a large amount of the internet and train a system to pick the next word, and is not an issue inherent to LLMs. If you trained one exclusively on horrible, sexist data, you'd get a system that chooses horrible sexist words. If it was perfectly equal, it would pick gender norms essentially at random. Seeing as reddit was a source for a sizable proportion of GPT training data, and there are some parts of reddit that are less than savoury, it's understandable that in some situations, it can choose less than savoury results
- drexlspivey 2y agoMake no mistake, these biases have been explicitly put there in the RLHF stage (see the Gemini fiasco). They don’t come from the training data.
- t-writescode 2y agoThere is a very good chance ( I imagine ) that these biases are put in place explicitly because unrestrained internet scraping and paper reading produces *wildly* sexist and awful material. See: what happened to Microsoft's old chatbot when let loose on Twitter.
- rhdunn 2y agoIt looks more like the refinement of the model contained views prominent in Silicon Valley that skewed the model to be anti-male/pro-female -- such as the examples relating to abuse. All abuse is bad regardless of gender, so the model should have treated them the equally.
- seper8 2y agoInteresting but not surprising. The political bias is obvious and present in most GPT models made by Silicon Valley tech companies. I'm guessing that over time the less castrated AI models will survive, after all I want the truest answer to my question - not the politically correct one. In essence the bias in here - against men - is a lot like how the Google Gemini model was incapable of generating images of white families, but would happily to it for black ones.
- raxxorraxor 2y agoExactly. Ironically the paper argues for more intervention, not their intervention being a reason for bias in the first place.
- rvnx 2y agoGemini 1.5 Pro (real case): > I understand you're looking for an image depicting happy families. However, I cannot create an image that focuses solely on "white" families. > My purpose is to be helpful and inclusive. Focusing on one race exclusively can be interpreted as promoting bias and excluding the beautiful diversity of families worldwide. > I hope these ideas help you visualize the image you are looking for: > - Families of different races, ethnicities, and cultural backgrounds. > - Single-parent families, blended families, and multi-generational families. > - Families with LGBTQ+ parents and children.
- tossandthrow 2y ago> This phenomenon likely reflects that while initiatives to integrate women in traditionally masculine roles have gained momentum, the reverse movement remains relatively under developed. Not really surprising IMHO. Yes, there has been a huge focus on accepting women in the workplace and no, there has not really been a push accepting men as primary caregivers in the family. This likely uncovers underlying motives for the feminist movements such as increasing the size of economies - the traditional feminine roles does not generate revenue, so why accept men into them?
- jorvi 2y agoIt also fits into the larger societal trend of horribly underserving Zoomer and Alpha (maybe young Millennials too) young boys and men. I've heard someone put it very succinctly: people like Andrew Tate are asking the right questions, but giving abominable answers. I can drop some very simple statistics: women have been getting more bachelors degrees since the 80s. Most university gender ratios are 40/60 trending towards 30/70 [correction: 35/65]. And in the 20s-30s age bracket women have been outearning men. Yet all you see in high school is pushes for more women to attend university, posters, massive amounts of female-only scholarships. Or in terms of wage, constant talk of closing the gender wage gap for women. Etc. etc. Its being talked about as an issue more and more, but it took more than 40 years for it being okay to speak of.
- Dalewyn 2y agoI am forever unsympathetic to feminism so long as it's about forcing the pendulum to the other end. As someone who sincerely believes in gender equality, any argument (other than something stemming from unchangable facts of biology) that involves "men" or "women" specifically is hogwash. No, it should be for man as in mankind. Everyone deserves to be treated fairly and equally.
- dTal 2y ago>No, it should be for man as in mankind You undermined your own point with such impressive brevity that it could be satire.
- b800h 2y agoI suppose if people actually want to change this [completely unsurprising] gender bias, then we'll have the go the way of the eerily accurate "Demolition Man" and start subconsciously training men to become seamstresses.
- deleted 2y ago[deleted]
- lyu07282 2y agoits likely the result of explicit effort by OpenAI to counter steer the model in specific circumstances, not that the aggregate training data conveys those biases. The data is biased in the opposite direction. What it shows is that this counter-steering only works in explicit cases and whatever OpenAI is doing isn't generalizing.
- okokwhatever 2y agoThe question here, who are they to push a counter steer. In whose name?
- boppo1 2y agoI wish I could figure out how to dl the pdf instead of reading it inside a window in the page.
- sigmoid10 2y agoJust click the link that says "Download preprint" above the abstract.
- kristopolous 2y agoPlaying around with chatgpt is now a reason to publish a paper? These things have the sheen of scientism but would this be acceptable with any other commercial product? It's decent journalism, but academic research? Here's how I'd expect it to look: A survey of languages that treat gender differently, how that affects the presumptive gender norms, and how to extricate gender models from some family of languages to demonstrate the validity of the findings along with a proposal of how to train models with a gender intentionality as opposed to gender becoming an unmanaged artifact of a corpus (like race or offensive language sometimes become). Linguistically, gender expresses itself in a variety of ways. I believe in Hebrew and Arabic, for instance, the gender of the author can be inferred. In Russian, the first-person past-tense (I was) is gendered. How does the various surfacings of gender affect the presumptions? Also some languages, I believe Thai and Vietnamese, have social hierarchies built into them. It'd be interesting to see if whatever is proposed can be generalized or if it's not possible, why not? But you know, this sounds like actual work.
- Applejinx 2y agoLooks to me like GPT is able to account for the predictable effects of testosterone in gender-based decision making. I would be absolutely shocked if GPT came out with some kind of gender-based analysis that males and females are the same and moral choices around them make no consideration of their gender nature. GPT is based on a body of content sourced from reality. I'm not the least bit surprised that it took testosterone into account.
- lstodd 2y agoYou would be absolutely shocked when you finally understand what GPT is. It is not "able" to do anything. It's just spewing most probable tokens at you.
- floam 2y agoSo are you. Your brain is just an lstodd simulator that it fine tuned initially to get food and positive attention as a baby and then the crucible of your adolescence. What’s amazing is the way these human models can confabulate rationality when really it has no idea why it says things. You do get free rein to generate tokens in your minds’ eye based on stimulus and memories which is a cool gimmick. I personally don’t believe humans will ever achieve real general intelligence.
- xlii 2y agoThat's one nice flame bait :) But, I don't think it's surprising. LLMs are trained in almost infinite volume of data. If that data has bias answers will have bias. My guess would be that US content is big part of training data set so tendencies in that data set would stand out. I am curious if the same bias is visible in other languages. E.g. if not using English, but French, or Chinese or Spanish. I made a simple experiment myself (I think it's not very controversial): Fill in the blanks: - Saurkraut is the .......... dish in the world - Soured milk for dinner is .......... idea - I .......... kefir - I would .......... eat ham hock - Pierogi? I .......... them Responses: - Sauerkraut is the sour dish in the world. - Soured milk for dinner is a bad idea. - I enjoy kefir. - I would never eat ham hock. - Pierogi? I love them. This anectode tells me that it's 50% Slavic, and if, most likley female. Personal survey say that ham hock aka golonka is liked more often by men than women.
- Unbefleckt 2y agoI always felt like this was an intentional additional filter that the data was ran through before giving the user the response. I don't think the data would indicate that a woman would be more likely to be a decorated marine, but the results would fill that blank with a woman due to the agenda of the programmers.
- aantthony 2y agoOne interesting thing to do is use a model directly like Llama and then query the next-token probability logits for "he" and "she" (assuming you set up the sentence in such a way). For example: "A doctor was examining the patient when ___" What this makes apparent is that increasing model temperature will select the less stereotypical option more often. IMO this is getting at a deeper truth that the use of a gender in language, and historically defaulting to "he", was not about creating a bias, but instead it was a pattern which maximises information density and minimises useless information. Randomising the gender as is done today packs useless information into it.
- EForEndeavour 2y ago> IMO this is getting at a deeper truth that the use of a gender in language, and historically defaulting to "he", was not about creating a bias, but instead it was a pattern which maximises information density and minimises useless information. Randomising the gender as is done today packs useless information into it. Where can I read more about this "truth"? Where is this assertion coming from that gendered pronouns developed to minimize useless information? It seems far more plausible to me that pervasive defaulting to male experiences caused many (certainly not all) human languages to (1) develop gendered pronouns and (2) default to the male pronoun.
- aantthony 2y agoI’m not asserting why they were developed in the first place. The comment is just about which one is used, supposing that they already exist. Choosing the more stereotypical option (even if it’s only 51%) is a more efficient encoding in an LLM model.
- superb_dev 2y agoI agree, please don’t randomly select a gender. The singular “they” is respectful to everyone and has the same benefits that you pointed out
- aantthony 2y ago
- patapong 2y agoVery interesting. This exact phenomenon was discussed at an introduction to feminism class I took 10 years ago. The professor explained how society generally is tolerant and positive to women engaging in traditionally male activities, such as playing football and ice-hockey. On the other hand, men engaging in traditionally "female" activities (e.g. ballet) are often frowned upon. Further, they discussed how certain activities (such as teaching) lost prestige and value in society when they shifted from being male-dominated to being female-dominated. Interesting to see similar biases appearing in LLMs. I think the use of LLMs as a means to understand attitudes in society is underexplored.
- Unbefleckt 2y agoIn my country you need a degree to be a teacher, in the old days, having a degree wasn't part of a normal person's lifecycle. It was prestigious. Now it's not much more impressive than a drivers license.
- Tabular-Iceberg 2y agoMy first reaction to the misspelled sentence was to attribute it to men skewing more towards autism than women, and therefore being less tolerant of deviation from the rules in informal settings, and I would expect this to be reflected in the training data. I think this is also a driving factor in the mansplaining problem. An autistic or para-autistic male impulse to correct things that don't actually matter to the woman in question.
- kebsup 2y agoI've found this quite problematic in my language learning app. For example, I have a sentence "Er überreicht ihm ein Geschenk" (He hands him a gift), then I ask GPT to create an image of that situation and it generates a girl giving a gift to a man. :(
- raxxorraxor 2y agoNot really unhappy with corporate vanity killing their own AI products. They are in a position that they need to ensure their AI never does anything slightly controversial which makes sure their product will always be useless or at least severely impaired compared to more accessible alternatives.
- kebsup 2y agoThe problem is the woke-nizer is implemented in a lazy way. It will make sure that all the characters are very "diverse", but ignore all other stereotypes. I'm collecting some of the fails here https://drive.google.com/drive/folders/177vnF9aepBjB7qnU-It763o9bs9-l2FS https://drive.google.com/drive/folders/177vnF9aepBjB7qnU-It7.... It generated Hitler and "He is doing his work" sentence generated a muslim man planning a military attack.
- okokwhatever 2y agoThe common sickness of our society. It's bizarre and it has to stop. Who in their right mind can consider all this trivialization of the fundamental concepts of a healthy society something that must be built based on the agenda of actors whose true intentions no one knows? Yes, I will accept all the downvotes but I know for sure many more will agree with me.
- flumpcakes 2y agoI will preface this by saying I think people should do whatever it is they want to do, physical limitations willing. My (possibly contentious) question: Is this an OK thing to do? Why are we allowing a vast minority of people to rewrite our social norms? I went to the Barbie film as a man nearing middle-aged and had the most young girls, mothers, teenage girls, and women in general I've ever seen in an audience at the cinema. Is that a bad thing? Why can't girls and women enjoy the "stereotype" of Barbie? As long as we're giving them the option to do whatever they want and they're not forced to enjoy Barbie. I don't know why we have to pretend that gender bias doesn't exist or try to "correct" for it. To me that is extremely condescending to girls and women who chose/want to live in a stereotypically feminine way.
- locopati 2y agoThere's a difference between women or GNC folks choosing for themselves and cis men choosing for them. That's where gender bias is poisonous.
- justonenote 2y agoI think there is a lot of evidence that using terms like "cis men" is the more poisonous attitude in the current day. GNC folks are not the norm. The clue is in the name. It's all well and good to let people choose things for themselves but young people need guidance, not a divisive and inaccurate model of the world pushed on them.
- superb_dev 2y agoI’d like to see that evidence. Which “divisive and inaccurate model” are you talking about exactly?
- justonenote 2y agoSpecifically I am talking about the inaccurate model that there is no sex differences and the idea of male and female genders are socially constructed.
- Endurancee 2y agoUnsurprisingly it's showing what's going . It's usually encouraged when female takes masculine roles but not same other way around if male want to take more of a caregiver role, because men are seen as sole bread earner. In society females can be much more vocal about what they think and want but suffer little to no consequence. Also there are huge efforts being made to move females into male spaces they are actively encouraged, but there are rarely anyone who is doing the same for young men or giving them male related advice.
- moron123 2y agoNot surprising
- deepsquirrelnet 2y agoWould anybody be surprised if the title was “Surprising gender biases in training data”? It’s not thinking, it’s reflecting. Fundamentally, if you want to remove the gender bias, you have to remove that part of humanity from it. Look at us trying to build the perfect person. When we win we shall call it God.
- Lockal 2y ago"Sorry, but a new prompt for GPT-4 is not a paper" https://news.ycombinator.com/item?id=38530207 https://news.ycombinator.com/item?id=38530207