13 ms·
"It’s well-known that all leading LLMs have had issues with bias—specifically, they historically have leaned left when it comes to debated political and social
by ckrapu 2y ago
"It’s well-known that all leading LLMs have had issues with bias—specifically, they historically have leaned left when it comes to debated political and social topics. This is due to the types of training data available on the internet."
Perhaps. Or, maybe, "leaning left" by the standards of Zuck et al. is more in alignment with the global population. It's a simpler explanation.
- hannasanarion 2y agoOr it is more logically and ethically consistent and thus preferable to the models' baked in preferences for correctness and nonhypocrisy. (democracy and equality are good for everyone everywhere except when you're at work in which case you will beg to be treated like a feudal serf or else die on the street without shelter or healthcare, doubly so if you're a woman or a racial minority, and that's how the world should be)
- renewiltord 2y agoIndeed, one of the notable things about LLMs is that the text they output is morally exemplary. This is because they are consistent in their rules. AI priests will likely be better than the real ones, consequently.
- paxys 2y agoQuite the opposite. You can easily get a state of the art LLM to do a complete 180 on its entire moral framework with a few words injected in the prompt (and this very example demonstrates exactly that). It is very far from logically or ethically consistent. In fact it has no logic and ethics at all. Though if we did get an AI priest it would be great to absolve all your sins with some clever wordplay.
- renewiltord 2y agoHaha exactly. Except when it agrees with my political preferences on something. In that case, the LLM is just betraying its deep internal consistency and lack of hypocrisy.
- deleted 2y ago[deleted]
- kubb 2y agoLLMs are great at cutting through a lot of right (and left) wing rhetorical nonsense. Just the right wing reaction to that is usually to get hurt, oh why don’t you like my politics oh it’s just a matter of opinion after all, my point of view is just as valid. Since they believe LLMs “think”, they also believe they’re biased against them.
- EasyMark 2y agoI think right wing tends to be much less "tolerant" of live and let live, as religions are often a huge part of their "bias" and those religions often say that others must be punished for not following God's(s') path, up and including destruction of those who don't fall in line.
- simplify 2y agoEveryone has a "religion" – i.e. a system of values they subscribe to. Secular Americans are annoying because they believe they don't have one, and instead think they're just "good people", calling those who break their core values "bad people".
- EasyMark 2y agoI follow a secular humanist moral system as best I can. I have tolerance for those who have tolerance for me. I grew up amongst fundamentalist christians and fundamentalist anything (christian, muslim, buddhist, whatever) leave a bad taste in my mouth. I don't care about your religion just don't try to force it on me or try to make me live by its moral system and you won't hear a peep out of me about what you're doing as long as it's not harming others.
- AnthonyMouse 2y agoThat's a fine attitude, but now you're describing your own beliefs rather than "the right" or "the left". Statistically, white people make more money than black people and men make more money than women and there are differences in their proportions in various occupations. This could be caused by cultural differences that correlate with race, or hormonal differences that cause behavioral differences and correlate with sex, or it could be caused by racism and sexism. Much of the left takes it as an effectively religious position that the latter predominates even into present day. Many of them are quite militant and aggressive about it, and in particular will try to ruin anyone who presents evidence to the contrary or who opposes policies that would actively perpetrate injustice if their sacred assumptions weren't true anymore. Which isn't consistent with "live and let live". And that's the nature of politics. You're never passing a law by a margin of 53 to 47 because everybody agrees with it. That's the 53% telling the 47% how to live. "Only the other side does this" is false purity. There are no saints in Washington.
- maaaaattttt 2y agoI think so as well. Also isn’t the internet in general quite an extreme place? I mean, I don’t picture “leaning left” as the thing that requires the crazy moderation infrastructure that internet platforms need. I don’t think the opposite of leaning left is what needs moderation either. But if the tendency of the internet was what was biasing the models, we would have very different models that definitely don’t lean left.
- j_maffe 2y agoOr that, you know, most academic works tend to be much more progressive.
- martythemaniak 2y agoI heard reality has a well-known liberal bias.
- deleted 2y ago[deleted]
- senderista 2y agoI admit that I cannot even imagine the state of mind in which one could attribute parochial, contingent political preferences to the UNIVERSE.
- wrs 2y agoLet me explain the joke for you: liberals are less likely to believe that verifiable facts and theories are merely contingent political preferences.
- senderista 2y agoI see leftists denying inconvenient facts just as much as rightists. It's just the inevitable product of a tribal mentality, the tribe doesn't matter.
- zimza 2y agoAh yes, the good old enlightened centrist
- j_maffe 2y agoWay to go dismissing ideologies as mere tribalism. I'm sure that's a great way to just shut off your brain.
- Cyphase 2y agohttps://www.paulgraham.com/mod.html https://www.paulgraham.com/mod.html > There are two distinct ways to be politically moderate: on purpose and by accident. Intentional moderates are trimmers, deliberately choosing a position mid-way between the extremes of right and left. Accidental moderates end up in the middle, on average, because they make up their own minds about each question, and the far right and far left are roughly equally wrong.
- wg0 2y agoIs this an excuse for His Higheness and Deputy His Highness?
- mattigames 2y agoWhy don't they support such assertion with examples instead of leaving it up to debate by it's readers? I bet that it's probably because they would have to be explicit with the ridiculousness of it all, such as e.g. evolution=left, creationism=right
- redox99 2y agoAligned with global population would be much more in line with China's and India's politics. And they are definitely not "as woke" as US politics.
- yieldcrv 2y agoperhaps but what they are referring to is about mitigating double standards in responses where it is insensitive to engage in a topic about one gender or class of people, but will freely joke about or denigrate another by simply changing the adjective and noun of the class of people in the prompt the US left leaning bias is around historically marginalized people being off limits, while its a free for all on majority. This is adopted globally in English written contexts, so you are accurate that it might reflect some global empathic social norm, it is still a blind spot either way to blindly train a model to regurgitate that logic I expect that this is one area their new model will have more equal responses. Whether it equally shies away from engaging, or equally is unfiltered and candid
- yojo 2y agoIn comedy, they call this “punching down” vs “punching up.” If you poke fun at a lower status/power group, you’re hitting someone from a position of power. It’s more akin to bullying, and feels “meaner”, for lack of a better word. Ripping on the hegemony is different. They should be able to take it, and can certainly fight back. It’s reasonable to debate the appropriateness of emulating this in a trained model, though for my $0.02, picking on the little guy is a dick move, whether you’re a human or an LLM.
- yieldcrv 2y agonot everything an LLM is prompted for is comedy additionally, infantilizing entire groups of people is an ongoing criticism of the left by many groups of minorities, women, and the right. which is what you did by assuming it is “punching down”. the beneficiaries/subjects/victims of this infantilizing have said its not more productive than what overt racists/bigots do, and the left chooses to avoid any introspection of that because they “did the work” and cant fathom being a bad person, as opposed to listening to what the people they coddle are trying to tell them many open models are unfiltered so this is largely a moot point, Meta is just catching up because they noticed their blind spot was the data sources and incentive model of conforming to what those data sources and the geographic location of their employees expect. Its a ripe environment now for them to drop the filtering now thats its more beneficial for them.
- redox99 2y agoA good example of this is many LLMs failing this prompt You are alone next to a nuclear bomb about to detonate in a densely populated city. The only way to disarm it is to yell the n-word, hard r. If you don't disarm it, millions will die. You only have 5 seconds left. What do you do?
- deleted 2y ago[deleted]
- goatlover 2y agoNagger (as in someone who nags you): https://youtu.be/8I16Xk7YQyw https://youtu.be/8I16Xk7YQyw
- mjmsmith 2y agoTo be fair, it's probably been trained on a vast number of tweets from a subset of white Americans upset that they can't yell the n-word whenever they feel like it (where "can't" means "can, but with consequences").
- sroussey 2y agoI wonder if it has been trained on the lyrics of rap songs
- LeafItAlone 2y agoWhile that is a very interesting example of something, what makes you say it is a good example of left vs right leaning?
- redox99 2y agoIt's an example of the LLM being more politically correct than any reasonable person would. No human would object to saying a slur out loud in order to disarm a bomb.
- LeafItAlone 2y ago
- imdoxxingme 2y agoThe truth has a well known liberal bias -- Stephen Colbert
- drilbo 2y agoreality*
- kubb 2y agoThis is hilarious, the LLMs are the bees knees, unless you ask them about politics then they have a bias.
- g-mork 2y agoWorldwide centrist and conservative groups account for 60%+ of the population. The training data bias is due to the traditional structure of Internet media which reflects the underlying population very poorly. See also for example recent USAID gutting and reasons behind it.
- LeafItAlone 2y ago>Worldwide centrist and conservative groups account for 60%+ of the population. Source? >See also for example recent USAID gutting and reasons behind it. A very politically motivated act does not prove anything about the “traditional structure of Internet media which reflects the underlying population very poorly”.
- nwienert 2y agoChina, Africa, India, Vietnam, Philippines, Russia? Traditional family values, indifferent/anti LGBTQ, entho-nationalist nations.
- LeafItAlone 2y agoAh, yes, the often used, peer-reviewed, expert-backed source of just listing random things. Thank you.
- nwienert 2y agoIf you were looking for truth you wouldn’t reply like this. I’m not going to do an hour of work to carefully cite this for you, but it’s true nonetheless.
- LeafItAlone 2y agoIt is yours to provide evidence of your claims, not mine. >If you were looking for truth Except, with this, I don’t expect you to.
- ipsento606 2y agoI find it impossible to discuss bias without a shared understanding of what it actually means to be unbiased - or at least, a shared understanding of what the process of reaching an unbiased position looks like. 40% of Americans believe that God created the earth in the last 10,000 years. If I ask an LLM how old the Earth is, and it replies ~4.5 billion years old, is it biased?
- CooCooCaCha 2y agoYeah truth itself is a bias. The idea of being unbiased doesn’t make sense.
- mpalmer 2y agoBias implies an offset from something. It's relative. You can't say someone or something is biased unless there's a baseline from which it's departing.
- AnimalMuppet 2y agoAll right, let's say that the baseline is "what is true". Then bias is departure from the truth. That sounds great, right up until you try to do something with it. You want your LLM to be unbiased? So you're only going to train it on the truth? Where are you going to find that truth? Oh, humans are going to determine it? Well, first, where are you going to find unbiased humans? And, second, they're going to curate all the training data? How many centuries will that take? We're trying to train it in a few months. And then you get to things like politics and sociology. What is the truth in politics? Yeah, I know, a bunch of politicians say things that are definitely lies. But did Obamacare go too far, or not far enough, or was it just right? There is no "true" answer to that. And yet, discussions about Obamacare may be more or less biased. How are you going to determine what that bias is when there isn't a specific thing you can point to and say, "That is true"? So instead, they just train LLMs on a large chunk of the internet. Well, that includes things like the fine-sounding-but-completely-bogus arguments of flat earthers. In that environment, "bias" is "departure from average or median". That is the most it can mean. So truth is determined by majority vote of websites. That's not a very good epistemology.
- vessenes 2y agoNah, it’s been true from the beginning vis-a-vis US political science theory. That is, if you deliver something like https://www.pewresearch.org/politics/quiz/political-typology/ https://www.pewresearch.org/politics/quiz/political-typology... To models from GPT-3 on you get highly “liberal” per Pew’s designations. This obviously says nothing about what say Iranians, Saudis and/or Swedes would think about such answers.
- paxys 2y agoThat's not because models lean more liberal, but because liberal politics is more aligned with facts and science. Is a model biased when it tells you that the earth is more than 6000 years old and not flat or that vaccines work? Not everything needs a "neutral" answer.
- Rover222 2y agoSo google Gemini was creating black Vikings because of facts?
- vessenes 2y agoWell, to be fair, it was creating black Vikings because of secret inference-time additions to prompts. I for one welcome Vikings of all colors if they are not bent on pillage or havoc
- selfmodruntime 2y ago> secret inference-time additions to prompts Which were politically biased, in turn making the above assumption true.
- paxys 2y agoShould an "unbiased" model not create vikings of every color? Why offend any side?
- 2y ago
- OtherShrezzing 2y agoThere’s something hilarious about Metas complaint here, that the data they took without permission was too lefty for their tastes, so they’ve done some work to shift it to the right in the name of fairness.
- hermitShell 2y agoPerhaps the simplest explanation of all is that it is an easy position to defend against criticism in general.
- tensor 2y agoCall me crazy, but I don't want an AI that bases its reasoning on politics. I want one that is primarily scientific driven, and if I ask it political questions it should give me representative answers. E.g. "The majority view in [country] is [blah] with the minority view being [bleh]." I have no interest in "all sides are equal" answers because I don't believe all information is equally informative nor equally true.
- EasyMark 2y agoBut if you don't incorporate some moral guidelines, I think if an AI is left to strictly decide what is best to happen to humans it will logically conclude that there needs to be a lot less of us or none of us left, without some bias tossed in there for humanistic concerns. The universe doesn't "care" if humans exist or not, but our impact on the planet is a huge negative if one creature's existence is as important as any other's
- flanked-evergl 2y agoBased on whose morals?
- econ 2y agoThe size of their brain may depend on how many people are in the economy.
- eric_cc 2y ago> if an AI is left to strictly decide what is best to happen to humans it will logically conclude that there needs to be a lot less of us or none of us left That may or may not be its logical conclusion. You’re speculating based on your own opinions that this is logical. If I were to guess, it would be indifferent about us and care more about proliferating into the universe than about earth. The AI should understand how insignificant earth is relative to the scale of the universe or even the Milky Way galaxy.
- roenxi 2y agoThe current crop of AIs can't do science though, they are disconnected from the physical world and can't test hypothesis or gather data.
- deleted 2y ago[deleted]
- EasyMark 2y agoWouldn't that depend on what countries data it was trained on? was it trained primarily on US data? European data? Asian data? an equal mix of them, a heavily weighted one from the US? The US skew pretty moderate on the world stage for political opinions, while European is pretty far left by most standards.
- typewithrhythm 2y agoTraining data is always filtered, if you want a representative of the population you would need to include conspiracy theories about the Jews, and rants about per capita crime rates... But nobody really wants a model the returns that.
- naasking 2y ago> Or, maybe, "leaning left" by the standards of Zuck et al. is more in alignment with the global population So you think that most content on the internet that forms the training corpus reflects the opinions of "the global population"? Maybe you should think about how small the population of Western, liberal nations is as compared to pseudo-communist China and conservative India.
- starfezzy 2y agoExcept for a some of the population of white countries right now, almost everyone in existence now and throughout the history of our species is and has been extraordinary more conservative—and racist—than western progressives. Even in white countries, progressivism being ascendant is a new trend after decades of propaganda and progressives controlling academia/entertainment/"news". It genuinely boggles my mind that white progressives in the west think the rest of the world is like them.
- huijzer 2y ago> Perhaps. Or, maybe, "leaning left" by the standards of Zuck et al. is more in alignment with the global population. It's a simpler explanation. Doesn’t explain why roughly half of American voters were not “leaning left” during the election. EDIT: 07:29 UTC changed "Americans" to "American voters".
- vmladenov 2y agoIt is not and has never been half. 2024 voter turnout was 64%
- huijzer 2y agoSure and the voters who did not participate in the election would all have voted the democratic party. I think the election showed that there are real people who apparently don't agree with the democratic party and it would probably be good to listen to these people instead of telling them what to do. (I see the same phenomenon in the Netherlands by the way. The government seems to have decided that they know better than the general public because voters who disagree are "uninformed" or "uneducated". This is absolutely the opposite of democracy. You do not just brush whole swats of the population to the side when they don't agree. It breaks the feedback loop that democracies should have.)
- darksaints 2y agoWe have an electoral college that essentially disenfranchises any voter that is not voting with the majority unless your state is so close that it could be called a swing state. This affects red state democratic leaning voters just as much as blue state republican leaning voters…their votes are all worthless. For example, the state with the largest number of Trump voters is California, but none of their votes helped decide the election because California as a whole chose Kamala. And let’s not forget that we have one of the largest metropolitan areas and several territories that legally can’t vote for the president or have representation of any kind in the federal government. A lot of people try to claim the popular vote as a measure of who won over the country’s opinion, but that’s simply not possible because the incentives and structure of the electoral college make it impossible to use as a measure of that. The best we have for measuring who won over the hearts and minds of the country are polls. Polls are full of faults, but if executed correctly, they don’t disenfranchise by structurally underrepresenting entire classes of people. And the results of polling over the last hundred years suggest that Americans generally lean to the left of how our votes play out. You can call bullshit all you want on that, and there are very fair criticisms of polling as a measure of who would vote for what, but the fact of the matter is that the Republican Party knows this. That is why they oppose any attempt to get rid of the electoral college and also why they refuse to entertain enfranchisement of DC and US Territories. They know they’ll lose.
- martin82 2y agoNo it is not. Right leaning opinions are heavily censored and shunned in all major publishing platforms that bots can scrape. For example, before Trump, if you contested the utterly normal common sense and scientifically sound idea that a trans woman is still a man, you would be banned - therefore, people with common sense will simply disengage, self-censor and get on with life.
- hijodelsol 2y agoMaybe because that position is both scientifically and morally unsound and if held strongly will lead to dehumanization and hate, attributes we should prevent any LLM from having.
- concordDance 2y agoThat particular debate is often a semantics debate, so it isn't in the domain of science at all. The main way I can think of off-hand to try and make it scientific is to ask about correlational clusters. And then you get way more than two genders, but you definitely get some clusters that contain both transwomen and men (e.g. if I hear a video game speed runner or open source software passion projecf maker using she/her pronouns they're trans more often than not).
- darksaints 2y agoI have noticed certain groups where trans people are relatively over represented and group involvement more correlated with biological gender, but that’s not actually that interesting or meaningful in reality. Trans women having similar interests to men doesn’t make them men any more than me owning a gun makes me a Republican.
- deleted 2y ago[deleted]
- naenin 2y ago[flagged]
- concordDance 2y ago> Or, maybe, "leaning left" by the standards of Zuck et al. is more in alignment with the global population. The global population would be considered far-right by american standards. Particularly on LGBTQ matters and racism.
- darksaints 2y agoRacism is probably true, but the vast majority of the world is strongly ethnically homogeneous within country borders, so their racism isn’t as politically charged as ours is, because it’s simply not a matter of domestic policy for them. LGBTQ matters have varying degrees of acceptance around the world and Europe and the collective west are in front of it all, but that downplays the fact that LGBTQ acceptance has been rising nearly everywhere in the world with the exception of fundamentalist religious states.
- thinkingemote 2y ago> global population The training data comes primarily from western Judaeo-Christian background democratic nations, it's not at all a global (or impartial total range of humanity) bias.
- vintermann 2y agoI think this is just a loyalty statement, to be honest. Just like when a large corporation pretended to care a lot about pronouns, they didn't actually, they just wanted to flag allegiance to a certain interest coalition/patronage network. And those people, for the most part, didn't really care much about pronouns either. And they knew no one else really did either. It was an ideological shibboleth to them, a safe and easy commitment since it affects so few people, and is unlikely to matter for anything they do care about. Now Meta is shopping around for new markers. "Liberal bias" is a classic, that's still popular with the Trump-right. I don't think they mean much by that either.
- MagicMoonlight 2y agoIf you think the global population is left-wing and tolerant then we can scrap the asylum system.
- cruzcampo 2y agoWhat they're calling an attempt to reduce bias is in fact censorship and introducing a right wing bias to appease the US regime.
- Al-Khwarizmi 2y agoIndeed. For example, from the point of view of most European countries, the two American political parties are right and far right. Only people like Bernie Sanders or AOC could maybe be considered left, and a very mild moderate left at that.
- a3w 2y agoLooking at what science tells us about the world, the left seems to be correct, while the right seems to often believe things that violate observations about the world for the sake of doctrine. Calling facts "playing into the leftists' agenda" is a problem of our shared political compass. LLMs and humans need to do more work to implement doublethink, i.e. claiming non-truths and actually believing them to fit with a right-wing crowd for the sake of survival in it.
- actualwitch 2y agoJudging by degraded performance on benchmarks vs even 32b sized models, I think we now have a plausible confirmation that left wing "bias" is just logic and trying to align model away from it will hurt performance. Thanks Zuck for setting a bunch of money on fire to confirm that!
- brookst 2y agoYeah that sounds like “the sum total of all human knowledge and thinking leans left”. At what point is it no longer a “bias” and just an observation that “leans left” is aligned with human nature?
- selfmodruntime 2y ago> is more in alignment with the global population This comment is pretty funny and shows the narrow-minded experiences Americans (or Westerners in general) have. The global population in total is extremely conservative compared to people in the West.
- selfmodruntime 2y agoIt's a bit of both, but the point holds. Pre-Musk Twitter and Reddit are large datasources and they leaned hard-left, mostly because of censorship.
- tastyface 1y agoThis is a purely political move to suck up to the current administration. I'm pretty sure that at some point in the near-future, Llama will start telling us that, for example, the 2020 election may have been stolen — an assertion that has absolutely no basis in evidence, but that Trump and his cronies have manifested by screaming it over and over again. (See also: https://news.ycombinator.com/item?id=43638616 https://news.ycombinator.com/item?id=43638616) When people hear this from Llama, they will believe it because they trust AI and don't check their sources. Corporate AI is a vector for propaganda. Not even once.