3 ms·

The finding here is not so much gender bias but rather a generic leftward bias. Although the headline result is a large bias in favor of women, there's also a b

The finding here is not so much gender bias but rather a generic leftward bias. Although the headline result is a large bias in favor of women, there's also a bias towards people who put preferred pronouns on their CVs. This is a problem that's been widely known about in the AI industry for a long time now. It's easy to assume that this is deliberate, because of incidents like Google's notorious "everyone is black including popes and Nazis" faceplant. But OpenAI/Altman have commented on it in public, Meta (FAIR) have also stated clearly in their last Llama release that this is an unintentional problem that they are looking for ways to correct. The issue is likely that left-wing people are attracted to professions whose primary output is words rather than things. Actors, journalists and academics are all quite left-biased professions whose output consists entirely of words, and so the things they say will be over-represented in the training set. In contrast some of the most conservative industries are things like oil & gas, mining, farming and manufacturing, where the outputs are physical and thus invisible to LLMs. https://verdantlabs.com/politics_of_professions/ https://verdantlabs.com/politics_of_professions/ It's not entirely clear what can be done about this, but synthetic data and filtering will probably play a role. Even quite biased LLMs do understand the nature of political disagreements and what bias means, so can be used to curate out the worst of the input data. Ultimately though, the problem of left-wing people congregrating in places where quantity of verbal output is rewarded means they will inevitably dominate the training sets.

plaidfuji 1y ago
Exactly. LLMs as they exist today have just codified a bunch of left-leaning cultural norms from their training set, which is biased toward text generated by internet users from the ~90s to today (a distinctly - though decreasingly - left-leaning bloc). Of course they have a bunch of books and scholarly texts in there as well, but in my experience LLM resume review is substantially shallower in reasoning than more academic tasks. I don’t think it’s cross-referencing skills and experience to technical “knowledge” in a deep way.
datadrivenangel 1y ago
The issue is that much of the data will skew 'left' because clasically liberal values like equality and equity now are applied to everyone, and the media will roast any large company that has a model which is doing grok like things, so the incentives are to add filters which over-correct.
nullc 1y ago
> The issue is likely that left-wing people are attracted to professions whose primary output is words rather than things. Actors, journalists and academics are all quite left-biased professions whose output consists entirely of words, and so the things they say will be over-represented in the training set. Yet the 'base' models which aren't chat fine tuned seem to exhibit this far less strongly, -- though their different behavior makes an apples to apples comparison difficult. The effect may be because the instruct fine tuning radically reduces the output diversity, thus greatly amplifying an existing small bias, but even if it is just that it shows how fine tuning can be problematic. I have maybe a little doubt on your hopes for synthetic correction-- seems you're suggesting a positive feedback mechanism which tend to increase bias and I think would here if we assume that the bias is pervasive. E.g. that it won't just produce biased outputs but it will also judge its own biased outputs more favorably than it should.
- mike_hearn 1y ago
  Well, RLHF is nothing but synthetic correction in a sense. And modern models are trained on inputs that are heavily AI curated or generated. So there's no theoretical issue with it. ML training on its own outputs definitely can lead to runaway collapse if done naively, but the more careful ways it's being done now work fine. I suspect in the era when base models were made available there was much more explicit bias being introduced via post-training. Modern models are a lot saner when given trolly questions than they were a few years ago, and the internet hasn't changed much, so that must be due to adjustments made to the RLHF. Probably the absurdity of the results caused a bit of a reality check inside the training teams. The rapid expansion of AI labs would have introduced a more diverse workforce too. I doubt the bias can be removed entirely, but there's surely a lot of low hanging fruit there. User feedbacks and conversations have to be treated carefully as OpenAI's recent rollback shows, but in theory it's a source of text that should reflect the average person much better than Reddit comments do. And it's possible that the smartest models can be given an explicit theory of political mind.
Ancapistani 1y ago
I'm coming from a different political bias than most here, as an Anarcho-Capitalist. To put it bluntly and very broadly, think of my perspective as extreme right-libertarian. I think frontier model companies are finding that their base models exhibit some problematic "views". I don't have direct knowledge here, but here's a hypothetical example to illustrate my theory: _NOTE_: I'm trying to illustrate a point here, not stating a political or social view. This is technical point, not a political/ethical one. Please don't take this as a statement of my own beliefs -- that is explicitly not my intent. --- Let's say ChatGPT 4o, before fine-tuning, would confidently state that a black male is more likely to be convicted of a violent crime than a white male. Considering only demographic and crime statistics, that's true. It's also likely not what the user was asking, and depending on presentation could represent a huge reputational risk to OpenAI. So OpenAI would then presumably build and maintain a set of politically-sensitive prompts and their desired responses. That set would then be used to fine-tune and validate the model's outputs, adjusting them until the model no longer makes "factually true but incorrect and potentially socially abhorrent" statements. The impacts of this tuning are only validated within the scope of their test set; who know what impact they have on other responses. That's, at best, handled by a more general test set. My theory here is that frontier model companies are unintentionally introducing "left-wing bias" in an attempt to remove what is seen as "right-wing bias", but is actually a lack of emotional intelligence and/or awareness of social norms.

plaidfuji 1y ago

Exactly. LLMs as they exist today have just codified a bunch of left-leaning cultural norms from their training set, which is biased toward text generated by internet users from the ~90s to today (a distinctly - though decreasingly - left-leaning bloc). Of course they have a bunch of books and scholarly texts in there as well, but in my experience LLM resume review is substantially shallower in reasoning than more academic tasks. I don’t think it’s cross-referencing skills and experience to technical “knowledge” in a deep way.

datadrivenangel 1y ago

The issue is that much of the data will skew 'left' because clasically liberal values like equality and equity now are applied to everyone, and the media will roast any large company that has a model which is doing grok like things, so the incentives are to add filters which over-correct.

nullc 1y ago

> The issue is likely that left-wing people are attracted to professions whose primary output is words rather than things. Actors, journalists and academics are all quite left-biased professions whose output consists entirely of words, and so the things they say will be over-represented in the training set. Yet the 'base' models which aren't chat fine tuned seem to exhibit this far less strongly, -- though their different behavior makes an apples to apples comparison difficult. The effect may be because the instruct fine tuning radically reduces the output diversity, thus greatly amplifying an existing small bias, but even if it is just that it shows how fine tuning can be problematic. I have maybe a little doubt on your hopes for synthetic correction-- seems you're suggesting a positive feedback mechanism which tend to increase bias and I think would here if we assume that the bias is pervasive. E.g. that it won't just produce biased outputs but it will also judge its own biased outputs more favorably than it should.

mike_hearn 1y ago
Well, RLHF is nothing but synthetic correction in a sense. And modern models are trained on inputs that are heavily AI curated or generated. So there's no theoretical issue with it. ML training on its own outputs definitely can lead to runaway collapse if done naively, but the more careful ways it's being done now work fine. I suspect in the era when base models were made available there was much more explicit bias being introduced via post-training. Modern models are a lot saner when given trolly questions than they were a few years ago, and the internet hasn't changed much, so that must be due to adjustments made to the RLHF. Probably the absurdity of the results caused a bit of a reality check inside the training teams. The rapid expansion of AI labs would have introduced a more diverse workforce too. I doubt the bias can be removed entirely, but there's surely a lot of low hanging fruit there. User feedbacks and conversations have to be treated carefully as OpenAI's recent rollback shows, but in theory it's a source of text that should reflect the average person much better than Reddit comments do. And it's possible that the smartest models can be given an explicit theory of political mind.

Ancapistani 1y ago

I'm coming from a different political bias than most here, as an Anarcho-Capitalist. To put it bluntly and very broadly, think of my perspective as extreme right-libertarian. I think frontier model companies are finding that their base models exhibit some problematic "views". I don't have direct knowledge here, but here's a hypothetical example to illustrate my theory: _NOTE_: I'm trying to illustrate a point here, not stating a political or social view. This is technical point, not a political/ethical one. Please don't take this as a statement of my own beliefs -- that is explicitly not my intent. --- Let's say ChatGPT 4o, before fine-tuning, would confidently state that a black male is more likely to be convicted of a violent crime than a white male. Considering only demographic and crime statistics, that's true. It's also likely not what the user was asking, and depending on presentation could represent a huge reputational risk to OpenAI. So OpenAI would then presumably build and maintain a set of politically-sensitive prompts and their desired responses. That set would then be used to fine-tune and validate the model's outputs, adjusting them until the model no longer makes "factually true but incorrect and potentially socially abhorrent" statements. The impacts of this tuning are only validated within the scope of their test set; who know what impact they have on other responses. That's, at best, handled by a more general test set. My theory here is that frontier model companies are unintentionally introducing "left-wing bias" in an attempt to remove what is seen as "right-wing bias", but is actually a lack of emotional intelligence and/or awareness of social norms.