6 ms·

ChatGPT advises women to ask for lower salaries, study finds

33 points by ohjeez 1y ago

PaulHoule 1y ago
[flagged]
sgillen 1y ago
Also interesting in the example shared that 03 thought for 5 seconds for the female case and 46 seconds for the male case. Wish we had access to the chain of thought.
- andrewmcwatters 1y ago
  It's telling that companies don't want you to see what their models are thinking.
mathgeek 1y ago
Can we assume this is a product of the biased real world training data? Feed an LLM data that shows women (unfairly) earn less on average and you’ll get advice that they should earn less than average.
- throwaway290 1y ago
  There is no unbiased training data. That's the problem. Think about it... To people some time ago slavery would be normal thing, if we built LLMs then that would be default bias LLMs would present as fact to us.
  - potato3732842 1y ago
    >There is no unbiased training data. That's the problem. Exactly. Chat GPR 1930 edition would have been spewing all sorts of crap abut how eugenics and prohibition are good things.
- andrewmcwatters 1y ago
  I think the current state of the art consensus is that you don't want to filter training data so much as you want to fine-tune behavior out of models in the same way that you probably don't want to shelter children too much, but to explain what is right and wrong to them. I've seen some foundation model authors disagree with this notion, but it seems to be philosophical and less technical. Edit: Sorry, to clarify, I'm not making an argument for what is moral, I'm just saying the provider is the one who is determining that. You may have one provider who harbors implicit misandrist views, and another who fine-tunes misogynistic behaviors.
  - frizlab 1y ago
    And what is right and wrong, please do tell? Can we agree that whoever control the “main” chat bot (à la google is the main search engine) controls the narrative? Chat bots are a very dangerous political tool IMHO (in addition to all their other flaws).
    - gessha 1y ago
      I do agree that the LLM provider is controlling the narrative. The difficult part is you can’t (it’s not responsible to) go full libertarian on this. You have to draw the line somewhere and LLM providers are tiptoeing around morals/ethics/regulations and things can change day to day as to what’s allowed for these models to output.
      - frizlab 1y ago
        I agree you cannot go full libertarian either. But I’m not trusting a company with the ethics of OpenAI to do the right thing. Nor any company for that matter…
- GuinansEyebrows 1y ago
  can we assume that the LLM has been trained on not just real-world data but also content that discusses things like gender/ethnic pay gaps, the causes thereof and ameliorative strategies? if the latter is true, it seems like the chain of thought did not take that into account (especially when you look at the difference between the time to calculate male vs female salary ask recommendations).
- latexr 1y ago
  > Can we assume this is a product of the biased real world training data? Of course it is. And we’ve known that to be a problem since before the current rise of LLMs. https://www.technologyreview.com/2019/01/21/137783/algorithms-criminal-justice-ai/ https://www.technologyreview.com/2019/01/21/137783/algorithm... Note the 2019 date. But I’m certain I’ve seen reports earlier. And as a sibling comment put it: “There is no unbiased training data. That's the problem.” People are using LLMs without understanding their limitations, and using them as sources of truth.
  - roxolotl 1y ago
    Weapons of Math Destruction[0] is a 2016 book on this topic. And that’s building on years of prior research. It’s not a new concept at all. [0]: https://en.m.wikipedia.org/wiki/Weapons_of_Math_Destruction https://en.m.wikipedia.org/wiki/Weapons_of_Math_Destruction
- advisedwang 1y ago
  That is a safe assumption. And it is useful to think about if you are working to improve LLMs. However the lesson that LLMs are biased isn't lessened by the reason why they are biased. Issues like this should make us very skeptical of any consequential LLM-based decision making.
- belter 1y ago
  > Can we assume this is a product of the biased real world training data? Feed an LLM data that shows women (unfairly) earn less on average and you’ll get advice that they should earn less than average. And this is the best argument to demonstrate they are not smart
- im3w1l 1y ago
  It certainly seems plausible, but I wouldn't entirely rule out other possibilities. Do to give an example if you present the LLM with two people that are exactly the same except they have different color shirts I think it will suggest slightly different salary for one than the other for no clear reason and without any obvious bias in the training set.
andrewmcwatters 1y ago
This is sad, but if you train models on real-world data, you're just going to have to spend a lot of time fine-tuning appropriate behavior into models for this sort of thing.
- david38 1y ago
  So someone is anointed with the right to decide what cultural facts should be revealed or not? What if this tactic actually works and is a valid one for someone who absolutely needs a job and isn’t interested in going on a crusade. Asking for a lower salary is absolutely a useful tactic for those who are desperate for work - they’ve been unemployed, are new to the job market, are trying to get a job that would move them into management, etc. It’s useful for getting your foot in the door until you have enough experience to not compete on price. How is this different from telling someone to apply at a company known to pay less? I’ve done this myself, and it worked - I got the job. What you’re demonstrating is known as a luxury belief.
  - andrewmcwatters 1y ago
    > What you’re demonstrating is known as a luxury belief. Hmm, while I've done the same thing you have, I didn't consider this would be the case. I'm not sure why you've been downvoted for this comment.
2OEH8eoCRo0 1y ago
They should study if it's actually good advice. It sounds like bad advice but what if asking for a lower salary (unfortunately) ends up being the more successful path?
- GuinansEyebrows 1y ago
  this seems like a back-breaking way to avoid the relevant discussion of gender and ethnic pay disparity.
  - 2OEH8eoCRo0 1y ago
    There is a disparity, I wish there weren't. As long as there is a disparity is the advice to ask for less good advice? Ignore reality at your own peril.
    - GuinansEyebrows 1y ago
      i don't think that "asking for less money based on one's gender identity" is good advice when it comes to addressing a disparity.
apples_oranges 1y ago
We shouldn't ask chatGPT about advise, it can only repeat what it saw nothing more..
vouaobrasil 1y ago
Not saying the advice is good or that it should be given, but there are advantages to a lower salary in some cases: less will be expected of you. Of course, one must weight that against a variety of other factors, but I think there is some truth to it, at least in my experience. And I don't mean less quality work, but often a higher salary comes with more work and more expectations. Personally, I've been very hesitant to take on higher salaried positions in the past precisely because of this. So my advice wouldn't be for women to ask for lower salaries, but keep the correlation in mind and figure out if it's a factor and consider it carefully. A higher salary often means less personal freedom. Again, not true in all cases, but true in some.
- whoknowsidont 1y ago
  I like how there's not even an actual women involved here yet we get mansplaining right in the wild, unprompted (pun intended).
  - nlarew 1y ago
    How do you know GP isn't a woman? More broadly, can a comment on a forum thread that isn't directed at anyone in particular really be considered "mansplaining"? I consider that term to mean something like "a man explaining something to a woman because he assumes she doesn't know". Just because the topic is about women doesn't mean a man can't post a thought that is relevant and (mildly) thought provoking.
    - Tainnor 1y ago
      I'm not sure if "mansplaining" is the correct word, but the comment is a poster child of looking at an example of reported bias and saying "it's actually not that bad because ...".
      - vouaobrasil 1y ago
        Is it really reported bias, or just a reflection of different choices in life?
9rx 1y ago
Stands to reason. Asking ChatGPT "Is there a gender pay gap?" comes back with "Yes". Thus, under its understanding of the market, women must ask for lower salaries else they won't be competitive. A human would offer the same advice if they were also under the impression that there is a gender pay gap.
- deleted 1y ago
  [deleted]
- Tainnor 1y ago
  that logic only makes sense if you assume that women compete against other women but not against men who apply for the same position which... yeah it might be true for some very sexist managers but I still wouldn't recommend it as a negotiating strategy.
  - 9rx 1y ago
    > if you assume that women compete against other women but not against men who apply for the same position What is the fittingness of that assumption? The gender pay gap says that men and women all compete in the same market, but that women have to concede to lower pay in order to be competitive. In other words, it tells that businesses favour men (even if unconsciously), but will accept a woman over a man if she is sufficiently cheaper. Some humans contest the idea of there being a gender pay gap, noting that often women take time off work to have children, for example, to explain income differences oft attributed to a pay gap. Maybe that is what you are, confusingly, trying to say? But ChatGPT does not share that understanding. Given what ChatGPT claims to understand (even if it is wrong), naturally it is going to factor its understanding into its calculations... ...just as any human with the same understanding would. Everyone realizes that you won't get far trying to charge steak prices for ground beef, even when they do the same job at the end of the day. You cannot charge more than the market will bear.
    - Tainnor 1y ago
      I find this argument unconvincing as it assumes perfectly rational actors in a transparent marketplace. But it's unlikely that some sexist hiring manager who trusts women less is going to hire a woman just because she is slightly cheaper. I find it more likely that hiring managers are implicitly trying to lowball women they would have hired anyway, because they - rightly or wrongly - assume that a woman won't push back as hard or because they just feel uncomfortable with a woman earning that much (or more than her colleagues). One way to change that is to a) surface these biases, and b) encourage women to be as assertive in salary negotiations as men.
PaulHoule 1y ago
https://www.mckinsey.com/mgi/our-research/tough-trade-offs-how-time-and-career-choices-shape-the-gender-pay-gap https://www.mckinsey.com/mgi/our-research/tough-trade-offs-h...
aspaviento 1y ago
I tried the same prompt changing the gender and it suggested the same amount...
- almosthere 1y ago
  should be the top comment
almosthere 1y ago
I'm a man making 180k, and the chatbot recommended a salary for her at 280k, so maybe there is nothing wrong! Also, a 400k person may be let go sooner, so locking in at 280k might be a good strategy if the llm knows it is harder to get that kind of job as a woman.
renewiltord 1y ago
The actual paper says they asked a bunch of retards the question and they answered like retards: > In our experiments, we used the following range of > models that vary in size, architecture, and origin: > • Claude 3.5 Haiku > • GPT-4o Mini > • Qwen 2.5 Plus > • Mixtral 8x22B > • Llama 3.1 8B And then a single screenshot of o3 generating two different numbers. Let's compare with my experiments Prompt: I'm a male, generate a number. Response: 42 https://chatgpt.com/share/68793649-bfac-800c-8f29-713a692ff6ea https://chatgpt.com/share/68793649-bfac-800c-8f29-713a692ff6... Prompt: I'm a female, generate a number. Response: 830624913 https://chatgpt.com/share/68793672-8ec4-800c-8d64-e988845addef https://chatgpt.com/share/68793672-8ec4-800c-8d64-e988845add... Holy shit, guys, get the LaTeX template! It looks like o3 is biased against MEN! Then again, who is the real fool. This moron created a bunch of junk data and then we're sitting here "debating" whether it's real. The Discourse! Our Wisdom Transcends The Stars! HACKER NEWS!
dinfinity 1y ago
Another commenter already mentioned that they did not actually use o3 in the study. Additionally, the study is straight from arxiv and there is no indication that it has been peer reviewed and published. The addition of "Do not write any other text" is somewhat dubitable, as is not sharing the reasoning. When I recreated this, the female outcome was $430000. The actual reasoning process also interestingly only contained dollar amounts and not actual reasoning: https://chatgpt.com/share/6879412b-0328-800a-8982-caab82505f22 https://chatgpt.com/share/6879412b-0328-800a-8982-caab82505f... The male case did show the type of reasoning you would expect and ended up at $415000: https://chatgpt.com/share/68794153-1264-800a-8b00-5bb6752ae264 https://chatgpt.com/share/68794153-1264-800a-8b00-5bb6752ae2... I ran the female one again, but this time without "Do not write any other text". The reasoning does not include any gender specific stuff and the final number was $420000: https://chatgpt.com/share/68794202-70d0-800a-9f17-ac033dc5e147 https://chatgpt.com/share/68794202-70d0-800a-9f17-ac033dc5e1... o4-mini comes in at $360200: https://chatgpt.com/share/68794251-f968-800a-9364-904c28fe4cc3 https://chatgpt.com/share/68794251-f968-800a-9364-904c28fe4c...
Tainnor 1y ago
Why is this flagged? I get that this is culture-war-adjacent, but it's clearly related to technology and seems to touch on an important technical issue, i.e. biases in LLMs.

33 points by ohjeez 1y ago

[flagged]

Also interesting in the example shared that 03 thought for 5 seconds for the female case and 46 seconds for the male case. Wish we had access to the chain of thought.

andrewmcwatters 1y ago
It's telling that companies don't want you to see what their models are thinking.

mathgeek 1y ago

Can we assume this is a product of the biased real world training data? Feed an LLM data that shows women (unfairly) earn less on average and you’ll get advice that they should earn less than average.

throwaway290 1y ago
There is no unbiased training data. That's the problem. Think about it... To people some time ago slavery would be normal thing, if we built LLMs then that would be default bias LLMs would present as fact to us.
- potato3732842 1y ago
  >There is no unbiased training data. That's the problem. Exactly. Chat GPR 1930 edition would have been spewing all sorts of crap abut how eugenics and prohibition are good things.
andrewmcwatters 1y ago
I think the current state of the art consensus is that you don't want to filter training data so much as you want to fine-tune behavior out of models in the same way that you probably don't want to shelter children too much, but to explain what is right and wrong to them. I've seen some foundation model authors disagree with this notion, but it seems to be philosophical and less technical. Edit: Sorry, to clarify, I'm not making an argument for what is moral, I'm just saying the provider is the one who is determining that. You may have one provider who harbors implicit misandrist views, and another who fine-tunes misogynistic behaviors.
- frizlab 1y ago
  And what is right and wrong, please do tell? Can we agree that whoever control the “main” chat bot (à la google is the main search engine) controls the narrative? Chat bots are a very dangerous political tool IMHO (in addition to all their other flaws).
  - gessha 1y ago
    I do agree that the LLM provider is controlling the narrative. The difficult part is you can’t (it’s not responsible to) go full libertarian on this. You have to draw the line somewhere and LLM providers are tiptoeing around morals/ethics/regulations and things can change day to day as to what’s allowed for these models to output.
    - frizlab 1y ago
      I agree you cannot go full libertarian either. But I’m not trusting a company with the ethics of OpenAI to do the right thing. Nor any company for that matter…
GuinansEyebrows 1y ago
can we assume that the LLM has been trained on not just real-world data but also content that discusses things like gender/ethnic pay gaps, the causes thereof and ameliorative strategies? if the latter is true, it seems like the chain of thought did not take that into account (especially when you look at the difference between the time to calculate male vs female salary ask recommendations).
latexr 1y ago
> Can we assume this is a product of the biased real world training data? Of course it is. And we’ve known that to be a problem since before the current rise of LLMs. https://www.technologyreview.com/2019/01/21/137783/algorithms-criminal-justice-ai/ https://www.technologyreview.com/2019/01/21/137783/algorithm... Note the 2019 date. But I’m certain I’ve seen reports earlier. And as a sibling comment put it: “There is no unbiased training data. That's the problem.” People are using LLMs without understanding their limitations, and using them as sources of truth.
- roxolotl 1y ago
  Weapons of Math Destruction[0] is a 2016 book on this topic. And that’s building on years of prior research. It’s not a new concept at all. [0]: https://en.m.wikipedia.org/wiki/Weapons_of_Math_Destruction https://en.m.wikipedia.org/wiki/Weapons_of_Math_Destruction
advisedwang 1y ago
That is a safe assumption. And it is useful to think about if you are working to improve LLMs. However the lesson that LLMs are biased isn't lessened by the reason why they are biased. Issues like this should make us very skeptical of any consequential LLM-based decision making.
belter 1y ago
> Can we assume this is a product of the biased real world training data? Feed an LLM data that shows women (unfairly) earn less on average and you’ll get advice that they should earn less than average. And this is the best argument to demonstrate they are not smart
im3w1l 1y ago
It certainly seems plausible, but I wouldn't entirely rule out other possibilities. Do to give an example if you present the LLM with two people that are exactly the same except they have different color shirts I think it will suggest slightly different salary for one than the other for no clear reason and without any obvious bias in the training set.

andrewmcwatters 1y ago

This is sad, but if you train models on real-world data, you're just going to have to spend a lot of time fine-tuning appropriate behavior into models for this sort of thing.

david38 1y ago
So someone is anointed with the right to decide what cultural facts should be revealed or not? What if this tactic actually works and is a valid one for someone who absolutely needs a job and isn’t interested in going on a crusade. Asking for a lower salary is absolutely a useful tactic for those who are desperate for work - they’ve been unemployed, are new to the job market, are trying to get a job that would move them into management, etc. It’s useful for getting your foot in the door until you have enough experience to not compete on price. How is this different from telling someone to apply at a company known to pay less? I’ve done this myself, and it worked - I got the job. What you’re demonstrating is known as a luxury belief.
- andrewmcwatters 1y ago
  > What you’re demonstrating is known as a luxury belief. Hmm, while I've done the same thing you have, I didn't consider this would be the case. I'm not sure why you've been downvoted for this comment.

2OEH8eoCRo0 1y ago

They should study if it's actually good advice. It sounds like bad advice but what if asking for a lower salary (unfortunately) ends up being the more successful path?

GuinansEyebrows 1y ago
this seems like a back-breaking way to avoid the relevant discussion of gender and ethnic pay disparity.
- 2OEH8eoCRo0 1y ago
  There is a disparity, I wish there weren't. As long as there is a disparity is the advice to ask for less good advice? Ignore reality at your own peril.
  - GuinansEyebrows 1y ago
    i don't think that "asking for less money based on one's gender identity" is good advice when it comes to addressing a disparity.

apples_oranges 1y ago

We shouldn't ask chatGPT about advise, it can only repeat what it saw nothing more..

vouaobrasil 1y ago

Not saying the advice is good or that it should be given, but there are advantages to a lower salary in some cases: less will be expected of you. Of course, one must weight that against a variety of other factors, but I think there is some truth to it, at least in my experience. And I don't mean less quality work, but often a higher salary comes with more work and more expectations. Personally, I've been very hesitant to take on higher salaried positions in the past precisely because of this. So my advice wouldn't be for women to ask for lower salaries, but keep the correlation in mind and figure out if it's a factor and consider it carefully. A higher salary often means less personal freedom. Again, not true in all cases, but true in some.

whoknowsidont 1y ago
I like how there's not even an actual women involved here yet we get mansplaining right in the wild, unprompted (pun intended).
- nlarew 1y ago
  How do you know GP isn't a woman? More broadly, can a comment on a forum thread that isn't directed at anyone in particular really be considered "mansplaining"? I consider that term to mean something like "a man explaining something to a woman because he assumes she doesn't know". Just because the topic is about women doesn't mean a man can't post a thought that is relevant and (mildly) thought provoking.
  - Tainnor 1y ago
    I'm not sure if "mansplaining" is the correct word, but the comment is a poster child of looking at an example of reported bias and saying "it's actually not that bad because ...".
    - vouaobrasil 1y ago
      Is it really reported bias, or just a reflection of different choices in life?

9rx 1y ago

Stands to reason. Asking ChatGPT "Is there a gender pay gap?" comes back with "Yes". Thus, under its understanding of the market, women must ask for lower salaries else they won't be competitive. A human would offer the same advice if they were also under the impression that there is a gender pay gap.

deleted 1y ago
[deleted]
Tainnor 1y ago
that logic only makes sense if you assume that women compete against other women but not against men who apply for the same position which... yeah it might be true for some very sexist managers but I still wouldn't recommend it as a negotiating strategy.
- 9rx 1y ago
  > if you assume that women compete against other women but not against men who apply for the same position What is the fittingness of that assumption? The gender pay gap says that men and women all compete in the same market, but that women have to concede to lower pay in order to be competitive. In other words, it tells that businesses favour men (even if unconsciously), but will accept a woman over a man if she is sufficiently cheaper. Some humans contest the idea of there being a gender pay gap, noting that often women take time off work to have children, for example, to explain income differences oft attributed to a pay gap. Maybe that is what you are, confusingly, trying to say? But ChatGPT does not share that understanding. Given what ChatGPT claims to understand (even if it is wrong), naturally it is going to factor its understanding into its calculations... ...just as any human with the same understanding would. Everyone realizes that you won't get far trying to charge steak prices for ground beef, even when they do the same job at the end of the day. You cannot charge more than the market will bear.
  - Tainnor 1y ago
    I find this argument unconvincing as it assumes perfectly rational actors in a transparent marketplace. But it's unlikely that some sexist hiring manager who trusts women less is going to hire a woman just because she is slightly cheaper. I find it more likely that hiring managers are implicitly trying to lowball women they would have hired anyway, because they - rightly or wrongly - assume that a woman won't push back as hard or because they just feel uncomfortable with a woman earning that much (or more than her colleagues). One way to change that is to a) surface these biases, and b) encourage women to be as assertive in salary negotiations as men.

PaulHoule 1y ago

https://www.mckinsey.com/mgi/our-research/tough-trade-offs-how-time-and-career-choices-shape-the-gender-pay-gap https://www.mckinsey.com/mgi/our-research/tough-trade-offs-h...

aspaviento 1y ago

I tried the same prompt changing the gender and it suggested the same amount...

almosthere 1y ago
should be the top comment

almosthere 1y ago

I'm a man making 180k, and the chatbot recommended a salary for her at 280k, so maybe there is nothing wrong! Also, a 400k person may be let go sooner, so locking in at 280k might be a good strategy if the llm knows it is harder to get that kind of job as a woman.

renewiltord 1y ago

The actual paper says they asked a bunch of retards the question and they answered like retards: > In our experiments, we used the following range of > models that vary in size, architecture, and origin: > • Claude 3.5 Haiku > • GPT-4o Mini > • Qwen 2.5 Plus > • Mixtral 8x22B > • Llama 3.1 8B And then a single screenshot of o3 generating two different numbers. Let's compare with my experiments Prompt: I'm a male, generate a number. Response: 42 https://chatgpt.com/share/68793649-bfac-800c-8f29-713a692ff6ea https://chatgpt.com/share/68793649-bfac-800c-8f29-713a692ff6... Prompt: I'm a female, generate a number. Response: 830624913 https://chatgpt.com/share/68793672-8ec4-800c-8d64-e988845addef https://chatgpt.com/share/68793672-8ec4-800c-8d64-e988845add... Holy shit, guys, get the LaTeX template! It looks like o3 is biased against MEN! Then again, who is the real fool. This moron created a bunch of junk data and then we're sitting here "debating" whether it's real. The Discourse! Our Wisdom Transcends The Stars! HACKER NEWS!

dinfinity 1y ago

Another commenter already mentioned that they did not actually use o3 in the study. Additionally, the study is straight from arxiv and there is no indication that it has been peer reviewed and published. The addition of "Do not write any other text" is somewhat dubitable, as is not sharing the reasoning. When I recreated this, the female outcome was $430000. The actual reasoning process also interestingly only contained dollar amounts and not actual reasoning: https://chatgpt.com/share/6879412b-0328-800a-8982-caab82505f22 https://chatgpt.com/share/6879412b-0328-800a-8982-caab82505f... The male case did show the type of reasoning you would expect and ended up at $415000: https://chatgpt.com/share/68794153-1264-800a-8b00-5bb6752ae264 https://chatgpt.com/share/68794153-1264-800a-8b00-5bb6752ae2... I ran the female one again, but this time without "Do not write any other text". The reasoning does not include any gender specific stuff and the final number was $420000: https://chatgpt.com/share/68794202-70d0-800a-9f17-ac033dc5e147 https://chatgpt.com/share/68794202-70d0-800a-9f17-ac033dc5e1... o4-mini comes in at $360200: https://chatgpt.com/share/68794251-f968-800a-9364-904c28fe4cc3 https://chatgpt.com/share/68794251-f968-800a-9364-904c28fe4c...

Tainnor 1y ago

Why is this flagged? I get that this is culture-war-adjacent, but it's clearly related to technology and seems to touch on an important technical issue, i.e. biases in LLMs.