6 ms·

there maybe better examples of why AI biases provide deep systemic problems but: CV screening? are contemporary LLM really worse that previous screening tech an

by 627467 2y ago

there maybe better examples of why AI biases provide deep systemic problems but: CV screening? are contemporary LLM really worse that previous screening tech and processes?

ivan_gammel 2y ago
You can combat bias in a team of people, but in LLM you may not even know it exists.
- fallingknife 2y ago
  How can you combat bias in a team of humans if humans are susceptible to bias? How do you even know what neutral is if it's not a physically measurable quantity?
  - Larrikin 2y ago
    This sounds like an argument for never educating people and doing nothing about the problem. Just because you graduated from a CS program doesn't mean you will always write bug free code using the ideal algorithm and design pattern. But you know what you should be striving for, can more readily identify issues, and maybe sometimes you actually are perfect.
    - AnthonyMouse 2y ago
      > This sounds like an argument for never educating people and doing nothing about the problem. I feel like it's really the opposite. There is a major problem with using AI to screen CVs, which is that AI is bad at screening CVs. It excludes good candidates and offers mediocre ones, and opens up a bunch of hacks the equivalent of black hat SEO, which gives an undesired advantage to people inclined to dirty tricks. But "it's racist" is a political hot button, so before people can start to point that out, someone scrambles to push the hot button and suck all the oxygen out of the room. Then the purveyors of snake oil can munge the algorithm until it passes the naive oversimplified racial bias test someone made up (probably making it even worse at its intended purpose), and then claim "the problem" is solved, as if that was the main and only problem.
  - lanstin 2y ago
    They send resumes with the same accomplishments and names that indicate race and observe the relative degrees of acceptance. Everytime they do this test, considerable bias is found to still exist.
    - ekianjo 2y ago
      remove names then. problem solved.
      - TylerE 2y ago
        There are still plenty of proxies that can amount to the same thing. Not a lot of white graduates of HBCUs, for instance. Are we removing the names of all schools, too?
        ekianjo 2y ago
        if you dont want to favor ivy leagues that would be the next thing to remove yes
    - mensetmanusman 2y ago
      Still using resumes, have an ai interview for knowledge and communication skill.
      - wondertag 2y ago
        Like many others, we've built a product to do this. I think it gives a much higher signal on candidates than a resume, and is more "blind" in terms of bias than a human. It's based on voice messages. Open to all here if anyone wants to take it for a spin. https://candit.net https://candit.net
        lanstin 2y ago
        How do you get an unbiased evaluation of someone with an accent that differs from your training set? It took about a decade for my friends with Indian English accents to be able to use voice controls in luxury cars. And honestly young people from India have a much more American accent than my generation.
  - ivan_gammel 2y ago
    It‘s 2024. There are decades or research with a lot of data, there exist best practices and products that help combating the bias. The toolkit available to HRs is rich and there’s plenty of things that work. E.g. I have absolutely no problem hiring women in tech roles, because I simply adjusted my process. I heard complaints from other engineering leaders that they want to increase diversity but struggle to find candidates: they do not realize that they have bias in job descriptions, they apply same screening criteria and do the post-interview evaluation the same way regardless of candidate gender or background. Many people can do the job, but they communicate it differently and it is important to account for that when you see the CV or talk to them.
- lolinder 2y ago
  Isn't not knowing that bias exists just the same problem that the average HR department has? You solve that by educating people about biases. If anything that should be easier with LLMs because you're not asking people to alter their sense of self ("I wouldn't do that!"), you're just informing them of technical limitations.
  - ivan_gammel 2y ago
    I don’t understand your argument. The whole purpose of AI-driven recruitment is to reduce human effort by doing some work for them. If AI due to existing bias reduces diversity in your pool of candidates, what can you do with it? You either do not notice the bias and just think that those people are indeed the best fit, you willingly accept the bias („But it reduces our hiring costs! We deal with diversity later, when this technology matures!“), or you go to see the unfiltered stream and put such an amount of work into it that makes AI useless. HR departments are not the people who will build their own solution, they will buy it from a third party and they won’t have control over or the budget for the fine-tuning.
- ekianjo 2y ago
  of course you can know it exists. you just need to observe inputs and outputs and compare with precious non LLM ways.
  - ivan_gammel 2y ago
    You do not understand how HR works then. They are not LLM researchers and just apply the available toolkit, e.g. organizing regular training for hiring managers. When they buy AI-driven solution from a third party, they have no tools neither to detect if it’s just a statistical fluctuation or there’s a bias they need to find a workaround for, nor to act on this information in a cost-efficient way. Most likely they won‘t even have enough data to reach any statistical significance. The vendor in theory could do it, but most of them are startups now and they do not have diversity in their OKRs, at least not now. It’s not the UVP of their product.
dudinax 2y ago
Of course they could be. A company that doesn't want a racial bias won't intentionally filter on names, but might accidentally deploy an LLM that can discern race from name.
- lanstin 2y ago
  The humans will do it unconsciously if they can see the names. It is like orchestras used to use auditions where the judges could see the people; when they went to auditions where they couldn't see the people, the number of women being hired went right up.
  - thaumasiotes 2y ago
    See, this is the problem with just deciding to invent your own facts. There is a famous study on this topic, usually presented in much the way you describe. But usually people are more careful not to lie about the results. The rate of women being hired went down, not up. The "success" for women was contained to advancement through particular audition rounds. https://archive.is/xmvp2 https://archive.is/xmvp2 The original paper is not exactly a model of investigative integrity: >> Women are about 5 percentage points more likely to be hired than are men in a completely blind audition, although the effect is not statistically significant. The effect is nil, however, when there is a semifinal round, perhaps as a result of the unusual effects of the semifinal round.
    - ZeroGravitas 2y ago
      Are you citing a weird racist blogger on medium that disagrees with the stated conclusion of the paper? Is that how science works now? Abstract from the original paper: > A change in the audition procedures of symphony orchestras--adoption of "blind" auditions with a "screen" to conceal the candidate's identity from the jury--provides a test for sex-biased hiring. Using data from actual auditions, in an individual fixed-effects framework, we find that the screen increases the probability a woman will be advanced and hired. Although some of our estimates have large standard errors and there is one persistent effect in the opposite direction, the weight of the evidence suggests that the blind audition procedure fostered impartiality in hiring and increased the proportion women in symphony orchestras. https://www.aeaweb.org/articles?id=10.1257/aer.90.4.715 https://www.aeaweb.org/articles?id=10.1257/aer.90.4.715
no_wizard 2y ago
You realize this is a huge problem right? Even if that was my only complaint (it’s not!) that is 100% not acceptable. If it’s doing it with CVs it’s doing it with other things in other scenarios. Another is that a health insurance company was caught using AI to determine if a claim should be denied or not which lead to a scandal as a whistleblower leaked the practice, as it was wrought with errors and ethical concerns
justinclift 2y ago
It's almost guaranteed there are people trying to sell an AI version of Robodebt right now to the Australian government, even though the last (non-AI) version of it was an absolute cluster fuck: https://en.wikipedia.org/wiki/Robodebt_scheme https://en.wikipedia.org/wiki/Robodebt_scheme Other governments around the world have done similar (non-AI) things in the past with similar terrible results. They'll likely try an AI version of things too in the near future, just because "AI" apparently solves all the problems. Ugh.
BobbyJo 2y ago
I think a worse feature for them to have is consistency. Imagine being someone that has fallen through the cracks of software a significant number of employers are using... Basically soft-locked out of employment with no recourse.
- corimaith 2y ago
  I'd argue consistency in discrimination is easier because it makes it easier to detect and account for. A range of different people interviewing are going to be alot harder to pin down.
  - account42 2y ago
    Discrimination in this sense is a statistical property. If Jimbob gets rejected everywhere as an individual it isn't really discrimination but that doesn't mean it isn't an issue.
deleted 2y ago
[deleted]

by 627467 2y ago

there maybe better examples of why AI biases provide deep systemic problems but: CV screening? are contemporary LLM really worse that previous screening tech and processes?

ivan_gammel 2y ago

You can combat bias in a team of people, but in LLM you may not even know it exists.

fallingknife 2y ago
How can you combat bias in a team of humans if humans are susceptible to bias? How do you even know what neutral is if it's not a physically measurable quantity?
- Larrikin 2y ago
  This sounds like an argument for never educating people and doing nothing about the problem. Just because you graduated from a CS program doesn't mean you will always write bug free code using the ideal algorithm and design pattern. But you know what you should be striving for, can more readily identify issues, and maybe sometimes you actually are perfect.
  - AnthonyMouse 2y ago
    > This sounds like an argument for never educating people and doing nothing about the problem. I feel like it's really the opposite. There is a major problem with using AI to screen CVs, which is that AI is bad at screening CVs. It excludes good candidates and offers mediocre ones, and opens up a bunch of hacks the equivalent of black hat SEO, which gives an undesired advantage to people inclined to dirty tricks. But "it's racist" is a political hot button, so before people can start to point that out, someone scrambles to push the hot button and suck all the oxygen out of the room. Then the purveyors of snake oil can munge the algorithm until it passes the naive oversimplified racial bias test someone made up (probably making it even worse at its intended purpose), and then claim "the problem" is solved, as if that was the main and only problem.
- lanstin 2y ago
  They send resumes with the same accomplishments and names that indicate race and observe the relative degrees of acceptance. Everytime they do this test, considerable bias is found to still exist.
  - ekianjo 2y ago
    remove names then. problem solved.
    - TylerE 2y ago
      There are still plenty of proxies that can amount to the same thing. Not a lot of white graduates of HBCUs, for instance. Are we removing the names of all schools, too?
      - ekianjo 2y ago
        if you dont want to favor ivy leagues that would be the next thing to remove yes
  - mensetmanusman 2y ago
    Still using resumes, have an ai interview for knowledge and communication skill.
    - wondertag 2y ago
      Like many others, we've built a product to do this. I think it gives a much higher signal on candidates than a resume, and is more "blind" in terms of bias than a human. It's based on voice messages. Open to all here if anyone wants to take it for a spin. https://candit.net https://candit.net
      - lanstin 2y ago
        How do you get an unbiased evaluation of someone with an accent that differs from your training set? It took about a decade for my friends with Indian English accents to be able to use voice controls in luxury cars. And honestly young people from India have a much more American accent than my generation.
- ivan_gammel 2y ago
  It‘s 2024. There are decades or research with a lot of data, there exist best practices and products that help combating the bias. The toolkit available to HRs is rich and there’s plenty of things that work. E.g. I have absolutely no problem hiring women in tech roles, because I simply adjusted my process. I heard complaints from other engineering leaders that they want to increase diversity but struggle to find candidates: they do not realize that they have bias in job descriptions, they apply same screening criteria and do the post-interview evaluation the same way regardless of candidate gender or background. Many people can do the job, but they communicate it differently and it is important to account for that when you see the CV or talk to them.
lolinder 2y ago
Isn't not knowing that bias exists just the same problem that the average HR department has? You solve that by educating people about biases. If anything that should be easier with LLMs because you're not asking people to alter their sense of self ("I wouldn't do that!"), you're just informing them of technical limitations.
- ivan_gammel 2y ago
  I don’t understand your argument. The whole purpose of AI-driven recruitment is to reduce human effort by doing some work for them. If AI due to existing bias reduces diversity in your pool of candidates, what can you do with it? You either do not notice the bias and just think that those people are indeed the best fit, you willingly accept the bias („But it reduces our hiring costs! We deal with diversity later, when this technology matures!“), or you go to see the unfiltered stream and put such an amount of work into it that makes AI useless. HR departments are not the people who will build their own solution, they will buy it from a third party and they won’t have control over or the budget for the fine-tuning.
ekianjo 2y ago
of course you can know it exists. you just need to observe inputs and outputs and compare with precious non LLM ways.
- ivan_gammel 2y ago
  You do not understand how HR works then. They are not LLM researchers and just apply the available toolkit, e.g. organizing regular training for hiring managers. When they buy AI-driven solution from a third party, they have no tools neither to detect if it’s just a statistical fluctuation or there’s a bias they need to find a workaround for, nor to act on this information in a cost-efficient way. Most likely they won‘t even have enough data to reach any statistical significance. The vendor in theory could do it, but most of them are startups now and they do not have diversity in their OKRs, at least not now. It’s not the UVP of their product.

dudinax 2y ago

Of course they could be. A company that doesn't want a racial bias won't intentionally filter on names, but might accidentally deploy an LLM that can discern race from name.

lanstin 2y ago
The humans will do it unconsciously if they can see the names. It is like orchestras used to use auditions where the judges could see the people; when they went to auditions where they couldn't see the people, the number of women being hired went right up.
- thaumasiotes 2y ago
  See, this is the problem with just deciding to invent your own facts. There is a famous study on this topic, usually presented in much the way you describe. But usually people are more careful not to lie about the results. The rate of women being hired went down, not up. The "success" for women was contained to advancement through particular audition rounds. https://archive.is/xmvp2 https://archive.is/xmvp2 The original paper is not exactly a model of investigative integrity: >> Women are about 5 percentage points more likely to be hired than are men in a completely blind audition, although the effect is not statistically significant. The effect is nil, however, when there is a semifinal round, perhaps as a result of the unusual effects of the semifinal round.
  - ZeroGravitas 2y ago
    Are you citing a weird racist blogger on medium that disagrees with the stated conclusion of the paper? Is that how science works now? Abstract from the original paper: > A change in the audition procedures of symphony orchestras--adoption of "blind" auditions with a "screen" to conceal the candidate's identity from the jury--provides a test for sex-biased hiring. Using data from actual auditions, in an individual fixed-effects framework, we find that the screen increases the probability a woman will be advanced and hired. Although some of our estimates have large standard errors and there is one persistent effect in the opposite direction, the weight of the evidence suggests that the blind audition procedure fostered impartiality in hiring and increased the proportion women in symphony orchestras. https://www.aeaweb.org/articles?id=10.1257/aer.90.4.715 https://www.aeaweb.org/articles?id=10.1257/aer.90.4.715

no_wizard 2y ago

You realize this is a huge problem right? Even if that was my only complaint (it’s not!) that is 100% not acceptable. If it’s doing it with CVs it’s doing it with other things in other scenarios. Another is that a health insurance company was caught using AI to determine if a claim should be denied or not which lead to a scandal as a whistleblower leaked the practice, as it was wrought with errors and ethical concerns

justinclift 2y ago

It's almost guaranteed there are people trying to sell an AI version of Robodebt right now to the Australian government, even though the last (non-AI) version of it was an absolute cluster fuck: https://en.wikipedia.org/wiki/Robodebt_scheme https://en.wikipedia.org/wiki/Robodebt_scheme Other governments around the world have done similar (non-AI) things in the past with similar terrible results. They'll likely try an AI version of things too in the near future, just because "AI" apparently solves all the problems. Ugh.

BobbyJo 2y ago

I think a worse feature for them to have is consistency. Imagine being someone that has fallen through the cracks of software a significant number of employers are using... Basically soft-locked out of employment with no recourse.

corimaith 2y ago
I'd argue consistency in discrimination is easier because it makes it easier to detect and account for. A range of different people interviewing are going to be alot harder to pin down.
- account42 2y ago
  Discrimination in this sense is a statistical property. If Jimbob gets rejected everywhere as an individual it isn't really discrimination but that doesn't mean it isn't an issue.

deleted 2y ago

[deleted]