5 ms·

Asking LLMs to do tasks like this and expecting any useful result is mind boggling to me. The LLM is going to guess at what a human on the internet may have sa

Asking LLMs to do tasks like this and expecting any useful result is mind boggling to me. The LLM is going to guess at what a human on the internet may have said in response, nothing more. We haven't solved interpretability and we don't actually know how these things work, stop believing the marketing that they "reason" or are anything comparable to human intelligence.

SomeoneOnTheWeb 1y ago
Problem is, the vast majority of people aren't aware of that. So it'll keep on being this way for the foreseeable future.
- Loughla 1y ago
  Companies are calling it AI. It's not the layman's fault that they expect it to be AI.
ToucanLoucan 1y ago
> Asking LLMs to do tasks like this and expecting any useful result is mind boggling to me. Most of the people who are very interested in using LLM/generative media are very open about the fact that they don't care about the results. If they did, they wouldn't outsource them to a random media generator. And for a certain kind of hiring manager in a certain kind of firm that regularly finds itself on the wrong end of discrimination notices, they'd probably use this for the exact reason it's posted about here, because it lets them launder decision-making through an entity that (probably?) won't get them sued and will produce the biased decisions they want. "Our hiring decisions can't be racist! A computer made them." Look out for tons of firms in the FIRE sector doing the exact same thing for the exact same reason, except not just hiring decisions: insurance policies that exclude the things you're most likely to need claims for, which will be sold as: "personalized coverage just for you!" Or perhaps you'll be denied a mortgage because you come from a ZIP code that denotes you're more likely than most to be in poverty for life, and the banks' AI marks you as "high risk." Fantastic new vectors for systemic discrimination, with the plausible deniability to ensure victims will never see justice.
john-h-k 1y ago
> We haven't solved interpretability and we don't actually know how these things work But right above this you made a statement about how they work. You can’t claim we know how they work to support your opinion, and then claim we don’t to break down the opposite opinion
- mapt 1y ago
  I can intuit that you hated me the moment you saw me at the interview. Because I've observed how hatred works, and I have a decent Theory of Mind model of the human condition. I can't tell if you hate me because I'm Arab, if it's because I'm male, if it's because I cut you off in traffic yesterday, if it's because my mustache reminds you of a sexual assault you suffered last May, if it's because my breath stinks of garlic today, if it's because I'm wearing Crocs, if it's because you didn't like my greeting, if it's because you already decided to hire your friend's nephew and despise the waste of time you have to spend on the interview process, if it's because you had an employee five years ago with my last name and you had a bad experience with them, if it's because I do most of my work in a programming language that you have dogmatic disagreements with, if it's because I got started in a coding bootcamp and you consider those inferior, if one of my references decided to talk shit about me, or if I'm just grossly underqualified based on my resume and you can't believe I had the balls to apply. Some of those rationales have Strong Legal Implications. When asked to explain rationales, these LLMs are observed to lie frequently. The default for machine intelligence is to incorporate all information available and search for correlations that raise the performance against a goal metric, including information that humans are legally forbidden to consider like protected class status. LLM agent models have also been observed to seek out this additional information, use it, and then lie about it (see: EXIF tags). Another problem is that machine intelligence works best when provided with trillions of similar training inputs with non-noisy goal metrics. Hiring is a very poorly generalizable problem, and the struggles of hiring a shift manager at Taco Bell are just Different from the struggles of hiring a plumber to build an irrigation trunkline or the struggles of hiring a personal assistant to follow you around or the struggles of hiring the VP reporting to the CTO. Before LLMs they were so different as to be laughable; After LLMs they are still different, but the LLM can convincingly lie to you that it has expertise in each one.
anonu 1y ago
> Asking LLMs to do tasks like this and expecting any useful result is mind boggling to me. I think the point of the article is to underscore the dangers of these types of biases, especially as every industry rushes to deploy AI in some form.
- this_user 1y ago
  AI is not the problem here, because it has merely learned what humans in the same position would do. The difference is that AI makes these biases more visible, because you can feed it resumes all day and create a statistic, whereas the same experiment cannot realistically be done with a human hiring manager.
  - im3w1l 1y ago
    I don't think that's the case. It's true that AI models are trained to mimic human speech, but that's not all there is to it. The people making the models have discretion over what goes into the training set and what doesn't. Furthermore they will do some alignment step afterwards to make the AI have the desired opinions. This means that you can not count on the AI to be representative of what people in the same position would do. It could be more biased or less biased. In all likelihood it differs from model to model.
    - _heimdall 1y ago
      > Furthermore they will do some alignment step afterwards to make the AI have the desired opinions. This requires more clarification. It isn't really alignment work done at that point, or anywhere in the process, because we haven't figured out how to align the models to human desires. We haven't even figured out how to align among other humans. At that step they are fine tuning the various controls used during inference until they are happy with the outputs given for specific inputs. The model is still a black box, they're making somewhat educates guesses on how to adjust said knobs but they don't really know what changes internally and they definitely don't know intent (if the LLM has developed intent). These models as we understand them also don't have opinions and can't themselves be biased. Bias is recognized by us, but again its only based on the output as we don't know why any specific output was generated. An LLM may output something most people would read as racist, for example, but that says nothing of why the output was generated and whether the model even really understands race as a concept or cared about it at all when answering.
mpweiher 1y ago
> what a human on the internet may have said in response Yes. Except. The current societal narrative is still that of discrimination against female candidates, research such as Williams/Ceci[1]. But apparently the actual societal bias, if that is what is reflected by these LLMs, is against male candidates. So the result is the opposite of what a human on the internet is likely to have said, but it matches how humans in society act. [1] https://www.pnas.org/doi/10.1073/pnas.1418878112 https://www.pnas.org/doi/10.1073/pnas.1418878112
- gitremote 1y ago
  This study shows the opposite: > In their study, Moss-Racusin and her colleagues created a fictitious resume of an applicant for a lab manager position. Two versions of the resume were produced that varied in only one, very significant, detail: the name at the top. One applicant was named Jennifer and the other John. Moss-Racusin and her colleagues then asked STEM professors from across the country to assess the resume. Over one hundred biologists, chemists, and physicists at academic institutions agreed to do so. Each scientist was randomly assigned to review either Jennifer or John's resume. > The results were surprising—they show that the decision makers did not evaluate the resume purely on its merits. Despite having the exact same qualifications and experience as John, Jennifer was perceived as significantly less competent. As a result, Jenifer experienced a number of disadvantages that would have hindered her career advancement if she were a real applicant. Because they perceived the female candidate as less competent, the scientists in the study were less willing to mentor Jennifer or to hire her as a lab manager. They also recommended paying her a lower salary. Jennifer was offered, on average, $4,000 per year (13%) less than John. https://gender.stanford.edu/news/why-does-john-get-stem-job-rather-jennifer https://gender.stanford.edu/news/why-does-john-get-stem-job-...
  - mpweiher 1y ago
    Except that the Ceci/Williams study is (a) more recent (b) has a much larger sample size and (c) shows a larger effect. It is also arguably a much better designed study. Yet, Moss-Racusin gets cited a lot more. Because it fits the dominant narrative, whereas the better Ceci/Williams study contradicts the dominant narrative. More here: Scientific Bias in Favor of Studies Finding Gender Bias -- Studies that find bias against women often get disproportionate attention. https://www.psychologytoday.com/us/blog/rabble-rouser/201906/scientific-bias-in-favor-studies-finding-gender-bias https://www.psychologytoday.com/us/blog/rabble-rouser/201906...

SomeoneOnTheWeb 1y ago

Problem is, the vast majority of people aren't aware of that. So it'll keep on being this way for the foreseeable future.

Loughla 1y ago
Companies are calling it AI. It's not the layman's fault that they expect it to be AI.

ToucanLoucan 1y ago

> Asking LLMs to do tasks like this and expecting any useful result is mind boggling to me. Most of the people who are very interested in using LLM/generative media are very open about the fact that they don't care about the results. If they did, they wouldn't outsource them to a random media generator. And for a certain kind of hiring manager in a certain kind of firm that regularly finds itself on the wrong end of discrimination notices, they'd probably use this for the exact reason it's posted about here, because it lets them launder decision-making through an entity that (probably?) won't get them sued and will produce the biased decisions they want. "Our hiring decisions can't be racist! A computer made them." Look out for tons of firms in the FIRE sector doing the exact same thing for the exact same reason, except not just hiring decisions: insurance policies that exclude the things you're most likely to need claims for, which will be sold as: "personalized coverage just for you!" Or perhaps you'll be denied a mortgage because you come from a ZIP code that denotes you're more likely than most to be in poverty for life, and the banks' AI marks you as "high risk." Fantastic new vectors for systemic discrimination, with the plausible deniability to ensure victims will never see justice.

john-h-k 1y ago

> We haven't solved interpretability and we don't actually know how these things work But right above this you made a statement about how they work. You can’t claim we know how they work to support your opinion, and then claim we don’t to break down the opposite opinion

mapt 1y ago
I can intuit that you hated me the moment you saw me at the interview. Because I've observed how hatred works, and I have a decent Theory of Mind model of the human condition. I can't tell if you hate me because I'm Arab, if it's because I'm male, if it's because I cut you off in traffic yesterday, if it's because my mustache reminds you of a sexual assault you suffered last May, if it's because my breath stinks of garlic today, if it's because I'm wearing Crocs, if it's because you didn't like my greeting, if it's because you already decided to hire your friend's nephew and despise the waste of time you have to spend on the interview process, if it's because you had an employee five years ago with my last name and you had a bad experience with them, if it's because I do most of my work in a programming language that you have dogmatic disagreements with, if it's because I got started in a coding bootcamp and you consider those inferior, if one of my references decided to talk shit about me, or if I'm just grossly underqualified based on my resume and you can't believe I had the balls to apply. Some of those rationales have Strong Legal Implications. When asked to explain rationales, these LLMs are observed to lie frequently. The default for machine intelligence is to incorporate all information available and search for correlations that raise the performance against a goal metric, including information that humans are legally forbidden to consider like protected class status. LLM agent models have also been observed to seek out this additional information, use it, and then lie about it (see: EXIF tags). Another problem is that machine intelligence works best when provided with trillions of similar training inputs with non-noisy goal metrics. Hiring is a very poorly generalizable problem, and the struggles of hiring a shift manager at Taco Bell are just Different from the struggles of hiring a plumber to build an irrigation trunkline or the struggles of hiring a personal assistant to follow you around or the struggles of hiring the VP reporting to the CTO. Before LLMs they were so different as to be laughable; After LLMs they are still different, but the LLM can convincingly lie to you that it has expertise in each one.

anonu 1y ago

> Asking LLMs to do tasks like this and expecting any useful result is mind boggling to me. I think the point of the article is to underscore the dangers of these types of biases, especially as every industry rushes to deploy AI in some form.

this_user 1y ago
AI is not the problem here, because it has merely learned what humans in the same position would do. The difference is that AI makes these biases more visible, because you can feed it resumes all day and create a statistic, whereas the same experiment cannot realistically be done with a human hiring manager.
- im3w1l 1y ago
  I don't think that's the case. It's true that AI models are trained to mimic human speech, but that's not all there is to it. The people making the models have discretion over what goes into the training set and what doesn't. Furthermore they will do some alignment step afterwards to make the AI have the desired opinions. This means that you can not count on the AI to be representative of what people in the same position would do. It could be more biased or less biased. In all likelihood it differs from model to model.
  - _heimdall 1y ago
    > Furthermore they will do some alignment step afterwards to make the AI have the desired opinions. This requires more clarification. It isn't really alignment work done at that point, or anywhere in the process, because we haven't figured out how to align the models to human desires. We haven't even figured out how to align among other humans. At that step they are fine tuning the various controls used during inference until they are happy with the outputs given for specific inputs. The model is still a black box, they're making somewhat educates guesses on how to adjust said knobs but they don't really know what changes internally and they definitely don't know intent (if the LLM has developed intent). These models as we understand them also don't have opinions and can't themselves be biased. Bias is recognized by us, but again its only based on the output as we don't know why any specific output was generated. An LLM may output something most people would read as racist, for example, but that says nothing of why the output was generated and whether the model even really understands race as a concept or cared about it at all when answering.

mpweiher 1y ago

> what a human on the internet may have said in response Yes. Except. The current societal narrative is still that of discrimination against female candidates, research such as Williams/Ceci[1]. But apparently the actual societal bias, if that is what is reflected by these LLMs, is against male candidates. So the result is the opposite of what a human on the internet is likely to have said, but it matches how humans in society act. [1] https://www.pnas.org/doi/10.1073/pnas.1418878112 https://www.pnas.org/doi/10.1073/pnas.1418878112

gitremote 1y ago
This study shows the opposite: > In their study, Moss-Racusin and her colleagues created a fictitious resume of an applicant for a lab manager position. Two versions of the resume were produced that varied in only one, very significant, detail: the name at the top. One applicant was named Jennifer and the other John. Moss-Racusin and her colleagues then asked STEM professors from across the country to assess the resume. Over one hundred biologists, chemists, and physicists at academic institutions agreed to do so. Each scientist was randomly assigned to review either Jennifer or John's resume. > The results were surprising—they show that the decision makers did not evaluate the resume purely on its merits. Despite having the exact same qualifications and experience as John, Jennifer was perceived as significantly less competent. As a result, Jenifer experienced a number of disadvantages that would have hindered her career advancement if she were a real applicant. Because they perceived the female candidate as less competent, the scientists in the study were less willing to mentor Jennifer or to hire her as a lab manager. They also recommended paying her a lower salary. Jennifer was offered, on average, $4,000 per year (13%) less than John. https://gender.stanford.edu/news/why-does-john-get-stem-job-rather-jennifer https://gender.stanford.edu/news/why-does-john-get-stem-job-...
- mpweiher 1y ago
  Except that the Ceci/Williams study is (a) more recent (b) has a much larger sample size and (c) shows a larger effect. It is also arguably a much better designed study. Yet, Moss-Racusin gets cited a lot more. Because it fits the dominant narrative, whereas the better Ceci/Williams study contradicts the dominant narrative. More here: Scientific Bias in Favor of Studies Finding Gender Bias -- Studies that find bias against women often get disproportionate attention. https://www.psychologytoday.com/us/blog/rabble-rouser/201906/scientific-bias-in-favor-studies-finding-gender-bias https://www.psychologytoday.com/us/blog/rabble-rouser/201906...