5 ms·

> Imagine applying for a job, only to find out that an algorithm powered by artificial intelligence (AI) rejected your resume before a human even saw it The ar

by Bjartr 1y ago

> Imagine applying for a job, only to find out that an algorithm powered by artificial intelligence (AI) rejected your resume before a human even saw it The article says this like it's a new problem. Automated resume screening is a long established practice at this point. That it'll be some LLM doing the screening instead of a keyword matcher doesn't change much. Although, it could be argued that an LLM would better approximate an actual human looking at the resume... including all the biases. It's not like companies take responsibility for such automated systems today. I think they're used partly for liability cya anyway. The fewer actual employees that look at resumes, the fewer that can screw up and expose the company to a lawsuit. An algorithm can screw up too of course, but it's a lot harder to show intent, which can affect the damages awarded I think. Of course IANAL, so this could be entirely wrong. Interesting to think about though.

GeorgeCurtis 1y ago
> It also influences hiring decisions, rental applications, loans, credit scoring, social media feeds, government services and even what news or information we see when we search online. I think, more frighteningly, the potential for it to make decisions on insurance claims and medical care.
- danielmarkbruce 1y ago
  There is an underlying assumption there that is certainly incorrect. So many stupid comments about AI boil down to "humans are incredibly good at X, we can't risk having AI do it". Humans are bad at all manner of things. There are all kinds of bad human decisions being made in insurance, health care, construction, investing, everywhere. It's one big joke to suggest we are good at all this stuff.
  - chii 1y ago
    The fear is that by delegating to an ai, there's no recourse if the outcome for the person is undesirable (correctly or not). What is needed from the AI is a trace/line-of-reasoning to which a decision is derived. Like a court judgement, which has explanations attached. This should be available (or be made as part of the decision documentation).
    - stephen_g 1y ago
      But also an appeals process where it can be escalated to a real person. There should be nothing where any kind of AI system can make a final decision. I think the safest would be for a reviewer in an appeal process to even not have any access to any of the AI's decision or reasoning, since if the incorrect decision was based on hallucinated information, a reviewer might be biased to think it's true even if it was imagined.
      - JimDabell 1y ago
        > There should be nothing where any kind of AI system can make a final decision. This would forbid things like spam filters.
        guappa 1y ago
        You never look into your spam folder?
        JimDabell 1y ago
        A huge amount of spam never makes it into spam folders; it’s discarded silently.
        guappa 1y ago
        If you use microsoft outlook most of the spam never makes it into the spam folder because it goes to the inbox. Do you have a source for your somewhat unbelievable claim?
        bandrami 1y ago
        Mail server op here: mail exchangers (mine included) absolutely silently drop insane amounts of email submissions without any indication to the sender or the envelope recipient. At most there's a log the operator can look at somewhere that notes the rejection and whatever rules it was based on.
  - kazinator 1y ago
    Bad decisions in insurance are, roughly speaking, on the side of over-approving. AI will perform tirelessly and consistently at maximizing rejections. It will leave no stone unturned in search for justifications why a claim ought to be denied.
    - squidbeak 1y ago
      This has an easy public policy fix through something like a national insurance claim assessment agency, with an impartial prompt, which AI will make reasonably cheap to fund. It's always been perverse that insurance companies judge the merits of their own liabilities.
    - roenxi 1y ago
      That isn't what the incentives point too; in a free market the insurers are aligned to accuracy. An insurer who has a reputation of being unreasonable about payouts won't have any customers - what is the point of taking out a policy if you expect the insurer to be unreasonable about paying? It'd take an odd customer to sign up for that. If they over-approve they will be unprofitable because their premiums aren't high enough. If they under-approve it'll be because their customers go elsewhere.
      - Coffeewine 1y ago
        But I expect my insurer to be unreasonable about paying today. It’s just that A) I didn’t choose this insurer, my employer did and on balance the total package isn’t such that I want a new employer and B) I expect pretty much all my available insurance companies to be unreasonable.
        roenxi 1y ago
        I did say "free market"; that is sort of the standard disclaimer to show I'm not talking about madness that is the US healthcare system. Or if you have your employer choosing insurance for you and it isn't health-related then that is kinda weird and I'm not surprised it is going badly.
      - 1y ago
  - otabdeveloper4 1y ago
    AI will be used to justify the existence of bad decisions. Now that we have an excuse in the form of "AI" we don't need to fix or own our bad decision mistakes.
  - j1436go 1y ago
    Looking at the current state of AI models that assist in software engineering I don't have much faith in it being any better, quite the contrary.
  - buescher 1y ago
    "Human's aren't perfect at X, so it doesn't matter if an unaccountable person who's plausibly wrong about everything does it or an AI". What's the difference, anyway?
  - bluefirebrand 1y ago
    "A computer can never be held accountable, therefore a computer should never make a management decision" Quote from an IBM training manual from 1979 Seems just as true and even more relevant today than it was back then
    - thunky 1y ago
      That's why you hold the humans that run the computer accountable.
      - bluefirebrand 1y ago
        That's sort of the whole point though The computer allows the humans a lot of leeway. For one thing, the computer represents a large diffusion of responsibility. Do you hold the programmers responsible? The managers who told them to build the software? How about the hardware manufacturers or the IT people who built or installed the system? Maybe the Executives? What if the program was built with all good intentions and just has a critical exploit that someone abused? It's just not so straightforward to hold the humans accountable when there are so many humans that touch any piece of commercial software
        thunky 1y ago
        That's fair but I was referering to the humans that delegate "business decisions" to computers, which is what I thought the context was... For example, if American Airlines uses a computer to help decide who gets refunds, they can't then blame the computer when it discriminates against group X, or steals money, because it was their own "business decision" that is responsible for that action (with the assist from a stupid computer they chose to use). This is different from when their landing gear doesn't go down because of a software flaw in some component. They didn't produce the component and they didn't they didn't choose to delegate their "business decisions" to it, so as long as they used an approved vendor etc they should be ok. Choosing the vendor, the maintainence schedules, etc, etc: those are the "business decisions" they're responsible for.
- Saigonautica 1y ago
  I know someone in this space. The insurance forms are processed first-pass with AI or ML (I forget which). Then the remainder are processed by humans in Viet Nam. This is not for the USA. I've also vaguely heard of a large company that provides just this as a service -- basically a factory where insurance claims are processed by humans here in VN, in one of the less affluent regions. I recall they had some minor problems with staffing as it's not a particularly pleasant job (it's very boring). On the other hand, the region has few employment opportunities, so perhaps it's good for some people too. I'm not sure which country this last one is processing forms for. It may, or may not be the USA. I don't really have an opinion to offer -- I just thought you might find that interesting.
- PieTime 1y ago
  I know people who’ve died because of AI algorithms years ago. They’ve implemented state programs with no legal oversight and only governed by an algorithm.
BlueTemplar 1y ago
Good point : like when a killing is done by a machine that wasn't directly operated, the perpetrator might be found guilty, but for manslaughter rather than murder ?
JKCalhoun 1y ago
> The article says this like it's a new problem. I suspect though there might be something different today in terms of scale. Bigger corporations perhaps did some kind of screening (I am not aware of it though — at Apple I was asked to personally submit resumés for people I knew that were looking for engineering jobs — perhaps there was automation in other parts of the company). I doubt the restaurants around Omaha were doing any automation on screening resumés. That probably just got a lot easier with the pervasiveness of LLMs.
lotu 1y ago
> An algorithm can screw up too of course, but it's a lot harder to show intent, which can affect the damages awarded I think. I personally think an algorithm would be easier to show intent, or a least willful negligence, it would also magnify the harm. A employee might only make mistakes on occasion, but an algorithm will make it every single time. The benefit of an algorithm is it does not need to be reminded do or not to do something, and it's actions are easier to interrogate than a humans.

by Bjartr 1y ago

GeorgeCurtis 1y ago

> It also influences hiring decisions, rental applications, loans, credit scoring, social media feeds, government services and even what news or information we see when we search online. I think, more frighteningly, the potential for it to make decisions on insurance claims and medical care.

danielmarkbruce 1y ago
There is an underlying assumption there that is certainly incorrect. So many stupid comments about AI boil down to "humans are incredibly good at X, we can't risk having AI do it". Humans are bad at all manner of things. There are all kinds of bad human decisions being made in insurance, health care, construction, investing, everywhere. It's one big joke to suggest we are good at all this stuff.
- chii 1y ago
  The fear is that by delegating to an ai, there's no recourse if the outcome for the person is undesirable (correctly or not). What is needed from the AI is a trace/line-of-reasoning to which a decision is derived. Like a court judgement, which has explanations attached. This should be available (or be made as part of the decision documentation).
  - stephen_g 1y ago
    But also an appeals process where it can be escalated to a real person. There should be nothing where any kind of AI system can make a final decision. I think the safest would be for a reviewer in an appeal process to even not have any access to any of the AI's decision or reasoning, since if the incorrect decision was based on hallucinated information, a reviewer might be biased to think it's true even if it was imagined.
    - JimDabell 1y ago
      > There should be nothing where any kind of AI system can make a final decision. This would forbid things like spam filters.
      - guappa 1y ago
        You never look into your spam folder?
        JimDabell 1y ago
        A huge amount of spam never makes it into spam folders; it’s discarded silently.
        guappa 1y ago
        If you use microsoft outlook most of the spam never makes it into the spam folder because it goes to the inbox. Do you have a source for your somewhat unbelievable claim?
        bandrami 1y ago
        Mail server op here: mail exchangers (mine included) absolutely silently drop insane amounts of email submissions without any indication to the sender or the envelope recipient. At most there's a log the operator can look at somewhere that notes the rejection and whatever rules it was based on.
- kazinator 1y ago
  Bad decisions in insurance are, roughly speaking, on the side of over-approving. AI will perform tirelessly and consistently at maximizing rejections. It will leave no stone unturned in search for justifications why a claim ought to be denied.
  - squidbeak 1y ago
    This has an easy public policy fix through something like a national insurance claim assessment agency, with an impartial prompt, which AI will make reasonably cheap to fund. It's always been perverse that insurance companies judge the merits of their own liabilities.
  - roenxi 1y ago
    That isn't what the incentives point too; in a free market the insurers are aligned to accuracy. An insurer who has a reputation of being unreasonable about payouts won't have any customers - what is the point of taking out a policy if you expect the insurer to be unreasonable about paying? It'd take an odd customer to sign up for that. If they over-approve they will be unprofitable because their premiums aren't high enough. If they under-approve it'll be because their customers go elsewhere.
    - Coffeewine 1y ago
      But I expect my insurer to be unreasonable about paying today. It’s just that A) I didn’t choose this insurer, my employer did and on balance the total package isn’t such that I want a new employer and B) I expect pretty much all my available insurance companies to be unreasonable.
      - roenxi 1y ago
        I did say "free market"; that is sort of the standard disclaimer to show I'm not talking about madness that is the US healthcare system. Or if you have your employer choosing insurance for you and it isn't health-related then that is kinda weird and I'm not surprised it is going badly.
    - 1y ago
- otabdeveloper4 1y ago
  AI will be used to justify the existence of bad decisions. Now that we have an excuse in the form of "AI" we don't need to fix or own our bad decision mistakes.
- j1436go 1y ago
  Looking at the current state of AI models that assist in software engineering I don't have much faith in it being any better, quite the contrary.
- buescher 1y ago
  "Human's aren't perfect at X, so it doesn't matter if an unaccountable person who's plausibly wrong about everything does it or an AI". What's the difference, anyway?
- bluefirebrand 1y ago
  "A computer can never be held accountable, therefore a computer should never make a management decision" Quote from an IBM training manual from 1979 Seems just as true and even more relevant today than it was back then
  - thunky 1y ago
    That's why you hold the humans that run the computer accountable.
    - bluefirebrand 1y ago
      That's sort of the whole point though The computer allows the humans a lot of leeway. For one thing, the computer represents a large diffusion of responsibility. Do you hold the programmers responsible? The managers who told them to build the software? How about the hardware manufacturers or the IT people who built or installed the system? Maybe the Executives? What if the program was built with all good intentions and just has a critical exploit that someone abused? It's just not so straightforward to hold the humans accountable when there are so many humans that touch any piece of commercial software
      - thunky 1y ago
        That's fair but I was referering to the humans that delegate "business decisions" to computers, which is what I thought the context was... For example, if American Airlines uses a computer to help decide who gets refunds, they can't then blame the computer when it discriminates against group X, or steals money, because it was their own "business decision" that is responsible for that action (with the assist from a stupid computer they chose to use). This is different from when their landing gear doesn't go down because of a software flaw in some component. They didn't produce the component and they didn't they didn't choose to delegate their "business decisions" to it, so as long as they used an approved vendor etc they should be ok. Choosing the vendor, the maintainence schedules, etc, etc: those are the "business decisions" they're responsible for.
Saigonautica 1y ago
I know someone in this space. The insurance forms are processed first-pass with AI or ML (I forget which). Then the remainder are processed by humans in Viet Nam. This is not for the USA. I've also vaguely heard of a large company that provides just this as a service -- basically a factory where insurance claims are processed by humans here in VN, in one of the less affluent regions. I recall they had some minor problems with staffing as it's not a particularly pleasant job (it's very boring). On the other hand, the region has few employment opportunities, so perhaps it's good for some people too. I'm not sure which country this last one is processing forms for. It may, or may not be the USA. I don't really have an opinion to offer -- I just thought you might find that interesting.
PieTime 1y ago
I know people who’ve died because of AI algorithms years ago. They’ve implemented state programs with no legal oversight and only governed by an algorithm.

BlueTemplar 1y ago

Good point : like when a killing is done by a machine that wasn't directly operated, the perpetrator might be found guilty, but for manslaughter rather than murder ?

JKCalhoun 1y ago

> The article says this like it's a new problem. I suspect though there might be something different today in terms of scale. Bigger corporations perhaps did some kind of screening (I am not aware of it though — at Apple I was asked to personally submit resumés for people I knew that were looking for engineering jobs — perhaps there was automation in other parts of the company). I doubt the restaurants around Omaha were doing any automation on screening resumés. That probably just got a lot easier with the pervasiveness of LLMs.

lotu 1y ago

> An algorithm can screw up too of course, but it's a lot harder to show intent, which can affect the damages awarded I think. I personally think an algorithm would be easier to show intent, or a least willful negligence, it would also magnify the harm. A employee might only make mistakes on occasion, but an algorithm will make it every single time. The benefit of an algorithm is it does not need to be reminded do or not to do something, and it's actions are easier to interrogate than a humans.