5 ms·

There is an underlying assumption there that is certainly incorrect. So many stupid comments about AI boil down to "humans are incredibly good at X, we can't r

There is an underlying assumption there that is certainly incorrect. So many stupid comments about AI boil down to "humans are incredibly good at X, we can't risk having AI do it". Humans are bad at all manner of things. There are all kinds of bad human decisions being made in insurance, health care, construction, investing, everywhere. It's one big joke to suggest we are good at all this stuff.

chii 1y ago
The fear is that by delegating to an ai, there's no recourse if the outcome for the person is undesirable (correctly or not). What is needed from the AI is a trace/line-of-reasoning to which a decision is derived. Like a court judgement, which has explanations attached. This should be available (or be made as part of the decision documentation).
- stephen_g 1y ago
  But also an appeals process where it can be escalated to a real person. There should be nothing where any kind of AI system can make a final decision. I think the safest would be for a reviewer in an appeal process to even not have any access to any of the AI's decision or reasoning, since if the incorrect decision was based on hallucinated information, a reviewer might be biased to think it's true even if it was imagined.
  - JimDabell 1y ago
    > There should be nothing where any kind of AI system can make a final decision. This would forbid things like spam filters.
    - guappa 1y ago
      You never look into your spam folder?
      - JimDabell 1y ago
        A huge amount of spam never makes it into spam folders; it’s discarded silently.
        guappa 1y ago
        If you use microsoft outlook most of the spam never makes it into the spam folder because it goes to the inbox. Do you have a source for your somewhat unbelievable claim?
        bandrami 1y ago
        Mail server op here: mail exchangers (mine included) absolutely silently drop insane amounts of email submissions without any indication to the sender or the envelope recipient. At most there's a log the operator can look at somewhere that notes the rejection and whatever rules it was based on.
        guappa 1y ago
        With no access to logs the claim is just handwaving.
        bandrami 1y ago
        But I have access to logs, which is why I can describe to you how minimal the logging of email rejections is
        sokoloff 1y ago
        I run a family email server. I’ve worked with a few mail sysadmins trying to diagnose why some of our mail doesn’t get through. Even with a devoted and cooperative admin on the other side, I can absolutely believe that general logging levels are low and lots of mail is silently discarded as we’ve seen that in targeted tests when we both wanted to see the mail and associated log trail.
        sjsdaiuasgdia 1y ago
- danielmarkbruce 1y ago
  That's also a silly idea. Companies get sued all the time. There is recourse. If you run a bank that runs an AI model to make decisions on home loans and the effect is no black people get a loan, you are going to find yourself in court the exact same way as if you hire loan officers who are a bunch of racists. There is no difference. It is available to the entity running inference. Every single token and all the probabilities of every choice. But every decision made at every company isn't subject to a court hearing. So no, that's also a silly idea.
  - kazinator 1y ago
    Companies pretty much never get sued by anyone who is from a low income or otherwise disadvantaged group. It has to be pro bono work, or a big class action suit.
    - danielmarkbruce 1y ago
      Completely wrong. Companies get sued by various government agencies all the time. The FTC, DOJ, SEC, EPA, CFPB, probably others I can't think of. Even if you were right, AI doesn't change any of it - companies are liable.
      - cess11 1y ago
        So when the corporations and the state decide to be nasty, that's OK?
        genewitch 1y ago
        I've been getting a real "well, they deserved it" vibe off this site tonight. Thanks for not being that way.
        ToucanLoucan 1y ago
        The user base for this site is incredibly privileged. Not meant as a judgement just a stated observation.
- jdietrich 1y ago
  The EU has already addressed this in the GDPR. Using AI is fine, but individuals have a right to demand a manual review. Companies and government agencies can delegate work to AI, but they can't delegate responsibility. https://gdpr-info.eu/art-22-gdpr/ https://gdpr-info.eu/art-22-gdpr/
  - briandear 1y ago
    A manual review then it’ll be rejected as well. Some HR person who doesn’t know the difference between Java and JavaScript isn’t going to result in better decisions. The problem is and has always been top of funnel screening.
- const_cast 1y ago
  The danger is that AI can be biased, so if we don't know how it's making it's decisions it could be stupid. I mean, imagine that making an insurance claim with a black-sounding name results in a 5% greater chance of being rejected. How would we even know if this is the case? And, how do we prevent that? Now, of course humans are biased too, but there's no guarantee that the biases of humans are the same as the biases of whatever AI model is chosen. And with humans we can hold them accountable to some degree. We need that accountability with AI agents.
kazinator 1y ago
Bad decisions in insurance are, roughly speaking, on the side of over-approving. AI will perform tirelessly and consistently at maximizing rejections. It will leave no stone unturned in search for justifications why a claim ought to be denied.
- squidbeak 1y ago
  This has an easy public policy fix through something like a national insurance claim assessment agency, with an impartial prompt, which AI will make reasonably cheap to fund. It's always been perverse that insurance companies judge the merits of their own liabilities.
- roenxi 1y ago
  That isn't what the incentives point too; in a free market the insurers are aligned to accuracy. An insurer who has a reputation of being unreasonable about payouts won't have any customers - what is the point of taking out a policy if you expect the insurer to be unreasonable about paying? It'd take an odd customer to sign up for that. If they over-approve they will be unprofitable because their premiums aren't high enough. If they under-approve it'll be because their customers go elsewhere.
  - Coffeewine 1y ago
    But I expect my insurer to be unreasonable about paying today. It’s just that A) I didn’t choose this insurer, my employer did and on balance the total package isn’t such that I want a new employer and B) I expect pretty much all my available insurance companies to be unreasonable.
    - roenxi 1y ago
      I did say "free market"; that is sort of the standard disclaimer to show I'm not talking about madness that is the US healthcare system. Or if you have your employer choosing insurance for you and it isn't health-related then that is kinda weird and I'm not surprised it is going badly.
  - 1y ago
otabdeveloper4 1y ago
AI will be used to justify the existence of bad decisions. Now that we have an excuse in the form of "AI" we don't need to fix or own our bad decision mistakes.
j1436go 1y ago
Looking at the current state of AI models that assist in software engineering I don't have much faith in it being any better, quite the contrary.
buescher 1y ago
"Human's aren't perfect at X, so it doesn't matter if an unaccountable person who's plausibly wrong about everything does it or an AI". What's the difference, anyway?
bluefirebrand 1y ago
"A computer can never be held accountable, therefore a computer should never make a management decision" Quote from an IBM training manual from 1979 Seems just as true and even more relevant today than it was back then
- thunky 1y ago
  That's why you hold the humans that run the computer accountable.
  - bluefirebrand 1y ago
    That's sort of the whole point though The computer allows the humans a lot of leeway. For one thing, the computer represents a large diffusion of responsibility. Do you hold the programmers responsible? The managers who told them to build the software? How about the hardware manufacturers or the IT people who built or installed the system? Maybe the Executives? What if the program was built with all good intentions and just has a critical exploit that someone abused? It's just not so straightforward to hold the humans accountable when there are so many humans that touch any piece of commercial software
    - thunky 1y ago
      That's fair but I was referering to the humans that delegate "business decisions" to computers, which is what I thought the context was... For example, if American Airlines uses a computer to help decide who gets refunds, they can't then blame the computer when it discriminates against group X, or steals money, because it was their own "business decision" that is responsible for that action (with the assist from a stupid computer they chose to use). This is different from when their landing gear doesn't go down because of a software flaw in some component. They didn't produce the component and they didn't they didn't choose to delegate their "business decisions" to it, so as long as they used an approved vendor etc they should be ok. Choosing the vendor, the maintainence schedules, etc, etc: those are the "business decisions" they're responsible for.

chii 1y ago

The fear is that by delegating to an ai, there's no recourse if the outcome for the person is undesirable (correctly or not). What is needed from the AI is a trace/line-of-reasoning to which a decision is derived. Like a court judgement, which has explanations attached. This should be available (or be made as part of the decision documentation).

stephen_g 1y ago
But also an appeals process where it can be escalated to a real person. There should be nothing where any kind of AI system can make a final decision. I think the safest would be for a reviewer in an appeal process to even not have any access to any of the AI's decision or reasoning, since if the incorrect decision was based on hallucinated information, a reviewer might be biased to think it's true even if it was imagined.
- JimDabell 1y ago
  > There should be nothing where any kind of AI system can make a final decision. This would forbid things like spam filters.
  - guappa 1y ago
    You never look into your spam folder?
    - JimDabell 1y ago
      A huge amount of spam never makes it into spam folders; it’s discarded silently.
      - guappa 1y ago
        If you use microsoft outlook most of the spam never makes it into the spam folder because it goes to the inbox. Do you have a source for your somewhat unbelievable claim?
        bandrami 1y ago
        Mail server op here: mail exchangers (mine included) absolutely silently drop insane amounts of email submissions without any indication to the sender or the envelope recipient. At most there's a log the operator can look at somewhere that notes the rejection and whatever rules it was based on.
        guappa 1y ago
        With no access to logs the claim is just handwaving.
        bandrami 1y ago
        But I have access to logs, which is why I can describe to you how minimal the logging of email rejections is
        sokoloff 1y ago
        I run a family email server. I’ve worked with a few mail sysadmins trying to diagnose why some of our mail doesn’t get through. Even with a devoted and cooperative admin on the other side, I can absolutely believe that general logging levels are low and lots of mail is silently discarded as we’ve seen that in targeted tests when we both wanted to see the mail and associated log trail.
        sjsdaiuasgdia 1y ago
danielmarkbruce 1y ago
That's also a silly idea. Companies get sued all the time. There is recourse. If you run a bank that runs an AI model to make decisions on home loans and the effect is no black people get a loan, you are going to find yourself in court the exact same way as if you hire loan officers who are a bunch of racists. There is no difference. It is available to the entity running inference. Every single token and all the probabilities of every choice. But every decision made at every company isn't subject to a court hearing. So no, that's also a silly idea.
- kazinator 1y ago
  Companies pretty much never get sued by anyone who is from a low income or otherwise disadvantaged group. It has to be pro bono work, or a big class action suit.
  - danielmarkbruce 1y ago
    Completely wrong. Companies get sued by various government agencies all the time. The FTC, DOJ, SEC, EPA, CFPB, probably others I can't think of. Even if you were right, AI doesn't change any of it - companies are liable.
    - cess11 1y ago
      So when the corporations and the state decide to be nasty, that's OK?
      - genewitch 1y ago
        I've been getting a real "well, they deserved it" vibe off this site tonight. Thanks for not being that way.
        ToucanLoucan 1y ago
        The user base for this site is incredibly privileged. Not meant as a judgement just a stated observation.
jdietrich 1y ago
The EU has already addressed this in the GDPR. Using AI is fine, but individuals have a right to demand a manual review. Companies and government agencies can delegate work to AI, but they can't delegate responsibility. https://gdpr-info.eu/art-22-gdpr/ https://gdpr-info.eu/art-22-gdpr/
- briandear 1y ago
  A manual review then it’ll be rejected as well. Some HR person who doesn’t know the difference between Java and JavaScript isn’t going to result in better decisions. The problem is and has always been top of funnel screening.
const_cast 1y ago
The danger is that AI can be biased, so if we don't know how it's making it's decisions it could be stupid. I mean, imagine that making an insurance claim with a black-sounding name results in a 5% greater chance of being rejected. How would we even know if this is the case? And, how do we prevent that? Now, of course humans are biased too, but there's no guarantee that the biases of humans are the same as the biases of whatever AI model is chosen. And with humans we can hold them accountable to some degree. We need that accountability with AI agents.

kazinator 1y ago

Bad decisions in insurance are, roughly speaking, on the side of over-approving. AI will perform tirelessly and consistently at maximizing rejections. It will leave no stone unturned in search for justifications why a claim ought to be denied.

squidbeak 1y ago
This has an easy public policy fix through something like a national insurance claim assessment agency, with an impartial prompt, which AI will make reasonably cheap to fund. It's always been perverse that insurance companies judge the merits of their own liabilities.
roenxi 1y ago
That isn't what the incentives point too; in a free market the insurers are aligned to accuracy. An insurer who has a reputation of being unreasonable about payouts won't have any customers - what is the point of taking out a policy if you expect the insurer to be unreasonable about paying? It'd take an odd customer to sign up for that. If they over-approve they will be unprofitable because their premiums aren't high enough. If they under-approve it'll be because their customers go elsewhere.
- Coffeewine 1y ago
  But I expect my insurer to be unreasonable about paying today. It’s just that A) I didn’t choose this insurer, my employer did and on balance the total package isn’t such that I want a new employer and B) I expect pretty much all my available insurance companies to be unreasonable.
  - roenxi 1y ago
    I did say "free market"; that is sort of the standard disclaimer to show I'm not talking about madness that is the US healthcare system. Or if you have your employer choosing insurance for you and it isn't health-related then that is kinda weird and I'm not surprised it is going badly.
- 1y ago

otabdeveloper4 1y ago

AI will be used to justify the existence of bad decisions. Now that we have an excuse in the form of "AI" we don't need to fix or own our bad decision mistakes.

j1436go 1y ago

Looking at the current state of AI models that assist in software engineering I don't have much faith in it being any better, quite the contrary.

buescher 1y ago

"Human's aren't perfect at X, so it doesn't matter if an unaccountable person who's plausibly wrong about everything does it or an AI". What's the difference, anyway?

bluefirebrand 1y ago

"A computer can never be held accountable, therefore a computer should never make a management decision" Quote from an IBM training manual from 1979 Seems just as true and even more relevant today than it was back then

thunky 1y ago
That's why you hold the humans that run the computer accountable.
- bluefirebrand 1y ago
  That's sort of the whole point though The computer allows the humans a lot of leeway. For one thing, the computer represents a large diffusion of responsibility. Do you hold the programmers responsible? The managers who told them to build the software? How about the hardware manufacturers or the IT people who built or installed the system? Maybe the Executives? What if the program was built with all good intentions and just has a critical exploit that someone abused? It's just not so straightforward to hold the humans accountable when there are so many humans that touch any piece of commercial software
  - thunky 1y ago
    That's fair but I was referering to the humans that delegate "business decisions" to computers, which is what I thought the context was... For example, if American Airlines uses a computer to help decide who gets refunds, they can't then blame the computer when it discriminates against group X, or steals money, because it was their own "business decision" that is responsible for that action (with the assist from a stupid computer they chose to use). This is different from when their landing gear doesn't go down because of a software flaw in some component. They didn't produce the component and they didn't they didn't choose to delegate their "business decisions" to it, so as long as they used an approved vendor etc they should be ok. Choosing the vendor, the maintainence schedules, etc, etc: those are the "business decisions" they're responsible for.