10 ms·

UnitedHealth uses AI model with 90% error rate to deny care, lawsuit alleges

163 points by kurthr 3y ago

deleted 3y ago
[deleted]
jghn 3y ago
So glad to be a United customer!
kelseyfrog 3y ago
What's an acceptable error rate for such a model? What's the human baseline? I ask these questions[1][2] for nearly every model I'm requested to develop. It usually throws off PMs who aren't used to any acceptable errors. It's always fun to get a request that's several orders of magnitude better than human performance. Sometimes it's actually possible to deliver, but often it shifts the conversation into feasibility and expectations. Anyone developing a model like this will have errors and definitionally makes systemic. Regardless of performance it's important to address that. 1. In this order 2. Sometimes I study human baselines but ask before revealing because it prevents anchoring.
- tekacs 3y ago
  Absolutely -- an academic environment I was in used to have a lot of government callers come by, asking for predictive power to solve any number of humanitarian crises, terrorism, crime and so on. We collectively used to ask them every time, 1. What data they had to work with. 2. What was the human baseline that they were currently executing with. 3. What performance would make a difference to this process. ... and as you say, the answers we would get would tend to sound something like. 1. Almost nothing. 2. Unclear, un-measured. 3. Something well beyond incredible, well-intentioned humans with incredible data. In that case the lesson was often to simply not try, but it does sometimes feel like other areas of government like E2E-breaking legislation, in that well-intentioned folks will try and try and try again and eventually get someone to fund and execute on the project, feasibility notwithstanding. In this instance with United... the data that we're getting via Ars and this case is so unclear and one-sided that it's hard to know how well things actually work... it'd be fun to consign them (for now) to "don't even try" if we had the info to do so, but it feels like it's a matter of time either way and so... I think the ideal with most of these cases is to eventually ship something that's actually high quality.
- raverbashing 3y ago
  > What's an acceptable error rate for such a model? What's the human baseline? Funny, I though it was socialized medicine that had "death panels" and such /s
  - kelnos 3y ago
    If anything, the "death panels" on private insurers (whatever form they might actually take) are likely much worse than a government-run "death panel".
    - willcipriano 3y ago
      I haven't heard the death panel canard brought out since Canada started it's death panels. It's like the classic idea, but instead of letting you die from lack of medical care they drive you into depression prior to killing you. > Are Canadians being driven to assisted suicide by poverty or healthcare crisis? https://www.theguardian.com/world/2022/may/11/canada-cases-right-to-die-laws https://www.theguardian.com/world/2022/may/11/canada-cases-r... > Why is Canada euthanising the poor? https://www.spectator.co.uk/article/why-is-canada-euthanising-the-poor-/ https://www.spectator.co.uk/article/why-is-canada-euthanisin...
- kelnos 3y ago
  > What's an acceptable error rate for such a model? Ideally, 0%. We're talking about people's lives and health here. But, right, realistically: > What's the human baseline? Good question, but I would hope it's much better than a 90% error rate. And if it's not, any sane company in any sane industry would figure out why things are that bad, and fix it. Of course, insurance companies are the scum of the earth, and have a huge incentive to get things wrong in this way.
  - kelseyfrog 3y ago
    Why is there a different standard between the model and a human? From the perspective of the business, whether a model or human causes the error is immaterial. > Any sane company in any sane industry would figure out why things are that bad, and fix it. The function of a thing is its consequences. So we have to conclude that the reason the model had a 90% error rate was to have a 90% error rate. It's aligned with business incentives to be so.
tekacs 3y ago
> Though few patients appeal coverage denials generally, when UnitedHealth members appeal denials based on nH Predict estimates—through internal appeals processes or through the federal Administrative Law Judge proceedings—over 90 percent of the denials are reversed, the lawsuit claims. This makes it obvious that the algorithm is wrongly denying coverage, it argues. The '90% error rate' in the title sounded like a pretty serious twisting of some numbers so I scanned the article for where it's quoted from ^, which... makes a mockery of the title. In case it's not obvious and to save parsing that, 'few people appeal' (which makes sense given the internal appeals process or Administrative Law Judge proceedings). Then, /of those cases/ 90% are overturned. - If 50 / 10,000 cases and 45 were overturned or 500 / 100,000 cases appealed and 450 were overturned... would all meet this '90% error' mark. - Similarly, 'overturned != error', because it may be as simple as a policy on United's side to not fight super hard and allow overturns (for any number of reasons, including just saving costs on the proceedings, it needn't be altruistic). Did Ars used to be better than this, or were they always like this on some issues? The /business/ side of medical insurance isn't really a topic that I know enough to weigh in on, but hopefully people can read the above sentence and parse what it's actually saying for themselves before reading the article. Irrespective of how I/we feel about the outcomes, this headline is awful.
- michaelmrose 3y ago
  Of the only claims that have been adjudicated 90% were found to have been erroneous. Without analysis there is no particular reason to believe that other denials were more justified. Few doctors on average make nonsensical requests for patient care while insurers issue spurious denials as a matter of course. You dont need to know the total number of denials instead you need a neutral party to analyze a random sample of denials. We could do this as matter of course and tie failing an adequate benchmark to ruinous penalties to company and CEO.
  - hayksaakian 3y ago
    I think the OP is suggesting that hypothetically speaking, people would only go through the hassle of appealing if they were pretty sure they would win to begin with
michaeljx 3y ago
What kind of an educated scientific person, working in health insurance and dealing with sick people, develops anything that makes this possible?
- manicennui 3y ago
  I worked on healthcare software for over a decade. The healthcare industry attracts the lowest quality people generally. Software engineers aren't giving up working at FAANG to go work in healthcare. A significant portion of UHG's revenue comes from billing and other healthcare software. Most of it is under their Optum brand.
- s5300 3y ago
  [dead]
- lispisok 3y ago
  Regular people will do horrible things to keep their paycheck coming. It's an underrated phenomena of humanity.
  - irrational 3y ago
    [flagged]
    - all2 3y ago
      It's a feature of humanity. We are intrinsically broken and prone to self preservation. No matter the systems we come up with to given our conduct, we always revert to our basic character traits.
  - HellDunkel 3y ago
    I strongly disagree with how this is presented. Doing half-assed or even passionate work for greedy insurance companies is not a horrible thing to do. Not thinking of how these companies can be reigned in and acting accordingly is far worse. We need to stop offloading moral responsibility from the people in charge to the regulae folks.
RedCardRef 3y ago
In my previous role, I worked on something similar to this. We built a model to predict which claims(CMS 1500) would be denied by the carrier(Insurance Provider) before they were submitted and allowing the hospital staff to adjust the Diagnosis/Procedure codes and validate the combination. The client had 20% denials, our model brought that denial rate down to 10%. We wanted to compare the performance of our model with commercial third party offerings and we enrolled into an early beta for a similar offering from ChangeHealthcare one of the big players in the healthcare industry, their model was wildly off(<50% accuracy), we thought we were missing something and went back and forth with their team, turns out they didnt know what they were doing, they just slapped AI on the API and tried to see if anyone would actually use it. That model never saw the light of day. After studying the data model and seeing the number of variables involved I would be skeptical of prediction models retaled to insurance in the healthcare industry.
- jpambrun 3y ago
  Now Change is owned by United Healthcare, so maybe the model you mentioned did live on..
  - catchnear4321 3y ago
    you either die the benevolent language model, or live long enough to become the… no, don’t like that. do not like that.
alberth 3y ago
Was denying 90% not the desired outcome? For an insurance company, this sounds like the ideal model.
- lusus_naturae 3y ago
  Stupid question—if an insurance provider keeps denying claims which could keep elderly patients healthy, and those patients then die, isn't the insurance provider just cutting off its own source of income? But I agree that it's about incentives, and insurance carriers are incorrectly incentivized for the service they provide vs. what they're expected to provide.
  - alberth 3y ago
    The old adage goes: “those who don’t use insurance, pay too much. And those who use insurance, pay too little”. It becomes a math problem for insurance companies to keep that equation positive. And since claims far exceed the premiums any single member pays, the easiest way to stay positive is to deny claims. Even if that sacrifices losing a few premiums.
  - bigfudge 3y ago
    >die, isn't the insurance provider just cutting off its own source of income? No because that person will have paid right up the the point they become expensive. Most people incur the vast majority of their health costs in the final year or even months of life. This is why insurance based medicine will always be problematic - there’s just too much incentive to cheap out on people when they are weakest and most powerless.
  - heavyset_go 3y ago
    Not really, the highest medical costs are incurred in old age. Insurance companies would have to make less costly payouts if they didn't have to cover older people.
  - gorjusborg 3y ago
    Another angle is premiums paid vs claims awarded. If an elderly patient has paid in 50k over a lifetime and is making a 100k claim, now is an okay time to lose that customer.
ahoka 3y ago
Less known for it, but EU GDPR allows for asking for a non automated decision.
causi 3y ago
We need regulations about what computers should and should not be allowed to do. Computers should not be allowed to make healthcare decisions. Computers should not be allowed to make legal threats or actions.
heavyset_go 3y ago
Bureaucracies and authorities love black box models because they can always point to them, shirk their own responsibility for decision making and say they're only following orders and that computers aren't biased. It's just a coincidence that the model they're paying for keeps making decisions that are beneficial to their own interests and reflect their own biases.
michaelteter 3y ago
I predict this that AI will be (intentionally) used as a scapegoat for plausible deniability in some industries. "We just followed our AI model." "We thought it was doing the job." "It's too complex to understand." Obviously once you know or believe that your AI model is wrong, and you are doing a disservice to your clients/patients... and especially once you begin discussing this internally, then you are aware of the problem and should not (get caught) sweeping it under the rug.
- iforgotpassword 3y ago
  You don't need AI for this, software is already treated as something God given by people. Government somewhere wrongly denying foodstamps to a huge amount of people? Software error, nothing anyone could possibly be responsible for. Getting hacked and all your customers data stolen (because you don't follow basic security best practices)? Well that just happens, just like a taiphoon or earthquake, nothing anyone could've done about this. But bonus points for demanding the government do something about those hacker gangs.
- blitzar 3y ago
  "Computer says no" is a catchphrase first used in the British sketch comedy television programme Little Britain in 2004. https://www.youtube.com/watch?v=0n_Ty_72Qds https://www.youtube.com/watch?v=0n_Ty_72Qds
- SigmundA 3y ago
  CEO of Brawndo: "The stock has dropped to zero and the computer did that auto-layoff thing to everybody!"
gorjusborg 3y ago
There needs to be a legal respponsibility of users of AI. AI isn't responsible, those acting with the aid of AI are.
zzzcsgo 3y ago
It's call UnitedHealthcare
- zzzcsgo 3y ago
  I guess the stupid HN title rule limited the characters
rchaud 3y ago
> Though few patients appeal coverage denials generally, when UnitedHealth members appeal denials based on nH Predict estimates—through internal appeals processes or through the federal Administrative Law Judge proceedings—over 90 percent of the denials are reversed, the lawsuit claims. That's the power of system defaults. Make "No" the default response, regardless of the validity of the request, and there's a good chance patients that are trying to manage the logistics of appointments, prescriptions and treatments will simply give up. AI will be used to hold up these existing practices; when glittering visions of lowered healthcare costs are trotted out by the op-ed writers and techno-optimists, this is really how that sausage is being made.
The-Old-Hacker 3y ago
66.5% of US bankruptcies are caused directly by medical expenses. https://www.retireguide.com/retirement-planning/risks/medical-bankruptcy-statistics/ https://www.retireguide.com/retirement-planning/risks/medica...

163 points by kurthr 3y ago

deleted 3y ago

[deleted]

jghn 3y ago

So glad to be a United customer!

kelseyfrog 3y ago

What's an acceptable error rate for such a model? What's the human baseline? I ask these questions[1][2] for nearly every model I'm requested to develop. It usually throws off PMs who aren't used to any acceptable errors. It's always fun to get a request that's several orders of magnitude better than human performance. Sometimes it's actually possible to deliver, but often it shifts the conversation into feasibility and expectations. Anyone developing a model like this will have errors and definitionally makes systemic. Regardless of performance it's important to address that. 1. In this order 2. Sometimes I study human baselines but ask before revealing because it prevents anchoring.

tekacs 3y ago
Absolutely -- an academic environment I was in used to have a lot of government callers come by, asking for predictive power to solve any number of humanitarian crises, terrorism, crime and so on. We collectively used to ask them every time, 1. What data they had to work with. 2. What was the human baseline that they were currently executing with. 3. What performance would make a difference to this process. ... and as you say, the answers we would get would tend to sound something like. 1. Almost nothing. 2. Unclear, un-measured. 3. Something well beyond incredible, well-intentioned humans with incredible data. In that case the lesson was often to simply not try, but it does sometimes feel like other areas of government like E2E-breaking legislation, in that well-intentioned folks will try and try and try again and eventually get someone to fund and execute on the project, feasibility notwithstanding. In this instance with United... the data that we're getting via Ars and this case is so unclear and one-sided that it's hard to know how well things actually work... it'd be fun to consign them (for now) to "don't even try" if we had the info to do so, but it feels like it's a matter of time either way and so... I think the ideal with most of these cases is to eventually ship something that's actually high quality.
raverbashing 3y ago
> What's an acceptable error rate for such a model? What's the human baseline? Funny, I though it was socialized medicine that had "death panels" and such /s
- kelnos 3y ago
  If anything, the "death panels" on private insurers (whatever form they might actually take) are likely much worse than a government-run "death panel".
  - willcipriano 3y ago
    I haven't heard the death panel canard brought out since Canada started it's death panels. It's like the classic idea, but instead of letting you die from lack of medical care they drive you into depression prior to killing you. > Are Canadians being driven to assisted suicide by poverty or healthcare crisis? https://www.theguardian.com/world/2022/may/11/canada-cases-right-to-die-laws https://www.theguardian.com/world/2022/may/11/canada-cases-r... > Why is Canada euthanising the poor? https://www.spectator.co.uk/article/why-is-canada-euthanising-the-poor-/ https://www.spectator.co.uk/article/why-is-canada-euthanisin...
kelnos 3y ago
> What's an acceptable error rate for such a model? Ideally, 0%. We're talking about people's lives and health here. But, right, realistically: > What's the human baseline? Good question, but I would hope it's much better than a 90% error rate. And if it's not, any sane company in any sane industry would figure out why things are that bad, and fix it. Of course, insurance companies are the scum of the earth, and have a huge incentive to get things wrong in this way.
- kelseyfrog 3y ago
  Why is there a different standard between the model and a human? From the perspective of the business, whether a model or human causes the error is immaterial. > Any sane company in any sane industry would figure out why things are that bad, and fix it. The function of a thing is its consequences. So we have to conclude that the reason the model had a 90% error rate was to have a 90% error rate. It's aligned with business incentives to be so.

tekacs 3y ago

> Though few patients appeal coverage denials generally, when UnitedHealth members appeal denials based on nH Predict estimates—through internal appeals processes or through the federal Administrative Law Judge proceedings—over 90 percent of the denials are reversed, the lawsuit claims. This makes it obvious that the algorithm is wrongly denying coverage, it argues. The '90% error rate' in the title sounded like a pretty serious twisting of some numbers so I scanned the article for where it's quoted from ^, which... makes a mockery of the title. In case it's not obvious and to save parsing that, 'few people appeal' (which makes sense given the internal appeals process or Administrative Law Judge proceedings). Then, /of those cases/ 90% are overturned. - If 50 / 10,000 cases and 45 were overturned or 500 / 100,000 cases appealed and 450 were overturned... would all meet this '90% error' mark. - Similarly, 'overturned != error', because it may be as simple as a policy on United's side to not fight super hard and allow overturns (for any number of reasons, including just saving costs on the proceedings, it needn't be altruistic). Did Ars used to be better than this, or were they always like this on some issues? The /business/ side of medical insurance isn't really a topic that I know enough to weigh in on, but hopefully people can read the above sentence and parse what it's actually saying for themselves before reading the article. Irrespective of how I/we feel about the outcomes, this headline is awful.

michaelmrose 3y ago
Of the only claims that have been adjudicated 90% were found to have been erroneous. Without analysis there is no particular reason to believe that other denials were more justified. Few doctors on average make nonsensical requests for patient care while insurers issue spurious denials as a matter of course. You dont need to know the total number of denials instead you need a neutral party to analyze a random sample of denials. We could do this as matter of course and tie failing an adequate benchmark to ruinous penalties to company and CEO.
- hayksaakian 3y ago
  I think the OP is suggesting that hypothetically speaking, people would only go through the hassle of appealing if they were pretty sure they would win to begin with

michaeljx 3y ago

What kind of an educated scientific person, working in health insurance and dealing with sick people, develops anything that makes this possible?

manicennui 3y ago
I worked on healthcare software for over a decade. The healthcare industry attracts the lowest quality people generally. Software engineers aren't giving up working at FAANG to go work in healthcare. A significant portion of UHG's revenue comes from billing and other healthcare software. Most of it is under their Optum brand.
s5300 3y ago
[dead]
lispisok 3y ago
Regular people will do horrible things to keep their paycheck coming. It's an underrated phenomena of humanity.
- irrational 3y ago
  [flagged]
  - all2 3y ago
    It's a feature of humanity. We are intrinsically broken and prone to self preservation. No matter the systems we come up with to given our conduct, we always revert to our basic character traits.
- HellDunkel 3y ago
  I strongly disagree with how this is presented. Doing half-assed or even passionate work for greedy insurance companies is not a horrible thing to do. Not thinking of how these companies can be reigned in and acting accordingly is far worse. We need to stop offloading moral responsibility from the people in charge to the regulae folks.

RedCardRef 3y ago

In my previous role, I worked on something similar to this. We built a model to predict which claims(CMS 1500) would be denied by the carrier(Insurance Provider) before they were submitted and allowing the hospital staff to adjust the Diagnosis/Procedure codes and validate the combination. The client had 20% denials, our model brought that denial rate down to 10%. We wanted to compare the performance of our model with commercial third party offerings and we enrolled into an early beta for a similar offering from ChangeHealthcare one of the big players in the healthcare industry, their model was wildly off(<50% accuracy), we thought we were missing something and went back and forth with their team, turns out they didnt know what they were doing, they just slapped AI on the API and tried to see if anyone would actually use it. That model never saw the light of day. After studying the data model and seeing the number of variables involved I would be skeptical of prediction models retaled to insurance in the healthcare industry.

jpambrun 3y ago
Now Change is owned by United Healthcare, so maybe the model you mentioned did live on..
- catchnear4321 3y ago
  you either die the benevolent language model, or live long enough to become the… no, don’t like that. do not like that.

alberth 3y ago

Was denying 90% not the desired outcome? For an insurance company, this sounds like the ideal model.

lusus_naturae 3y ago
Stupid question—if an insurance provider keeps denying claims which could keep elderly patients healthy, and those patients then die, isn't the insurance provider just cutting off its own source of income? But I agree that it's about incentives, and insurance carriers are incorrectly incentivized for the service they provide vs. what they're expected to provide.
- alberth 3y ago
  The old adage goes: “those who don’t use insurance, pay too much. And those who use insurance, pay too little”. It becomes a math problem for insurance companies to keep that equation positive. And since claims far exceed the premiums any single member pays, the easiest way to stay positive is to deny claims. Even if that sacrifices losing a few premiums.
- bigfudge 3y ago
  >die, isn't the insurance provider just cutting off its own source of income? No because that person will have paid right up the the point they become expensive. Most people incur the vast majority of their health costs in the final year or even months of life. This is why insurance based medicine will always be problematic - there’s just too much incentive to cheap out on people when they are weakest and most powerless.
- heavyset_go 3y ago
  Not really, the highest medical costs are incurred in old age. Insurance companies would have to make less costly payouts if they didn't have to cover older people.
- gorjusborg 3y ago
  Another angle is premiums paid vs claims awarded. If an elderly patient has paid in 50k over a lifetime and is making a 100k claim, now is an okay time to lose that customer.

ahoka 3y ago

Less known for it, but EU GDPR allows for asking for a non automated decision.

causi 3y ago

We need regulations about what computers should and should not be allowed to do. Computers should not be allowed to make healthcare decisions. Computers should not be allowed to make legal threats or actions.

heavyset_go 3y ago

Bureaucracies and authorities love black box models because they can always point to them, shirk their own responsibility for decision making and say they're only following orders and that computers aren't biased. It's just a coincidence that the model they're paying for keeps making decisions that are beneficial to their own interests and reflect their own biases.

michaelteter 3y ago

I predict this that AI will be (intentionally) used as a scapegoat for plausible deniability in some industries. "We just followed our AI model." "We thought it was doing the job." "It's too complex to understand." Obviously once you know or believe that your AI model is wrong, and you are doing a disservice to your clients/patients... and especially once you begin discussing this internally, then you are aware of the problem and should not (get caught) sweeping it under the rug.

iforgotpassword 3y ago
You don't need AI for this, software is already treated as something God given by people. Government somewhere wrongly denying foodstamps to a huge amount of people? Software error, nothing anyone could possibly be responsible for. Getting hacked and all your customers data stolen (because you don't follow basic security best practices)? Well that just happens, just like a taiphoon or earthquake, nothing anyone could've done about this. But bonus points for demanding the government do something about those hacker gangs.
blitzar 3y ago
"Computer says no" is a catchphrase first used in the British sketch comedy television programme Little Britain in 2004. https://www.youtube.com/watch?v=0n_Ty_72Qds https://www.youtube.com/watch?v=0n_Ty_72Qds
SigmundA 3y ago
CEO of Brawndo: "The stock has dropped to zero and the computer did that auto-layoff thing to everybody!"

gorjusborg 3y ago

There needs to be a legal respponsibility of users of AI. AI isn't responsible, those acting with the aid of AI are.

zzzcsgo 3y ago

It's call UnitedHealthcare

zzzcsgo 3y ago
I guess the stupid HN title rule limited the characters

rchaud 3y ago

The-Old-Hacker 3y ago

66.5% of US bankruptcies are caused directly by medical expenses. https://www.retireguide.com/retirement-planning/risks/medical-bankruptcy-statistics/ https://www.retireguide.com/retirement-planning/risks/medica...