6 ms·

AI assists clinicians in responding to patient messages at Stanford Medicine

67 points by namanyayg 3y ago

lumb63 3y ago
Fantastic. An additional way by which someone looking for help or support from a human can be denied that help and have their needs offloaded to a technology that probably can’t help them. Here’s how I see this playing out: practices start to use LLMs to answer patient queries to save money. People realize their queries aren’t actually being answered fully (or at all?) and stop contacting their doctors at all. Practices save money and patients get worse outcomes. Exactly as we’ve seen in customer service over the years - talking to a human is nearly impossible. An alternate possibility is the LLM gives such bad advice that it causes irreparable harm to a patient and nobody can be held liable because the doctor didn’t issue the advice and neither did the company which produced the LLM. The universe where an LLM used in this situation does anything positive for the patient is vanishingly small. But I’m sure providers and practices will love it, since it effectively will allow them to not do part of their job and increase margins respectively.
- sandspar 3y ago
  Why is there always this kneejerk "bad!" reaction to AI? You don't think there's any conceivable way that this may actually help people? You don't think that Stanford Medicine may know a bit more about the problem space than you do?
  - SoftTalker 3y ago
    I think it's conceivable but if the goal was to help people it would already have been done: address the systemic roadblocks that prevent more people from studying medicine, increase the supply of doctors and clinicians so that people don't have to wait months for a routine appointment.
    - jdlshore 3y ago
      That's a false dilemma. We can do both. Some people are addressing roadblocks (or attempting to; politics are hard), and some people are working on AI-assisted support. I'm as skeptical of GenAI as they come, but let's not pretend that the people working on GenAI customer support are the same people who should be working with the AMA to address the artificial constraints in medical education.
      - bluefirebrand 3y ago
        > That's a false dilemma. We can do both. The thing I notice about every argument where I see "we can do both", is that in reality we never actually do both That's why people get so adamant about choosing their way instead of the other way. There's always two options, we could do both, but we never, ever seem to
        jdlshore 3y ago
        Who’s this “we,” kemo sabe? Let’s summarize the situation here. A group of people have chosen to work on GenAI customer support. You’re criticizing them for not working on a difficult political problem instead. Meanwhile, what are you doing? Which of the political activist groups working on the problem you care about so much have you joined? How have you offered your support? If this issue is so important to you that you think “we” should be devoting our full attention to it, surely you’re out front leading the charge, right?
  - constantcrying 3y ago
    There is by far enough enthusiasm for AI already. >You don't think there's any conceivable way that this may actually help people? I don't. I believe that exactly as OP described people will notice they are talking to some AI and just give up, as they correctly understand that the AI does not have the abilities and understanding that might actually help them.
  - gpm 3y ago
    Because the vast majority of commercial deployments of "AI" (llms) are extremely unhelpful chat bots that just waste customers time to try and avoid having to have a human talk to them.
    - choilive 3y ago
      And the vast majority of commercial deployments of real humans to solve the same problem are extremely unhelpful that just waste customers time. Whats your point?
    - wruza 3y ago
      Decent LLMs are < 1 year old. How much commercial deployment they’ve seen?
      - unleaded 3y ago
        Too much
        wruza 3y ago
        Too much where? Let's ignore AI "startups" with no business idea beyond reselling system prompts for GPT-4 to end users. Does your bank have AI like last summer? A clinic? Any utility service? It is unlikely for most established businesses, cause they cannot move that fast. I just did a quick search on "{bank,hospital,...} integrated gpt-4 for client support" and similar queries and there's various "will be", "may use", but nothing specific. I believe ggp is mixing regular chat bots with integration of SotA in their statement.
        gpm 3y ago
        Off the top of my head: Air Canada, shopify, ikea, car dealerships, ... It's common enough I'm sure I've seen more than that, those are just the ones I remember because they've either personally annoyed me or been in the news. Health things around here have legal restrictions that definitely prevent them from using GPT, thankfully.
  - lostlogin 3y ago
    > Why is there always this kneejerk "bad!" reaction to AI? I don’t interpret the OP as slamming AI. It’s criticism of the way in which companies find ways to save a few dollars.
  - olliej 3y ago
    Because “AI” or LLMs have no understanding of what they’re producing, and every single attempt to use them in an environment where basic factual correctness is involved has confirmed this? Or the fact that the entire sales pitch is “this will be cheaper than actual people who know what they’re doing”, maybe a side claim of “outperforms people on our training set” coupled with “we accept no liability for inaccurate advice or poor outcomes”. The sole purpose of programs like these is to save money. It has nothing to do with helping anyone. The goal is to sound like your system is providing the service people paid for, without actually spending the money to provide that service. I will believe a company will is doing the right thing when the developers personally accept the criminal and legal liability that an actual doctor or nurse providing the service is subject to. But then to add insult to injury this just further increases the wait time for people by forcing them to deal with the “AI” before they get through to the further reduced pool of actual specialists and correspondingly increased wait times there as well. This is not rocket science to understand, and literally nothing I’ve said here is new - not the clear prioritization of cost cutting, the overselling of “AI”, and the generally unsafe use of randomized statistical models to provide factual information. My question is why you think that this time is different vs literally every other example?
  - tbalsam 3y ago
    This is because this is what the tendency has been like in practice for many years. While the possibilities are good, living in the world of what-if for ML is very similar to the world of what-if for blockchain. It is very promising but unfortunately the past trend with reality at both a surface and deeper dynamics level does not seem to agree with it being a solid motif long-term. This is why people are pessimistic about ML/AI in people-facing positions. It has happened with automation in the past, it is _currently happening_ with ML/AI systems in customer-facing applications, and the momentum of the field is headed in that direction. To me, it would be very silly to assume that the field would make a sudden 180 from decades of momentum in a rather established direction, with no established reason other than the potential for it to be good. This is why people tend to generally be upset about it, is my understanding. It's a good tool, and it is oftentimes not used well.
  - rsynnott 3y ago
    Given the long, unfortunate history of computerisation of medicine, I don’t think there should be a presumption that they necessarily know what they’re doing. If this is a disaster, it won’t be the first, and some previous medical-technical disasters have involved human factors; the operator either becoming too trusting of the machine, or too casual with its use.
- jncfhnb 3y ago
  Is it just me or has customer service never been easier, with real humans easily accessible for most major corporations
  - pbourke 3y ago
    Real humans who are unable to resolve your issue are quite accessible indeed.
    - jncfhnb 3y ago
      Like… who? I find that large banks, telecom, and what not have largely shifted their call centers to either America or the Philippines and you now get much more proficient customer service than, say, 10 years ago when everything was going to India. Chat based humans are also much more efficient imo.
- justrealist 3y ago
  I don't think you understand how stupid and googleable most customer support questions are. Doctors will have more time for real questions if they aren't answering idiotic questions. You are living in a bubble.
  - SoftTalker 3y ago
    Doctors would also have more time for real questions if they told fat people to exercise and quit eating so many pop-tarts instead of giving them shots with a host of risky side-effects that then have to be managed.
    - justrealist 3y ago
      Are you really trying to blame obesity on doctors? Impressive.
      - deleted 3y ago
        [deleted]
      - lupusreal 3y ago
        [flagged]
        ceejayoz 3y ago
        No, they’re ignoring the medical fact that “tell people to lose weight” has a basically zero percent success rate. The only intervention known to work reliably was bariatric surgery until stuff like Ozempic showed up recently.
        lupusreal 3y ago
        Cutting losses to spend time with other patients isn't ignoring that little medical fact. It's an acknowledgement of it; all the effort spent treating the symptoms of obesity is basically wasted, squandering the resources (time particularly) of the medical system.
- userbinator 3y ago
  An alternate possibility is the LLM gives such bad advice that it causes irreparable harm to a patient and nobody can be held liable because the doctor didn’t issue the advice and neither did the company which produced the LLM. I think someone will be liable; this has already been tested in at least one court with something less life-critical: https://news.ycombinator.com/item?id=39378235 https://news.ycombinator.com/item?id=39378235
- exe34 3y ago
  The doctor who signs off will definitely be liable. Unfortunately we live in a society where somebody both has to die and have litigious surviving relatives to make sure the problem becomes actionable.
ceejayoz 3y ago
“Just review the AI draft” is likely gonna kill some people by priming clinicians to agree with the result. https://www.npr.org/sections/health-shots/2013/02/11/171409656/why-even-radiologists-can-miss-a-gorilla-hiding-in-plain-sight https://www.npr.org/sections/health-shots/2013/02/11/1714096... > He took a picture of a man in a gorilla suit shaking his fist, and he superimposed that image on a series of slides that radiologists typically look at when they're searching for cancer. He then asked a bunch of radiologists to review the slides of lungs for cancerous nodules. He wanted to see if they would notice a gorilla the size of a matchbook glaring angrily at them from inside the slide. > But they didn't: 83 percent of the radiologists missed it, Drew says.
- wccrawford 3y ago
  Based on the code that CoPilot has produced for me, I can 100% see that happening, and rather a lot. So far, the code it produces looks really good at first glance, but once I really dig into what it's doing, I find numerous problems. I have to really think about that code to overcome the things it does wrong. I think many doctors are used to doing that already, since they have to correct their interns/assistants/etc when they come up with things that sound good, but ultimately aren't. But it's definitely something that they'll fail at sometimes.
  - stouset 3y ago
    This is what I really worry about in programming contexts. The norm is already for a distressing proportion of engineers to barely (if even that) understand code they’re working on, even when they’re the only author. In that situation it’s extremely easy to just blindly trust an “expert” third party, particularly one where you can’t ask it questions about how and why.
    - userbinator 3y ago
      They don't even deserve to be called "engineers". They're just guess-and-test amateurs.
- userbinator 3y ago
nicklecompte 3y ago
I haven't finished reading the paper closely, but it is disturbing that this is the only real reference to accuracy and relevance I could find: > Barriers to adoption include draft message voice and/or tone, content relevance (8 positive, 1 neutral, and 9 negative), and accuracy (4 positive and 5 negative). That's it! Outside of that it looks like almost all the metrics are about how this makes life easier for doctors. Maybe there is more in the supplement, I have not read closely. Look: I am not a policymaker or a physician. It's quite plausible that a few LLM hallucinations is an acceptable price to mitigate the serious threat of physician overwork and sleep deprivation. And GPT-4 is pretty good these days, so maybe the accuracy issues were minor and had little impact on patient care. What's not acceptable is motivated reasoning about productivity leading people to treat the accuracy and reliability of software as an afterthought.
- potatoman22 3y ago
  Those are just free-text comments from the ~80 physicians who used this tool. Tables 2-4 show the researchers measured various characteristics like message read/write time, utilization rate, work load, burnout, utility, and message quality. It's also worth noting that, from the abstract, the study's objective isn't to study the LLM's accuracy. This is a study of the effectiveness of this drafting system's implementation in the hospital. I'm not saying the accuracy isn't an important component of the system's effectiveness, but it's not the question they're answering.
  - nicklecompte 3y ago
    My point is that "accuracy is not the question they're answering" is a fatal flaw that makes this research pointless. Say I released a linear algebra library and extolled its performance benefits + ease of development, then offhandedly mentioned that "it has some concerns with accuracy" without giving further details. You wouldn't say "ah, he focused on performance rather than accuracy." You'd say "is this a scam?" It is never acceptable for healthcare software researchers and practitioners to ignore accuracy. LLMs don't change that. The only thing that sort of cynical laziness is good for is justifying bad decision-making by doctors and hospital administrators.
jvans 3y ago
Notice how all the benefits they mention are for physicians and not for the patients. That's a pretty backwards approach to healthcare
- Calavar 3y ago
  It's a benefit for health system C-suit execs because it makes physicians "more efficient," and it's ultimately the health system C-suite who will be signing off on the contracts for five years of AI messenger service or whatever, not the physicians.
- ipv6ipv4 3y ago
  Stanford medicine in a nutshell. Stanford medical has spread like a cancer around me, and all the clinics that were acquired immediately became awful.
  - raincom 3y ago
    Yes, that's how 'reputation laundering' works: just acquire all clinics and fool people with the tag "Stanford".
- SpaceLawnmower 3y ago
  Physician experience is important for patient experience. Doctors are by and large overworked, and respondih to too many patient messages are a major contributor to burnout.
  - jvans 3y ago
    Approaching the problem of doctor burnout by spamming patients with half correct responses to important medical questions is not a solution
jmount 3y ago
Likely paying some tall dollars to deal with cheap first line support. Probably a "direct to human" fee available soon.
throwaway918274 3y ago
this is horrifying
impish9208 3y ago
I bet they’ll still charge the regular “Physician Fee” though

67 points by namanyayg 3y ago

Fantastic. An additional way by which someone looking for help or support from a human can be denied that help and have their needs offloaded to a technology that probably can’t help them. Here’s how I see this playing out: practices start to use LLMs to answer patient queries to save money. People realize their queries aren’t actually being answered fully (or at all?) and stop contacting their doctors at all. Practices save money and patients get worse outcomes. Exactly as we’ve seen in customer service over the years - talking to a human is nearly impossible. An alternate possibility is the LLM gives such bad advice that it causes irreparable harm to a patient and nobody can be held liable because the doctor didn’t issue the advice and neither did the company which produced the LLM. The universe where an LLM used in this situation does anything positive for the patient is vanishingly small. But I’m sure providers and practices will love it, since it effectively will allow them to not do part of their job and increase margins respectively.

sandspar 3y ago
Why is there always this kneejerk "bad!" reaction to AI? You don't think there's any conceivable way that this may actually help people? You don't think that Stanford Medicine may know a bit more about the problem space than you do?
- SoftTalker 3y ago
  I think it's conceivable but if the goal was to help people it would already have been done: address the systemic roadblocks that prevent more people from studying medicine, increase the supply of doctors and clinicians so that people don't have to wait months for a routine appointment.
  - jdlshore 3y ago
    That's a false dilemma. We can do both. Some people are addressing roadblocks (or attempting to; politics are hard), and some people are working on AI-assisted support. I'm as skeptical of GenAI as they come, but let's not pretend that the people working on GenAI customer support are the same people who should be working with the AMA to address the artificial constraints in medical education.
    - bluefirebrand 3y ago
      > That's a false dilemma. We can do both. The thing I notice about every argument where I see "we can do both", is that in reality we never actually do both That's why people get so adamant about choosing their way instead of the other way. There's always two options, we could do both, but we never, ever seem to
      - jdlshore 3y ago
        Who’s this “we,” kemo sabe? Let’s summarize the situation here. A group of people have chosen to work on GenAI customer support. You’re criticizing them for not working on a difficult political problem instead. Meanwhile, what are you doing? Which of the political activist groups working on the problem you care about so much have you joined? How have you offered your support? If this issue is so important to you that you think “we” should be devoting our full attention to it, surely you’re out front leading the charge, right?
- constantcrying 3y ago
  There is by far enough enthusiasm for AI already. >You don't think there's any conceivable way that this may actually help people? I don't. I believe that exactly as OP described people will notice they are talking to some AI and just give up, as they correctly understand that the AI does not have the abilities and understanding that might actually help them.
- gpm 3y ago
  Because the vast majority of commercial deployments of "AI" (llms) are extremely unhelpful chat bots that just waste customers time to try and avoid having to have a human talk to them.
  - choilive 3y ago
    And the vast majority of commercial deployments of real humans to solve the same problem are extremely unhelpful that just waste customers time. Whats your point?
  - wruza 3y ago
    Decent LLMs are < 1 year old. How much commercial deployment they’ve seen?
    - unleaded 3y ago
      Too much
      - wruza 3y ago
        Too much where? Let's ignore AI "startups" with no business idea beyond reselling system prompts for GPT-4 to end users. Does your bank have AI like last summer? A clinic? Any utility service? It is unlikely for most established businesses, cause they cannot move that fast. I just did a quick search on "{bank,hospital,...} integrated gpt-4 for client support" and similar queries and there's various "will be", "may use", but nothing specific. I believe ggp is mixing regular chat bots with integration of SotA in their statement.
        gpm 3y ago
        Off the top of my head: Air Canada, shopify, ikea, car dealerships, ... It's common enough I'm sure I've seen more than that, those are just the ones I remember because they've either personally annoyed me or been in the news. Health things around here have legal restrictions that definitely prevent them from using GPT, thankfully.
- lostlogin 3y ago
  > Why is there always this kneejerk "bad!" reaction to AI? I don’t interpret the OP as slamming AI. It’s criticism of the way in which companies find ways to save a few dollars.
- olliej 3y ago
  Because “AI” or LLMs have no understanding of what they’re producing, and every single attempt to use them in an environment where basic factual correctness is involved has confirmed this? Or the fact that the entire sales pitch is “this will be cheaper than actual people who know what they’re doing”, maybe a side claim of “outperforms people on our training set” coupled with “we accept no liability for inaccurate advice or poor outcomes”. The sole purpose of programs like these is to save money. It has nothing to do with helping anyone. The goal is to sound like your system is providing the service people paid for, without actually spending the money to provide that service. I will believe a company will is doing the right thing when the developers personally accept the criminal and legal liability that an actual doctor or nurse providing the service is subject to. But then to add insult to injury this just further increases the wait time for people by forcing them to deal with the “AI” before they get through to the further reduced pool of actual specialists and correspondingly increased wait times there as well. This is not rocket science to understand, and literally nothing I’ve said here is new - not the clear prioritization of cost cutting, the overselling of “AI”, and the generally unsafe use of randomized statistical models to provide factual information. My question is why you think that this time is different vs literally every other example?
- tbalsam 3y ago
  This is because this is what the tendency has been like in practice for many years. While the possibilities are good, living in the world of what-if for ML is very similar to the world of what-if for blockchain. It is very promising but unfortunately the past trend with reality at both a surface and deeper dynamics level does not seem to agree with it being a solid motif long-term. This is why people are pessimistic about ML/AI in people-facing positions. It has happened with automation in the past, it is _currently happening_ with ML/AI systems in customer-facing applications, and the momentum of the field is headed in that direction. To me, it would be very silly to assume that the field would make a sudden 180 from decades of momentum in a rather established direction, with no established reason other than the potential for it to be good. This is why people tend to generally be upset about it, is my understanding. It's a good tool, and it is oftentimes not used well.
- rsynnott 3y ago
  Given the long, unfortunate history of computerisation of medicine, I don’t think there should be a presumption that they necessarily know what they’re doing. If this is a disaster, it won’t be the first, and some previous medical-technical disasters have involved human factors; the operator either becoming too trusting of the machine, or too casual with its use.
jncfhnb 3y ago
Is it just me or has customer service never been easier, with real humans easily accessible for most major corporations
- pbourke 3y ago
  Real humans who are unable to resolve your issue are quite accessible indeed.
  - jncfhnb 3y ago
    Like… who? I find that large banks, telecom, and what not have largely shifted their call centers to either America or the Philippines and you now get much more proficient customer service than, say, 10 years ago when everything was going to India. Chat based humans are also much more efficient imo.
justrealist 3y ago
I don't think you understand how stupid and googleable most customer support questions are. Doctors will have more time for real questions if they aren't answering idiotic questions. You are living in a bubble.
- SoftTalker 3y ago
  Doctors would also have more time for real questions if they told fat people to exercise and quit eating so many pop-tarts instead of giving them shots with a host of risky side-effects that then have to be managed.
  - justrealist 3y ago
    Are you really trying to blame obesity on doctors? Impressive.
    - deleted 3y ago
      [deleted]
    - lupusreal 3y ago
      [flagged]
      - ceejayoz 3y ago
        No, they’re ignoring the medical fact that “tell people to lose weight” has a basically zero percent success rate. The only intervention known to work reliably was bariatric surgery until stuff like Ozempic showed up recently.
        lupusreal 3y ago
        Cutting losses to spend time with other patients isn't ignoring that little medical fact. It's an acknowledgement of it; all the effort spent treating the symptoms of obesity is basically wasted, squandering the resources (time particularly) of the medical system.
userbinator 3y ago
An alternate possibility is the LLM gives such bad advice that it causes irreparable harm to a patient and nobody can be held liable because the doctor didn’t issue the advice and neither did the company which produced the LLM. I think someone will be liable; this has already been tested in at least one court with something less life-critical: https://news.ycombinator.com/item?id=39378235 https://news.ycombinator.com/item?id=39378235
exe34 3y ago
The doctor who signs off will definitely be liable. Unfortunately we live in a society where somebody both has to die and have litigious surviving relatives to make sure the problem becomes actionable.

ceejayoz 3y ago

“Just review the AI draft” is likely gonna kill some people by priming clinicians to agree with the result. https://www.npr.org/sections/health-shots/2013/02/11/171409656/why-even-radiologists-can-miss-a-gorilla-hiding-in-plain-sight https://www.npr.org/sections/health-shots/2013/02/11/1714096... > He took a picture of a man in a gorilla suit shaking his fist, and he superimposed that image on a series of slides that radiologists typically look at when they're searching for cancer. He then asked a bunch of radiologists to review the slides of lungs for cancerous nodules. He wanted to see if they would notice a gorilla the size of a matchbook glaring angrily at them from inside the slide. > But they didn't: 83 percent of the radiologists missed it, Drew says.

wccrawford 3y ago
Based on the code that CoPilot has produced for me, I can 100% see that happening, and rather a lot. So far, the code it produces looks really good at first glance, but once I really dig into what it's doing, I find numerous problems. I have to really think about that code to overcome the things it does wrong. I think many doctors are used to doing that already, since they have to correct their interns/assistants/etc when they come up with things that sound good, but ultimately aren't. But it's definitely something that they'll fail at sometimes.
- stouset 3y ago
  This is what I really worry about in programming contexts. The norm is already for a distressing proportion of engineers to barely (if even that) understand code they’re working on, even when they’re the only author. In that situation it’s extremely easy to just blindly trust an “expert” third party, particularly one where you can’t ask it questions about how and why.
  - userbinator 3y ago
    They don't even deserve to be called "engineers". They're just guess-and-test amateurs.
userbinator 3y ago

nicklecompte 3y ago

I haven't finished reading the paper closely, but it is disturbing that this is the only real reference to accuracy and relevance I could find: > Barriers to adoption include draft message voice and/or tone, content relevance (8 positive, 1 neutral, and 9 negative), and accuracy (4 positive and 5 negative). That's it! Outside of that it looks like almost all the metrics are about how this makes life easier for doctors. Maybe there is more in the supplement, I have not read closely. Look: I am not a policymaker or a physician. It's quite plausible that a few LLM hallucinations is an acceptable price to mitigate the serious threat of physician overwork and sleep deprivation. And GPT-4 is pretty good these days, so maybe the accuracy issues were minor and had little impact on patient care. What's not acceptable is motivated reasoning about productivity leading people to treat the accuracy and reliability of software as an afterthought.

potatoman22 3y ago
Those are just free-text comments from the ~80 physicians who used this tool. Tables 2-4 show the researchers measured various characteristics like message read/write time, utilization rate, work load, burnout, utility, and message quality. It's also worth noting that, from the abstract, the study's objective isn't to study the LLM's accuracy. This is a study of the effectiveness of this drafting system's implementation in the hospital. I'm not saying the accuracy isn't an important component of the system's effectiveness, but it's not the question they're answering.
- nicklecompte 3y ago
  My point is that "accuracy is not the question they're answering" is a fatal flaw that makes this research pointless. Say I released a linear algebra library and extolled its performance benefits + ease of development, then offhandedly mentioned that "it has some concerns with accuracy" without giving further details. You wouldn't say "ah, he focused on performance rather than accuracy." You'd say "is this a scam?" It is never acceptable for healthcare software researchers and practitioners to ignore accuracy. LLMs don't change that. The only thing that sort of cynical laziness is good for is justifying bad decision-making by doctors and hospital administrators.

jvans 3y ago

Notice how all the benefits they mention are for physicians and not for the patients. That's a pretty backwards approach to healthcare

Calavar 3y ago
It's a benefit for health system C-suit execs because it makes physicians "more efficient," and it's ultimately the health system C-suite who will be signing off on the contracts for five years of AI messenger service or whatever, not the physicians.
ipv6ipv4 3y ago
Stanford medicine in a nutshell. Stanford medical has spread like a cancer around me, and all the clinics that were acquired immediately became awful.
- raincom 3y ago
  Yes, that's how 'reputation laundering' works: just acquire all clinics and fool people with the tag "Stanford".
SpaceLawnmower 3y ago
Physician experience is important for patient experience. Doctors are by and large overworked, and respondih to too many patient messages are a major contributor to burnout.
- jvans 3y ago
  Approaching the problem of doctor burnout by spamming patients with half correct responses to important medical questions is not a solution

jmount 3y ago

Likely paying some tall dollars to deal with cheap first line support. Probably a "direct to human" fee available soon.

throwaway918274 3y ago

this is horrifying

impish9208 3y ago

I bet they’ll still charge the regular “Physician Fee” though