9 ms·

A neurology ICU nurse on AI in hospitals

110 points by redrove 2y ago

Festro 2y ago
"We didn’t call it AI at first." Because the first things described in the article are not AI. They are ML at most. Then the article discusses a patient needs scoring method, moving from their own Low/Medium/High model to a scoring method on an unbound linear scale. The author appears to struggle with being able to tell if 240 is high or not. They don't state if they ever had training or saw documentation for the scoring method. Seems odd to not have these things but that if they did the scores would be a lot easier to interpret. Then they finally get to AI, and it's a pilot scheme for writing patient notes. That's all. If it sucks and hallucinates information it's not going to go live anywhere. No matter how many tech bros try to force it through. If the feedback model for the pilot is bad then the author should take issue with that. It's important that such tests give testers an adequate method to flag issues. Very much an AI = bad article. AI converging with medical technology is a really dangerous space, for obvious reasons. An article like this does make me worry it's being rushed through, but not because of the author's objections, instead because of their ignorance of what is and what isn't AI, and then on the other side of the apparent lack of consultation being offered by the technology providers even during testing stages.
- ta988 2y ago
  The industry calls AI even the dumbest linear system, I don't think it is right to blame non industry people if they don't use the right words after that.
- ToucanLoucan 2y ago
  > AI converging with medical technology is a really dangerous space, for obvious reasons. An article like this does make me worry it's being rushed through, but not because of the author's objections, instead because of their ignorance of what is and what isn't AI I mean, worth pointing out that OpenAI has been shoving LLMs into people's faces for going on about a year and half now at global scale and calling it AI, to a degree that now we have to call AI AGI, and LLM's get called AI, even though there is nothing intelligent about them whatsoever. Just saying when the marketing for a tech is grinding the edge of being misinformation itself, it's pretty normal for the end users to end up pretty goddamn confused by the end.
  - bryanlarsen 2y ago
    For 60 years the Turing Test was the standard benchmark for artificial intelligence. Now machines can pass the test. The only goalpost moving I can see is the moving done by the people who insist that LLM's aren't AI.
  - bitwize 2y ago
    The simple state machines and decision trees that make enemies move and attack in video games, we also call AI. AI is a loose term and always has been. It's like the term "robot": we call any machine a robot which either a) resembles a human or part of one (e.g., a robotic arm), or b) can perform some significant human labor that involves decision making (e.g., a robot car that drives itself). Similarly, AI is anything that makes, or seems to make, judgements that we think of as the exclusive purview of humans. Decision trees used to be thought of as AI, but today they are not (except, again, in the context of video games where they're used to control agents intended to seem alive).
    - ToucanLoucan 2y ago
      > The simple state machines and decision trees that make enemies move and attack in video games, we also call AI. Yes but no game companies have ever marketed their game asserting that the things your shooting are, in fact, self-aware conscious intelligences.
      - bitwize 2y ago
        Tell me you weren't around for the PlayStation 2 marketing blitz without telling me you weren't around for the PlayStation 2 marketing blitz. They actually named the thing's CPU the Emotion Engine.
  - PhasmaFelis 2y ago
    That's been the technical definition in the AI research community for at least 50 years. "AI = machines that think like people" is the sci-fi definition.
- develatio 2y ago
  > No matter how many tech bros try to force it through. Are you sure about that? My wife is a nurse and she has to deal with multiple machines that were put in her unit just because the hospital has a contract with XYZ brand. It doesn't matter at all if these machines are 5x more expensive, x10 slower, x80 less effective, etc... compared with the "other" machines (from other brands). I'm actually terrified the same might happen with this.
- happytoexplain 2y ago
  I think the effort to keep the definition of AI narrow is not useful (and futile, besides). By both common usage and even most formal definitions, it's an umbrella term, and sometimes refers to a specific thing under that umbrella that we already have other words for if we would like to be more specific (ML, deep learning, LLM, GPT, genAI, neural net, etc).
- bryanlarsen 2y ago
  > If it sucks and hallucinates information it's not going to go live anywhere. It hallucinates and it's live in many places. My doctor uses it. AFAICT the hallucination rate is fairly low. It's transcription and summarization which has a lower hallucination rate than when it's asked for an answer. It's massively better than the alternative. Taking the time to write down proper notes is the difference between 4 patients per hour and 6 and a commensurate drop in income. So notes virtually always get short-changed. Very occasionally hallucinated notes are better than notes that are almost always incomplete.
  - fhfjfk 2y ago
    What about increasing the supply of doctors rather than decreasing the time they spend with patients? I question the underlying premise that efficiency needs to increase.
    - daemin 2y ago
      Because the USA is for-profit healthcare industry they need to optimise and increase efficiency. That's the only way to make the numbers go up. Therefore fewer doctors, fewer nurses, fewer administrators (maybe), and more paying patients.
      - anadem 2y ago
        > more paying patients that's a chilling thought; don't give them ideas
        slater 2y ago
        Surprising that you imply they haven't already been capitalizing (ha!) on that idea since the 1970s :D
      - bryanlarsen 2y ago
        It's no better in other countries. Other country's health care systems are typically monopsonies, which means we get crappier results for lower prices. So instead of the doctor choosing to do 6 patients per hour instead of 4, it's the government.
Loughla 2y ago
Healthcare is a massive cost for people, businesses, and governments. >So we basically just become operators of the machines. Driving down the cost of manufacturing because of process standardization and automation brought down the cost of consumer goods, and labor's value. If you don't think this is coming for every single area of business, you're foolish. Driving down labor costs is the golden goose. We've been able to collect some eggs through technology, but AI and the like will be able to cut that goose open and take all the eggs.
- happytoexplain 2y ago
  "don't think will happen" != "don't think is good"
- coliveira 2y ago
  > will be able to cut that goose open and take all the eggs You're completely right! AI will kill the golden goose, that is what this metaphor is all about.
- thefz 2y ago
  > Driving down the cost of manufacturing because of process standardization and automation brought down the cost of consumer goods, and labor's value. And quality as well.
  - Spivak 2y ago
    Not really, the quality we're able to produce at scale is the best it's ever been in the world. The "quality of everything has been in decline" isn't due to advances in manufacturing but the economic factors that are present even without those advances. Rapid inflation over a relatively short period of time has forced everyone to desperately try and maintain close to their pre-inflation prices because workers didn't and aren't going to get the corresponding wage increases. I hope eventually there's a tipping point where the powers that be realize that our economy can't work unless the wealth that naturally accumulates at the top gets fed back into the bottom but the US pretty unanimously voted for the exact opposite so I desperately I hope I'm wrong or that this path can work too.
coliveira 2y ago
Everyone should be terrified. The "promise" of AI is the following: remove any kind of remaining communication between humans, because that is "inefficient", and replace it with an AI that will mediate all human interactions (in business and even in other areas). In a few years, AIs trained by big corps will run the show and humans will be required to interface with them to do anything of value. Similar to what they want to do nowadays with mobile/enterprise systems, but at a much deeper level.
- anthonyskipper 2y ago
  Some of us look forward to that future where you mostly just interact with AI. The one depressing us is not turning running over our goverment to AI. The sooner we can do that the better, you can't trust humans.
  - maxehmookau 2y ago
    > Some of us look forward to that future where you mostly just interact with AI. What is it about that that appeals to you? I'm genuinely curious. A world without human interaction feels like a world I don't want to exist in.
    - cptaj 2y ago
      They expect AI bureaucracy to be more effective than human bureaucracy. I expect this to be entirely true in some cases.
    - andy_ppp 2y ago
      If you are autistic (for example) I'm guessing human interaction can be extremely difficult and very stressful and triggering. Machines are much more amenable and don't have loads of arbitrary unwritten rules the way humans do. Maybe the idea of being entrapped by bureaucracy introduced by machines will be better than the the bureaucracy introduced by humans?
      - add-sub-mul-div 2y ago
Mathnerd314 2y ago
> There’s a proper way to do this. Is there? Seems like people will complain however fast you roll out AI, so you might as well roll it out quickly and get it over with.
oldpersonintx 2y ago
[dead]
mro_name 2y ago
There's this earthquake phrase in past tense: > We felt like we had agency.
paulnpace 2y ago
I think something this article demonstrates is how AI implementation is resulting in building resistance to AI because AI is being forced onto people instead of being demanded by those people. Typically, the people doing the forcing don't understand very well the job the people being forced to adopt AI actually perform.
parasense 2y ago
I did a bunch of research essays into medical uses of AI/ML and I'm not terrified, in fact the single most significant use of these technologies is probably in or around healthcare. One of the most cited uses would be expert analysis of medical imaging, especially breast cancer imaging. There is a lot of context to unpack around breast cancer imaging, or more sucinctly put, controversial drama! The fact is there is a statisticalluy high rate of false positives in breast cancer diagnostics made by human doctors. This reality resulted in a big overall policy shift to have women breast scanned less often, depending on their age, or something like that. Because so many women were victimized with breast surgery that turned out to be false positive or whatever. The old saying to make an omlet one must break a few eges is sometimes used, and that's a terrible euphamism. AI has proven to be better at looking at medical image, and in the case of breast cancer seems to out perform humans. And of course the humans have a monotonous job revewing image after image, and they want to be safe instead of latter being sorry, so of course they have high false possitives. The machines never get tired, they never get biased (this is a bone of contention), and they never stop. Ultimatly a human doctor still has to review the images, and the machines simply inform if the doctor is being too agressive in diagnosis, or possibly missing something. The whole thing gets escellated if there is any disparity. The out come from early studdies is encouraging, but these studies take years, and are very expensive. One of the biggest problems is the technology proficiency of medical staff is low, and so we are now in a situation where software engineers are cross traning to be at the level of a nurse or even doctors in rare cases.
- buffington 2y ago
  One very important part your comment doesn't mention: a real human being has to actually take images for the AI to analyze. The amount of training a radiation technologist (the person who makes you put your body in uncomfortable positions when you break something) is significant. My partner has made a career of it, and the amount of school needed and clinical hours is non-trivial, and harder to do than becoming a nurse from what I understand. They need to know as much about bones as orthopedic surgeons while also knowing how radiation works, as well as how the entire imagining tech stack works, while also having the soft skills needed to guide injured/ill patients to do difficult things (often in the midst of medical trauma). The part where a doctor looks at images is really just a very small part of the entire "product." The radiologists who say "there's a broken arm" are never in the room, never see the patient, never have context. It's something that, frankly, an AI can do much more consistently and accurately at this point.
lekanwang 2y ago
As an investor in healthcare AI companies, I actually completely agree that there's a lot of bad implementations of AI in healthcare settings, and what practitioners call "alarm fatigue" as well as the feeling of loss of agency is a huge thing. I see a lot of healthcare orgs right now roll out some "AI" "solution" in isolation that raises one metric of interest, but fails to measure a bunch of other systemic measures. Two thoughts: 1: I think the industry could take cues from aerospace and the human factors research that's drastically improved safety there -- autopilot and autoland systems in commercial airliners are treated as one part of a holistic system with the pilot and first officer and flight attendants in keeping the plane running smoothly. Too few healthcare AI systems are evaluated holistically. 2: Similarly, if you're going to roll out a system, either there's staff buy-in, or the equilibrium level of some kind of quality/outcomes/compliance measure should increase that justifies staff angst and loss of agency. Not all AI systems are bad. One "AI" company we invested in, Navina, is actually loved by physicians using them, but the team also spent a LOT of time doing UX research and feedback with actual users and the support team is always super responsive.
- hartspear 2y ago
  Agreed, what was obviously missing from this person's experience was an opportunity to guide deployment in ways that work well for their understanding of the workflow. I think companies often fail to explore what actually happens on the ground of their organizations and make assumptions that are not accurate. AI could be a huge help to nurses but it takes conversations and direct insight during development to get it right.
heironimus 2y ago
This is the same technology story told thousands of times a day with nearly every technology. Medical seems to be especially bad at this. Take a very promising technology that could be very useful. Jump on it early without even trying to get buy in and without fully understanding the people that will use it. Then push a poor version of it. Now the nurses hate the tech, not the poor implementation of it. The techies then bypass the nurses because they are difficult, even though they could be their best resource for improvement.
cowmix 2y ago
This article feels “ripped from today’s headlines” for me, as my mother-in-law was recently in the ICU after a fall that caused head trauma. The level of AI-driven automated decision-making is unsettling, especially as it seems to allow large organizations to deflect accountability—“See? The AI made us do it!” I’m not entirely sure what guided her care—or lack thereof—but, as someone who frequently works in healthcare IT, I see these issues raised all the time. On the other hand, having access to my own “AI” was incredibly helpful during her incident. While in the ICU, speaking with her doctors, I used ChatGPT and Claude to become a better advocate for her by asking more informed questions. I could even take pictures of the monitors tracking her vitals, and ChatGPT helped me interpret the readings, which was surprisingly useful. In this “AI-first” world we’re heading into, individuals need their own tools to navigate the asymmetric power dynamic with large organizations. I wonder how long it will be until these public AI models get “tweaked” to limit their effectiveness in helping us question “the man.”
- cowmix 2y ago
  Side note: I tried the same questions with some local LLMs I’m running at home—unfortunately, they’re nowhere near as good or useful. I hope local models improve quickly, so we’re not left depending on the good graces of big LLM(tm).
- wing-_-nuts 2y ago
  The article feels very much like a union rep fighting automation. If AI is provably worse, we should see that come up in the AI making 'bad calls' vs the human team. You would even see affects on health outcomes. One place I'd really like an all seeing eye AI overlord is in nursing home care. I have seen family members lie in filth with clear signs of an infection. I am confident, if we hadn't visited, seen this, and got her out of there she would have died there, years before her time.
  - FireBeyond 2y ago
    Sadly, the one thing I took from my time as an EMT and paramedic was which nursing homes to consider and which to avoid. I filed more than one complaint with the DOH. It's a standing joke that whenever 911 crews respond to a nursing home, the report you'll get from staff will be a bingo game of: - "I just got on shift; I don't know why you were called." - "This is not my usual floor; I'm just covering while someone is on lunch. I don't know why you were called." - [utterly unrealistic set of vitals, in either direction, healthy, lively vitals for someone who is not thriving, or "should be unconscious" vitals for someone lively and spry] - [extended time waiting for patient notes, history, an outdated med list with everything they've taken in their ten years at the facility] And so on. I (generally) don't blame the floor staff (though some things, as you describe, are inexcusable) but management/ownership. The same management/ownership that has the policy to call 911 for anything more involved than a bandaid for some weird idea of managing liability, nurses that "aren't allowed" to do several interventions that they can, for the same reason, all the while the facility has a massive billboard out the front advertising "24/7 nursing care" (and fees/costs commensurate with that).
boohoo123 2y ago
100% agree AI will ruin healthcare. I'm an IT director at a rural mental health clinic and I see the push for AI across my state and it's scary what they want. All i can do is push back. Healthcare is a case by case personal connection, something AI can't do. It only reduces humans down to numbers and operates on that. There is no difference between healthcare AI to a web scraper on webmd or mayo clinic.
- RandomGuy4567 2y ago
  [dead]
- moralestapia 2y ago
  I'm not vouching for AI, I actually thing it will only make things worse. But, >Healthcare is a case by case personal connection [...] I haven't felt this with doctors in like 20 years.
  - boohoo123 2y ago
    I get that but thats from the corporatization of clinics. They push for numbers not client well being. They want doctors to meet with x amount of patients a day and doctors dont control their schedule. So typically the execs meet together and say if we schedule every 20 minutes instead of 30 minutes we can generate x amount of more dollars. Which is why doctors just get you in and out as fast as possible because its their head on the chopping block if they dont meet the measures. Best thing you can do is find smaller clinics, which are becoming increasingly rare because the cost it takes to run one.
- m463 2y ago
  Your comment makes me think of a videogame I played recently called Eliza. (the zachtronics version, not the original text-based game from the 60s)
- glonq 2y ago
  Like any tool, the right amount of AI in the right places could be used to improve healthcare, and wrong/wrong could be used to ruin it. If your healthcare system is in the hands of politicians and/or capitalists, then expect the worst.
theptip 2y ago
> As a nurse, you end up relying on intuition a lot. It’s in the way a patient says something, or just a feeling you get from how they look There is a longstanding tension between those who believe human intuition is trustworthy, and the “checklist manifesto” folks. Personally I want room for both, there are plenty of cases where for example the nurse/doctor’s intuition fails and they forget to ask about travel or outdoor activities and miss some obvious tropical disease, or something situational like Lyme’s. I’ve spent a fair amount of time in a hospital and the human touch is really invaluable. My hope is that AI can displace the busywork and leave nurses more time to do the actual care. But a concrete example of the thing an AI will struggle with is looking at the overlapping pain med schedule, spotting that the patient has not been exhibiting or complaining of pain, and delaying one med a couple hours from the scheduled time to make the night schedule more pleasant for the patient. It’s hard to quantify the tradeoffs here! (Maybe you could argue the patient should be given a digital menu to request this kind of thing…)
- RHSeeger 2y ago
  It's interesting to me because AI and intuition serve some of the same purpose; to help the person being served find the answer. And both have similar limitations in that you need to verify what they're telling you. - If your gut tells you it's Lyme Disease, you don't just check it off as Lyme Disease and call it a day. You run tests to find out if it is - If the AI tells you it it's Lyme Disease, you don't just check it off as Lyme Disease and call it a day. You run tests to find out if it is AI should (almost?) never be used as the system of record. But it can be amazing in saving time; by guiding you to the right answer.
- 8338550bff96 2y ago
  None of this has to do with. AI. At all. This is politics and policy
- edanm 2y ago
  > There is a longstanding tension between those who believe human intuition is trustworthy, and the “checklist manifesto” folks. I really think this is a false dichotomy. Both are needed. I even vaguely remember from the Checklist Manifesto book, that one of the checklist items was specifically designed to make sure people actually talk to each other and share plans/thoughts/concerns before things like surgeries, which seems like a great way to make sure that intuition gets heard.
rubatuga 2y ago
What terrifies me is people will turn their brains off and blindly trust AI.
taylodl 2y ago
AI is a tool. Doctors can use the tool to ensure they haven't overlooked anything. At the end of the day, it's still doctors who are practicing medicine and are responsible for treatment. Yes, there are a lot of bridges we need to cross with regards to the best practices for using semi-intelligent tools. These tools are in their infancy, so I expect there's going to be a lot we learn over the next five to ten years and a lot of policy and procedure that get put in place.
Melonotromo 2y ago
[dead]
throwaway4220 2y ago
“Physician burnout” from documentation was the excuse for AI adoption - Stop Citrix or VMware or whatever. make a responsive emr where you don’t have to click buttons like a monkey
- bearjaws 2y ago
  Epic and Cerner are your main enemies if reducing burn out is the problem. Even then, the continued consolidation and inflow of PE into healthcare will be the next big problems.
tqi 2y ago
> We didn’t call it AI at first. The first thing that happened was these new innovations just crept into our electronic medical record system. They were tools that monitored whether specific steps in patient treatment were being followed. If something was missed or hadn’t been done, the AI would send an alert. It was very primitive, and it was there to stop patients falling through the cracks. Journalists LOVED The Checklist Manifesto when it came out in 2009, I guess if you call it AI then they will hate it? Similarly, in the early 2020s intuition was bad because of implicit bias, but now I guess it is good?
qgin 2y ago
Am I reading incorrectly or does this entire article come down to: 1. A calculated patient acuity score 2. Speech-based note-taking I didn’t see any other AI taking over the hospital.
ilaksh 2y ago
This is a problem with management, not AI. The acuity system obviously doesn't work well and wasn't properly rolled out. It's clear that they did not even explain how it was supposed to work. That's a problem with that system and it's deployment, not AI in general. Recording verbal conversations instead of making doctors and nurses always type things is surely the result of a massive portion of doctors saying that record keeping was too awkward and time intensive. It is not logical to assume that there is a privacy concern that overrides the time saving and safety aspect of doing that. People make that assumption because they are pre-conditioned against surveillance and are not considering physician burnout with record keeping systems. It's true that there are large gaps in AI capability and that software rollouts are quite difficult and poor implementation can cause a significant burden on medical professionals as it has here. I actually think if it's as bad as he says with the acuity then that puts patients in danger and should result in firings or lawsuits. But that doesn't mean that AI isn't useful and won't continue to become more useful.
- RandomGuy4567 2y ago
  I completelly agree with you. I am a Nurse and also work on AI research (deep learning). In my opinion most of the tension between healthcare profesionals and the diffferent AI systems out there boil down to poor management and shitty implementation. As "decision support tools" there should not be much mistrust or sense of fear towards AI. There are though some concerning isues regarding patient privacy, and employee surveilance/rights. Specially when these systems are sharing sensitive data with third parties. But this can (and provably should) be regulated. Nothing of this "mean that AI isn't useful and won't continue to become more useful". Right on point!
mannyv 2y ago
This article is total bullshit. The author uses "AI" as a shortcut for "technology that I don't understand." EPIC is an EMR, and apparently scores patients using an "algorithm." Let's call that "AI" because AI is hot. Scribe sounds like a transcriber. Oh boy. I know of offices that have a literal scribe, a person who's job it is to follow a doc around and transcribe to Epic. Automated? Why not. Having just been in the ICU with a family member for 60 days, I can say that nurses are good at some things and horrible at other things. And big picture thinking isn't something most nurses seem to be good at. Listening to nurses talk about what's wrong in healthcare is like talking to soldiers about what's wrong in the military.

110 points by redrove 2y ago

"We didn’t call it AI at first." Because the first things described in the article are not AI. They are ML at most. Then the article discusses a patient needs scoring method, moving from their own Low/Medium/High model to a scoring method on an unbound linear scale. The author appears to struggle with being able to tell if 240 is high or not. They don't state if they ever had training or saw documentation for the scoring method. Seems odd to not have these things but that if they did the scores would be a lot easier to interpret. Then they finally get to AI, and it's a pilot scheme for writing patient notes. That's all. If it sucks and hallucinates information it's not going to go live anywhere. No matter how many tech bros try to force it through. If the feedback model for the pilot is bad then the author should take issue with that. It's important that such tests give testers an adequate method to flag issues. Very much an AI = bad article. AI converging with medical technology is a really dangerous space, for obvious reasons. An article like this does make me worry it's being rushed through, but not because of the author's objections, instead because of their ignorance of what is and what isn't AI, and then on the other side of the apparent lack of consultation being offered by the technology providers even during testing stages.

ta988 2y ago
The industry calls AI even the dumbest linear system, I don't think it is right to blame non industry people if they don't use the right words after that.
ToucanLoucan 2y ago
> AI converging with medical technology is a really dangerous space, for obvious reasons. An article like this does make me worry it's being rushed through, but not because of the author's objections, instead because of their ignorance of what is and what isn't AI I mean, worth pointing out that OpenAI has been shoving LLMs into people's faces for going on about a year and half now at global scale and calling it AI, to a degree that now we have to call AI AGI, and LLM's get called AI, even though there is nothing intelligent about them whatsoever. Just saying when the marketing for a tech is grinding the edge of being misinformation itself, it's pretty normal for the end users to end up pretty goddamn confused by the end.
- bryanlarsen 2y ago
  For 60 years the Turing Test was the standard benchmark for artificial intelligence. Now machines can pass the test. The only goalpost moving I can see is the moving done by the people who insist that LLM's aren't AI.
- bitwize 2y ago
  The simple state machines and decision trees that make enemies move and attack in video games, we also call AI. AI is a loose term and always has been. It's like the term "robot": we call any machine a robot which either a) resembles a human or part of one (e.g., a robotic arm), or b) can perform some significant human labor that involves decision making (e.g., a robot car that drives itself). Similarly, AI is anything that makes, or seems to make, judgements that we think of as the exclusive purview of humans. Decision trees used to be thought of as AI, but today they are not (except, again, in the context of video games where they're used to control agents intended to seem alive).
  - ToucanLoucan 2y ago
    > The simple state machines and decision trees that make enemies move and attack in video games, we also call AI. Yes but no game companies have ever marketed their game asserting that the things your shooting are, in fact, self-aware conscious intelligences.
    - bitwize 2y ago
      Tell me you weren't around for the PlayStation 2 marketing blitz without telling me you weren't around for the PlayStation 2 marketing blitz. They actually named the thing's CPU the Emotion Engine.
- PhasmaFelis 2y ago
  That's been the technical definition in the AI research community for at least 50 years. "AI = machines that think like people" is the sci-fi definition.
develatio 2y ago
> No matter how many tech bros try to force it through. Are you sure about that? My wife is a nurse and she has to deal with multiple machines that were put in her unit just because the hospital has a contract with XYZ brand. It doesn't matter at all if these machines are 5x more expensive, x10 slower, x80 less effective, etc... compared with the "other" machines (from other brands). I'm actually terrified the same might happen with this.
happytoexplain 2y ago
I think the effort to keep the definition of AI narrow is not useful (and futile, besides). By both common usage and even most formal definitions, it's an umbrella term, and sometimes refers to a specific thing under that umbrella that we already have other words for if we would like to be more specific (ML, deep learning, LLM, GPT, genAI, neural net, etc).
bryanlarsen 2y ago
> If it sucks and hallucinates information it's not going to go live anywhere. It hallucinates and it's live in many places. My doctor uses it. AFAICT the hallucination rate is fairly low. It's transcription and summarization which has a lower hallucination rate than when it's asked for an answer. It's massively better than the alternative. Taking the time to write down proper notes is the difference between 4 patients per hour and 6 and a commensurate drop in income. So notes virtually always get short-changed. Very occasionally hallucinated notes are better than notes that are almost always incomplete.
- fhfjfk 2y ago
  What about increasing the supply of doctors rather than decreasing the time they spend with patients? I question the underlying premise that efficiency needs to increase.
  - daemin 2y ago
    Because the USA is for-profit healthcare industry they need to optimise and increase efficiency. That's the only way to make the numbers go up. Therefore fewer doctors, fewer nurses, fewer administrators (maybe), and more paying patients.
    - anadem 2y ago
      > more paying patients that's a chilling thought; don't give them ideas
      - slater 2y ago
        Surprising that you imply they haven't already been capitalizing (ha!) on that idea since the 1970s :D
    - bryanlarsen 2y ago
      It's no better in other countries. Other country's health care systems are typically monopsonies, which means we get crappier results for lower prices. So instead of the doctor choosing to do 6 patients per hour instead of 4, it's the government.

Loughla 2y ago

Healthcare is a massive cost for people, businesses, and governments. >So we basically just become operators of the machines. Driving down the cost of manufacturing because of process standardization and automation brought down the cost of consumer goods, and labor's value. If you don't think this is coming for every single area of business, you're foolish. Driving down labor costs is the golden goose. We've been able to collect some eggs through technology, but AI and the like will be able to cut that goose open and take all the eggs.

happytoexplain 2y ago
"don't think will happen" != "don't think is good"
coliveira 2y ago
> will be able to cut that goose open and take all the eggs You're completely right! AI will kill the golden goose, that is what this metaphor is all about.
thefz 2y ago
> Driving down the cost of manufacturing because of process standardization and automation brought down the cost of consumer goods, and labor's value. And quality as well.
- Spivak 2y ago
  Not really, the quality we're able to produce at scale is the best it's ever been in the world. The "quality of everything has been in decline" isn't due to advances in manufacturing but the economic factors that are present even without those advances. Rapid inflation over a relatively short period of time has forced everyone to desperately try and maintain close to their pre-inflation prices because workers didn't and aren't going to get the corresponding wage increases. I hope eventually there's a tipping point where the powers that be realize that our economy can't work unless the wealth that naturally accumulates at the top gets fed back into the bottom but the US pretty unanimously voted for the exact opposite so I desperately I hope I'm wrong or that this path can work too.

coliveira 2y ago

Everyone should be terrified. The "promise" of AI is the following: remove any kind of remaining communication between humans, because that is "inefficient", and replace it with an AI that will mediate all human interactions (in business and even in other areas). In a few years, AIs trained by big corps will run the show and humans will be required to interface with them to do anything of value. Similar to what they want to do nowadays with mobile/enterprise systems, but at a much deeper level.

anthonyskipper 2y ago
Some of us look forward to that future where you mostly just interact with AI. The one depressing us is not turning running over our goverment to AI. The sooner we can do that the better, you can't trust humans.
- maxehmookau 2y ago
  > Some of us look forward to that future where you mostly just interact with AI. What is it about that that appeals to you? I'm genuinely curious. A world without human interaction feels like a world I don't want to exist in.
  - cptaj 2y ago
    They expect AI bureaucracy to be more effective than human bureaucracy. I expect this to be entirely true in some cases.
  - andy_ppp 2y ago
    If you are autistic (for example) I'm guessing human interaction can be extremely difficult and very stressful and triggering. Machines are much more amenable and don't have loads of arbitrary unwritten rules the way humans do. Maybe the idea of being entrapped by bureaucracy introduced by machines will be better than the the bureaucracy introduced by humans?
    - add-sub-mul-div 2y ago

Mathnerd314 2y ago

> There’s a proper way to do this. Is there? Seems like people will complain however fast you roll out AI, so you might as well roll it out quickly and get it over with.

oldpersonintx 2y ago

[dead]

mro_name 2y ago

There's this earthquake phrase in past tense: > We felt like we had agency.

paulnpace 2y ago

I think something this article demonstrates is how AI implementation is resulting in building resistance to AI because AI is being forced onto people instead of being demanded by those people. Typically, the people doing the forcing don't understand very well the job the people being forced to adopt AI actually perform.

parasense 2y ago

I did a bunch of research essays into medical uses of AI/ML and I'm not terrified, in fact the single most significant use of these technologies is probably in or around healthcare. One of the most cited uses would be expert analysis of medical imaging, especially breast cancer imaging. There is a lot of context to unpack around breast cancer imaging, or more sucinctly put, controversial drama! The fact is there is a statisticalluy high rate of false positives in breast cancer diagnostics made by human doctors. This reality resulted in a big overall policy shift to have women breast scanned less often, depending on their age, or something like that. Because so many women were victimized with breast surgery that turned out to be false positive or whatever. The old saying to make an omlet one must break a few eges is sometimes used, and that's a terrible euphamism. AI has proven to be better at looking at medical image, and in the case of breast cancer seems to out perform humans. And of course the humans have a monotonous job revewing image after image, and they want to be safe instead of latter being sorry, so of course they have high false possitives. The machines never get tired, they never get biased (this is a bone of contention), and they never stop. Ultimatly a human doctor still has to review the images, and the machines simply inform if the doctor is being too agressive in diagnosis, or possibly missing something. The whole thing gets escellated if there is any disparity. The out come from early studdies is encouraging, but these studies take years, and are very expensive. One of the biggest problems is the technology proficiency of medical staff is low, and so we are now in a situation where software engineers are cross traning to be at the level of a nurse or even doctors in rare cases.

buffington 2y ago
One very important part your comment doesn't mention: a real human being has to actually take images for the AI to analyze. The amount of training a radiation technologist (the person who makes you put your body in uncomfortable positions when you break something) is significant. My partner has made a career of it, and the amount of school needed and clinical hours is non-trivial, and harder to do than becoming a nurse from what I understand. They need to know as much about bones as orthopedic surgeons while also knowing how radiation works, as well as how the entire imagining tech stack works, while also having the soft skills needed to guide injured/ill patients to do difficult things (often in the midst of medical trauma). The part where a doctor looks at images is really just a very small part of the entire "product." The radiologists who say "there's a broken arm" are never in the room, never see the patient, never have context. It's something that, frankly, an AI can do much more consistently and accurately at this point.

lekanwang 2y ago

As an investor in healthcare AI companies, I actually completely agree that there's a lot of bad implementations of AI in healthcare settings, and what practitioners call "alarm fatigue" as well as the feeling of loss of agency is a huge thing. I see a lot of healthcare orgs right now roll out some "AI" "solution" in isolation that raises one metric of interest, but fails to measure a bunch of other systemic measures. Two thoughts: 1: I think the industry could take cues from aerospace and the human factors research that's drastically improved safety there -- autopilot and autoland systems in commercial airliners are treated as one part of a holistic system with the pilot and first officer and flight attendants in keeping the plane running smoothly. Too few healthcare AI systems are evaluated holistically. 2: Similarly, if you're going to roll out a system, either there's staff buy-in, or the equilibrium level of some kind of quality/outcomes/compliance measure should increase that justifies staff angst and loss of agency. Not all AI systems are bad. One "AI" company we invested in, Navina, is actually loved by physicians using them, but the team also spent a LOT of time doing UX research and feedback with actual users and the support team is always super responsive.

hartspear 2y ago
Agreed, what was obviously missing from this person's experience was an opportunity to guide deployment in ways that work well for their understanding of the workflow. I think companies often fail to explore what actually happens on the ground of their organizations and make assumptions that are not accurate. AI could be a huge help to nurses but it takes conversations and direct insight during development to get it right.

heironimus 2y ago

This is the same technology story told thousands of times a day with nearly every technology. Medical seems to be especially bad at this. Take a very promising technology that could be very useful. Jump on it early without even trying to get buy in and without fully understanding the people that will use it. Then push a poor version of it. Now the nurses hate the tech, not the poor implementation of it. The techies then bypass the nurses because they are difficult, even though they could be their best resource for improvement.

cowmix 2y ago

This article feels “ripped from today’s headlines” for me, as my mother-in-law was recently in the ICU after a fall that caused head trauma. The level of AI-driven automated decision-making is unsettling, especially as it seems to allow large organizations to deflect accountability—“See? The AI made us do it!” I’m not entirely sure what guided her care—or lack thereof—but, as someone who frequently works in healthcare IT, I see these issues raised all the time. On the other hand, having access to my own “AI” was incredibly helpful during her incident. While in the ICU, speaking with her doctors, I used ChatGPT and Claude to become a better advocate for her by asking more informed questions. I could even take pictures of the monitors tracking her vitals, and ChatGPT helped me interpret the readings, which was surprisingly useful. In this “AI-first” world we’re heading into, individuals need their own tools to navigate the asymmetric power dynamic with large organizations. I wonder how long it will be until these public AI models get “tweaked” to limit their effectiveness in helping us question “the man.”

cowmix 2y ago
Side note: I tried the same questions with some local LLMs I’m running at home—unfortunately, they’re nowhere near as good or useful. I hope local models improve quickly, so we’re not left depending on the good graces of big LLM(tm).
wing-_-nuts 2y ago
The article feels very much like a union rep fighting automation. If AI is provably worse, we should see that come up in the AI making 'bad calls' vs the human team. You would even see affects on health outcomes. One place I'd really like an all seeing eye AI overlord is in nursing home care. I have seen family members lie in filth with clear signs of an infection. I am confident, if we hadn't visited, seen this, and got her out of there she would have died there, years before her time.
- FireBeyond 2y ago
  Sadly, the one thing I took from my time as an EMT and paramedic was which nursing homes to consider and which to avoid. I filed more than one complaint with the DOH. It's a standing joke that whenever 911 crews respond to a nursing home, the report you'll get from staff will be a bingo game of: - "I just got on shift; I don't know why you were called." - "This is not my usual floor; I'm just covering while someone is on lunch. I don't know why you were called." - [utterly unrealistic set of vitals, in either direction, healthy, lively vitals for someone who is not thriving, or "should be unconscious" vitals for someone lively and spry] - [extended time waiting for patient notes, history, an outdated med list with everything they've taken in their ten years at the facility] And so on. I (generally) don't blame the floor staff (though some things, as you describe, are inexcusable) but management/ownership. The same management/ownership that has the policy to call 911 for anything more involved than a bandaid for some weird idea of managing liability, nurses that "aren't allowed" to do several interventions that they can, for the same reason, all the while the facility has a massive billboard out the front advertising "24/7 nursing care" (and fees/costs commensurate with that).

boohoo123 2y ago

100% agree AI will ruin healthcare. I'm an IT director at a rural mental health clinic and I see the push for AI across my state and it's scary what they want. All i can do is push back. Healthcare is a case by case personal connection, something AI can't do. It only reduces humans down to numbers and operates on that. There is no difference between healthcare AI to a web scraper on webmd or mayo clinic.

RandomGuy4567 2y ago
[dead]
moralestapia 2y ago
I'm not vouching for AI, I actually thing it will only make things worse. But, >Healthcare is a case by case personal connection [...] I haven't felt this with doctors in like 20 years.
- boohoo123 2y ago
  I get that but thats from the corporatization of clinics. They push for numbers not client well being. They want doctors to meet with x amount of patients a day and doctors dont control their schedule. So typically the execs meet together and say if we schedule every 20 minutes instead of 30 minutes we can generate x amount of more dollars. Which is why doctors just get you in and out as fast as possible because its their head on the chopping block if they dont meet the measures. Best thing you can do is find smaller clinics, which are becoming increasingly rare because the cost it takes to run one.
m463 2y ago
Your comment makes me think of a videogame I played recently called Eliza. (the zachtronics version, not the original text-based game from the 60s)
glonq 2y ago
Like any tool, the right amount of AI in the right places could be used to improve healthcare, and wrong/wrong could be used to ruin it. If your healthcare system is in the hands of politicians and/or capitalists, then expect the worst.

theptip 2y ago

> As a nurse, you end up relying on intuition a lot. It’s in the way a patient says something, or just a feeling you get from how they look There is a longstanding tension between those who believe human intuition is trustworthy, and the “checklist manifesto” folks. Personally I want room for both, there are plenty of cases where for example the nurse/doctor’s intuition fails and they forget to ask about travel or outdoor activities and miss some obvious tropical disease, or something situational like Lyme’s. I’ve spent a fair amount of time in a hospital and the human touch is really invaluable. My hope is that AI can displace the busywork and leave nurses more time to do the actual care. But a concrete example of the thing an AI will struggle with is looking at the overlapping pain med schedule, spotting that the patient has not been exhibiting or complaining of pain, and delaying one med a couple hours from the scheduled time to make the night schedule more pleasant for the patient. It’s hard to quantify the tradeoffs here! (Maybe you could argue the patient should be given a digital menu to request this kind of thing…)

RHSeeger 2y ago
It's interesting to me because AI and intuition serve some of the same purpose; to help the person being served find the answer. And both have similar limitations in that you need to verify what they're telling you. - If your gut tells you it's Lyme Disease, you don't just check it off as Lyme Disease and call it a day. You run tests to find out if it is - If the AI tells you it it's Lyme Disease, you don't just check it off as Lyme Disease and call it a day. You run tests to find out if it is AI should (almost?) never be used as the system of record. But it can be amazing in saving time; by guiding you to the right answer.
8338550bff96 2y ago
None of this has to do with. AI. At all. This is politics and policy
edanm 2y ago
> There is a longstanding tension between those who believe human intuition is trustworthy, and the “checklist manifesto” folks. I really think this is a false dichotomy. Both are needed. I even vaguely remember from the Checklist Manifesto book, that one of the checklist items was specifically designed to make sure people actually talk to each other and share plans/thoughts/concerns before things like surgeries, which seems like a great way to make sure that intuition gets heard.

rubatuga 2y ago

What terrifies me is people will turn their brains off and blindly trust AI.

taylodl 2y ago

AI is a tool. Doctors can use the tool to ensure they haven't overlooked anything. At the end of the day, it's still doctors who are practicing medicine and are responsible for treatment. Yes, there are a lot of bridges we need to cross with regards to the best practices for using semi-intelligent tools. These tools are in their infancy, so I expect there's going to be a lot we learn over the next five to ten years and a lot of policy and procedure that get put in place.

Melonotromo 2y ago

[dead]

throwaway4220 2y ago

“Physician burnout” from documentation was the excuse for AI adoption - Stop Citrix or VMware or whatever. make a responsive emr where you don’t have to click buttons like a monkey

bearjaws 2y ago
Epic and Cerner are your main enemies if reducing burn out is the problem. Even then, the continued consolidation and inflow of PE into healthcare will be the next big problems.

tqi 2y ago

> We didn’t call it AI at first. The first thing that happened was these new innovations just crept into our electronic medical record system. They were tools that monitored whether specific steps in patient treatment were being followed. If something was missed or hadn’t been done, the AI would send an alert. It was very primitive, and it was there to stop patients falling through the cracks. Journalists LOVED The Checklist Manifesto when it came out in 2009, I guess if you call it AI then they will hate it? Similarly, in the early 2020s intuition was bad because of implicit bias, but now I guess it is good?

qgin 2y ago

Am I reading incorrectly or does this entire article come down to: 1. A calculated patient acuity score 2. Speech-based note-taking I didn’t see any other AI taking over the hospital.

ilaksh 2y ago

This is a problem with management, not AI. The acuity system obviously doesn't work well and wasn't properly rolled out. It's clear that they did not even explain how it was supposed to work. That's a problem with that system and it's deployment, not AI in general. Recording verbal conversations instead of making doctors and nurses always type things is surely the result of a massive portion of doctors saying that record keeping was too awkward and time intensive. It is not logical to assume that there is a privacy concern that overrides the time saving and safety aspect of doing that. People make that assumption because they are pre-conditioned against surveillance and are not considering physician burnout with record keeping systems. It's true that there are large gaps in AI capability and that software rollouts are quite difficult and poor implementation can cause a significant burden on medical professionals as it has here. I actually think if it's as bad as he says with the acuity then that puts patients in danger and should result in firings or lawsuits. But that doesn't mean that AI isn't useful and won't continue to become more useful.

RandomGuy4567 2y ago
I completelly agree with you. I am a Nurse and also work on AI research (deep learning). In my opinion most of the tension between healthcare profesionals and the diffferent AI systems out there boil down to poor management and shitty implementation. As "decision support tools" there should not be much mistrust or sense of fear towards AI. There are though some concerning isues regarding patient privacy, and employee surveilance/rights. Specially when these systems are sharing sensitive data with third parties. But this can (and provably should) be regulated. Nothing of this "mean that AI isn't useful and won't continue to become more useful". Right on point!

mannyv 2y ago

This article is total bullshit. The author uses "AI" as a shortcut for "technology that I don't understand." EPIC is an EMR, and apparently scores patients using an "algorithm." Let's call that "AI" because AI is hot. Scribe sounds like a transcriber. Oh boy. I know of offices that have a literal scribe, a person who's job it is to follow a doc around and transcribe to Epic. Automated? Why not. Having just been in the ICU with a family member for 60 days, I can say that nurses are good at some things and horrible at other things. And big picture thinking isn't something most nurses seem to be good at. Listening to nurses talk about what's wrong in healthcare is like talking to soldiers about what's wrong in the military.