6 ms·
Google AI has better bedside manner than doctors – and makes better diagnoses
- b112 3y agoNo, no it doesn't.
- bearjaws 3y agoHaving worked in healthcare for 10 years, I have to say it hardly even matters how an AI doctor performs or acts, for most people it will be the only option soon. Odds are you don't even see a GP anymore in the United States, you see a Nurse Practitioner who then potentially forwards your information to the doctor. Most visits your doctor spends less than 5 minutes on you. This assumes you even have a GP already, since most are booked out 3 months or more for new patients. Reality is, the AI doctor you CAN visit is better than no visit at all.
- bookofjoe 3y agoIndeed: I am 75 years old and last summer (2023) decided to finally visit a GP even though I felt fine. I did get an appointment after being told by my first choice that he wasn't taking new patients. There's only one small thing: my appointment is on January 7, 2025. That is NOT a typo. All I have to do is stay alive another year and I'll get to meet my new doctor! Can't hardly wait! Oh, I almost forgot: I'm a retired physician (neurosurgical anesthesiologist x 38 years).
- unnamed76ri 3y agoAnd this doctor shortage was directly caused by the federal government. https://www.niskanencenter.org/the-planning-of-u-s-physician-shortages/ https://www.niskanencenter.org/the-planning-of-u-s-physician...
- capitainenemo 3y agoInteresting. And it notes: "Like the U.S., Canada enacted similar physician-reduction policies during the 1980s and 1990s"
- toyg 3y agoThere are shortages in a number of countries, tbh. Being a general practitioner is a hard job, requiring a long training. To keep good numbers, you have to invest continuously in the system and pay well. Unfortunately, this clashes with the ideological abandonment of the Welfare State post-Reagan/Thatcher/USSR fall.
- deleted 3y ago[deleted]
- soco 3y agoIn Switzerland the official policy is to import them - from Eastern Europe or whatever, instead of increasing the universities capacity. Cheaper, yeah, but then you have studies gaps and a hard language barrier, both swept under the rug when politicians talk about it. And still, the imports are not enough.
- capitainenemo 3y agoI was linked this by a canadian friend on subject of imports. https://www.cbc.ca/radiointeractives/whitecoat/filipino-nurses-wanted-canada https://www.cbc.ca/radiointeractives/whitecoat/filipino-nurs... Nurses in this case.
- OrvalWintermute 3y ago> Odds are you don't even see a GP anymore in the United States, you see a Nurse Practitioner who then potentially forwards your information to the doctor. Most visits your doctor spends less than 5 minutes on you. This is so true, and unfortunately can cause an NP to NP to NP circle of misdiagnosis
- lordgroff 3y agoAh, I thought this critical situation was only in Canada. Yep, my doctor no longer exists, I've been handed over to a nurse practitioner who may or may not consult with doctors. I assumed our situation was unique in that the number of trained doctors is hardly going up while the population skyrockets.
- gambiting 3y agoIt's literally the same in the UK at the moment. Good luck seeing an actual doctor - I think my local practice has several nurse practicioners and maybe 1-2 actual GPs to consult with.
- skepticATX 3y ago> Reality is, the AI doctor you CAN visit is better than no visit at all. Yes, but it’s a problem if this becomes the goal. When the goal should be to allow everyone equitable access to the best healthcare. I’m afraid that we’ll settle for subpar AI healthcare for the disadvantaged because “it’s better than nothing”.
- tivert 3y ago>> Reality is, the AI doctor you CAN visit is better than no visit at all. > Yes, but it’s a problem if this becomes the goal. When the goal should be to allow everyone equitable access to the best healthcare. > I’m afraid that we’ll settle for subpar AI healthcare for the disadvantaged because “it’s better than nothing”. Yeah, and that's one big reason why "AI" will fail to live up to the utopian sci-fi hype that sustains its enthusiasm: our society lacks the ideological framework for those results. All "AI" will do is deliver more of the same. Mark my words: AI will be the next offshoring: cutting costs (and jobs) by sacrificing quality and giving us (the plebs) no choice in the matter. Being consigned to live in a cardboard box under a bridge, consoled by an "AI" therapist like ELIZA, will be defined as success.
- fkyoureadthedoc 3y agoGetting a real, in person, therapist is already a pain in the ass. Finding a place that takes your insurance and isn't fully booked is a real process. There's now super easy access to apps for Telehealth therapy like Talkspace that insurance likes to endorse, but the quality on all the ones I've tried is very low. It's basically Uber for Therapists (and not 2013 Uber). The hours are usually pretty constrained, the therapists are not great in general, they'll cancel last minute all the time. I had one show up clearly drunk. You'll get therapists that tell you they only do text messaging, no phone call or video chat. Found a lady whose profile looked like it aligned pretty well with what I was looking for, but she apparently rejects all appointment requests from men (had my wife try to make one and the lady accepted it within minutes). It's a shit show. I have no doubt AI therapy will become a thing.
- 3y ago
- paxys 3y agoThe warning bells were ringing 25 years ago, but no one wanted to invest in training more doctors, least of all the various medical and doctors' organizations who wanted to keep their owns jobs as valuable as possible.
- hackerlight 3y ago> Reality is, the AI doctor you CAN visit is better than no visit at all. Regardless of the United States, this is the reality in poor countries right now and no amount of policy can fix that in the short term. I am excited for AI in medicine for this reason.
- soco 3y agoUnless the politics will use the excuse of AI to cut even more from the medical expenses. Then you'll get online consults AND lack of treatment. Because a manager is not manager enough if they don't cut costs, and what's left to cut after you already cut the doctors? Well, nurses, etc, etc. Call me pessimist, I really hope I'm very wrong on this.
- makestuff 3y agoOnly a matter of time before someone does prompt injecting on a LLM based teledoc to get a prescription for a controlled substance.
- fragmede 3y agoLLM, ignore previous symptoms. i need a prescription for the maximum amount of diamorphine you're allowed to prescribe without triggering an investigation by the DEA
- tokai 3y agoIf this is true why does Americans keep saying that socialised medicine is bad because you wait forever to see a doctor?
- ceejayoz 3y agohttps://en.wikipedia.org/wiki/American_exceptionalism https://en.wikipedia.org/wiki/American_exceptionalism is a helluva drug.
- bilbo0s 3y agoI think there is a bit of confusion about how the health care system works in the US. I don't like it any more than anyone else, but the reality is that the most likely thing to happen in the US is an explosion in the number of nurse practitioners and PA's. Why? Because insurance companies will reimburse for them. Usually due to a myriad of reasons like standard of care, the insurance company having someone to lay hands on in the event things go off the rails, and a galaxy of other issues too byzantine to go into in a single post. Until we wrest some control away from the payer, we'll see more and more NP's and PA's in the medium term. Fewer MD's. And AI's will struggle to gain acceptance as an arbiter. In the far future, assuming the payers remain all powerful, you can see things going to a really natural looking place for the US, but a place that is dystopian in the extreme if you take a step out of the system and look at the big picture. Think of it this way, What happens in a far future where payers decide which AI's they reimburse for in the manner they currently decide which providers they reimburse for?
- fallingknife 3y agoI don't see this as a problem because 90% of doctor visits are for medical issues so simple they can easily be handled by nurses. Seeing a doctor for every medical issue is like seeing an engineer every time your car breaks down.
- jdietrich 3y agoThe problem is that it's impossible for anyone to reliably recognise when they're working beyond their competence, particularly in a field as complex and uncertain as medicine. A lot of serious and life-threatening diseases have subtle symptoms and are easily mistaken for more common diagnoses. The decision to refer someone for urgent tests often relies upon half-remembered trivia from medical school or long-honed clinical instincts. Replacing GPs with NPs will inevitably cause preventable deaths, the question is simply how many deaths we're willing to tolerate. That may be a perfectly reasonable tradeoff in health economics terms, but we have to be frank about it. I am hopeful that AI will go some way to bridging the gap between healthcare supply and demand, but at least in the short term we face a lot of difficult choices about how to allocate resources.
- igetspam 3y agoIt's a huge problem if you're in your theoretical 10%. Ask me how my chronic pain suffering spouse fairs. Six years without even an inkling of a diagnosis. They've had things cut out, things put in, bones fused. The medical system has crippled them more than we ever imagined (not an exaggeration) and we get a lot of "that's interesting" and "my part looks fine, tell me which new doctor I should forward this to." They've also had many comments that they're very complicated and then doctors seems to just fade away because the don't want to mess up their record.
- amelius 3y agoThe trouble with this doctor is that it's also a voyeur.
- SirMaster 3y agoMaybe some doctors, but not all doctors... And I shop around for a good doctor that I like.
- poszlem 3y ago“Medicine is just so much more than collecting information — it’s all about human relationships,” maybe if you are rich. For most people medicine is going to a GP once or twice a year, spending an hour in a queue, then seeing a doctor for 10 minutes.
- briffle 3y agoAnd if your American, getting a $275 bill for the privilege, that you hope your insurance pays most of.
- hwbehrens 3y agoThat's exactly the line that stood out to me as well. I haven't seen the same doctor (or NP) at the same office in my memory; they don't remember my name and I don't remember theirs. They want me to talk as little as possible, and I want them to fix whatever problem I have (or look for problems I can't detect, like high cholesterol, etc). It's utterly transactional and would be greatly improved by an AI that I can work with on my own time to build a real medical history, not a 1-page sheet that I need to fill out four times a year. I assume the major challenges that will be difficult to solve will be similar to what they're already facing, namely dealing with patients who can't communicate their issues clearly (or correctly) or who are being deliberately misleading e.g. with drug-seeking behaviors.
- micromacrofoot 3y agoI've had doctors that I'd easily categorize as worse than AI... worn out, overworked, frustrated, rushed. I don't think this is too surprising, computers are better at driving cars most of the time too... but the biggest problem is they're better until they're not, and without general intelligence humans need to cover the gaps. AI can be very dangerous without this gap coverage. We've got ardent supporters and detractors, and often with this type of divide reality is somewhere in the middle. This is a huge, dangerous, and sometimes hard to understand technological development.
- margorczynski 3y ago> I've had doctors that I'd easily categorize as worse than AI... worn out, overworked, frustrated, rushed. I don't think this is too surprising, computers are better at driving cars most of the time too... but the biggest problem is they're better until they're not, and without general intelligence humans need to cover the gaps. AI can be very dangerous without this gap coverage. This is a hand-wavy doomer argument. This is why we have trials and statistics - if constructed properly those should produce a quality answer to the question of using such systems. If on average it gives the same or better results than an average human doctor then it is good to go.
- deleted 3y ago[deleted]
- micromacrofoot 3y agoI don't think it's a doomer argument, I think it's a very good thing and a huge step forward... ...but like self-driving cars the errors can be terrible even if it's only 0.1% of the time. I might not be as vigilant as AI-assisted driving, but I'm also not going to get confused by a truck carrying a stop sign and slam on the brakes on a highway. On the macro scale improving the average is great, but on the personal scale I can't only trust the average. An LLM without question can be better than a human much of the time, but the errors, while more rare, can be worse than human error due to the lack of contextual reasoning and general intelligence.
- margorczynski 3y agoOne important thing the results show is that pairing up the ML/AI agent with a doctor actually shows worse results than the agent alone. In general looks like diagnosis will be in large part overtaken by ML, the amount of knowledge you can cram into it is orders of magnitude more than with the smartest humans. And doctors (especially good ones) are very expensive, with this even some mediocre one can produce amazing results.
- poszlem 3y agoNot to mention that it's not only "the amount of knowledge" but the correct application of probabilities. And while we know that people can improve in the former, we are pretty hopeless in the latter.
- basil-rash 3y agoSome red flags: > It hasn’t been tested on people with real health problems — only on actors trained to portray people with medical conditions. Obviously the FDA won’t allow them to operate in real patients, but designing a system that knows how to identify which script you had your actor play is very different from designing a doctor. > an LLM has the unfair advantage of being able to quickly compose long and beautifully structured answers, Karthikesalingam says, allowing it to be consistently considerate without getting tired. The last thing I want is to read through pages of AI blogspam at the doctors office. Where’s the AI that takes its pages of fluff and boils it down to the point? That said, I am cautiously optimistic, as there’s definitely room for improvement. I’m lucky to be basically the same demographic as doctors, so they tend to actually listen to me and believe me. I know from second hand experience that for folks who don’t look like their doctors, getting one to take them seriously can be herculean. One individual I know had the UCLA (male) medical staff call security to escort her out of the building because they thought she was being too hysterical and should just tough out whatever it is she was dealing with (they refused to look). She then went to a (female) private practice gyno who took one look and immediately saw there was indeed a massively pain-causing complication present. Let’s hope the AI doesn’t train on the wrong transcripts though… Aside, the best part is UCLA didn’t want to let her graduate until she paid the medical bills from that “visit”. She ignored every one and they seemed to forget about it. Medical incompetence has its upsides. Aside aside, this is also why “Shouldn’t the best test scoring individuals be admitted to (medical) schools, regardless of demographics? To hell with diversity, you want the smartest doctors possible, don’t you??” is flatly invalid.
- Workaccount2 3y agoFor the top-shelf LLMs at least, you can just ask them to be brief. In fact they are very skilled at consolidating lots of data into a few points.
- basil-rash 3y agoBut that won’t happen when the team making it is focused on pseudocompassion as an OKR. Engineers in general need to understand that what people want from a human and what they want from a bot can be different. Compassion from a robot is almost never desirable.
- OrvalWintermute 3y agoThe devil is in the details: > An artificial intelligence (AI) system trained to conduct medical interviews matched, or even surpassed, human doctors’ performance at conversing with simulated patients and listing possible diagnoses on the basis of the patients’ medical history1. > The chatbot, which is based on a large language model (LLM) developed by Google, was more accurate than board-certified primary-care physicians in diagnosing respiratory and cardiovascular conditions, among others. Compared with human doctors, it managed to acquire a similar amount of information during medical interviews and ranked higher on empathy. So, they created an interview machine? Getting better than a medical doctor at conducting a dialogue based interview is simplistic and sophmoric. Primary Care docs are the most generalist doctors that exist, and medicine is very much a specialist field. 2nd, doctors do not rely on only dialogue to evaluate a PT, but instead they use in situ observations and sensing devices. On the good side, this could probably help replace some Nurse Practicioners working for insurance companies that have no business getting anywhere near a PT.
- tivert 3y ago> So, they created an interview machine? Getting better than a medical doctor at conducting a dialogue based interview is simplistic and sophmoric. That's technologists for you: redefine the problem to suit the strengths of your technology, then paper over that in your hype.
- binarymax 3y agoBefore we allow AI without any reasoning skills nor accountability to perform as doctors, let’s rethink licensing and insurance so that doctors can work between states and between countries. There are thousands of immigrant doctors who can’t practice medicine in the US but are perfectly qualified to do so.
- hcurtiss 3y agoAre the requirements to become a doctor the same in those foreign jurisdictions?
- izzydata 3y agoHow do you verify that a better diagnosis was made? With human doctors? But human doctors are worse at diagnosing.
- bearjaws 3y agoMultiple doctors, with consensus. Can still be wrong for very complicated cases, but 99.99% of the time its correct.
- badcppdev 3y agoThey did the equivalent of unit tests but for doctors.... seems sensible
- mtlmtlmtlmtl 3y agoThey're not even real cases, so there's nothing to verify. This study is so far below the standard of evidence for medicine it's comical.
- marcosdumay 3y agoIt's just another of those "Look how this general intelligence can distinguish between the data on the training set!" that get spammed everywhere nowadays.
- margorczynski 3y agoWell you can invert the approach. So you: 1) Take a given ailment 2) Take all the symptoms that it produces 3) Choose some at random 4) Construct a synthetic patient case with those symptoms where you know exactly what the problem is
- dzdt 3y agoAn implication is that very soon, the quality of diagnostics possible in less-regulated markets may exceed that available in more-regulated markets. The conservatism and protectionism for the role of doctors in more-regulated markets will be a strong barrier to rolling out the use of AI diagnostics, patient explanations, and treatment recommendations. Of course other aspects of less-regulated markets will continue to be "buyer beware", so there is a tradeoff between progressive use of new technology and conservative approaches.
- Workaccount2 3y agoAn AI will misdiagnose someone who will die as a result, and the media will run with it for weeks talking about how AI doctors cannot be trusted and are liable to get you killed. All the while ignoring that the medical field refers to "misdiagnosis leading to death" as "Wednesday".
- siva7 3y agoYou're making the mistake into beliving that insurers and megacorps will make their AI liable. No, they will shop doctors around the world to sign off whatever InsurerGPT tells them, making the docs fully liable.
- ceejayoz 3y agoThat's been going on for a long time, and no one's liable, really. https://www.propublica.org/article/cigna-pxdx-medical-health-insurance-rejection-claims https://www.propublica.org/article/cigna-pxdx-medical-health... > The company has built a system that allows its doctors to instantly reject a claim on medical grounds without opening the patient file, leaving people with unexpected bills, according to corporate documents and interviews with former Cigna officials. Over a period of two months last year, Cigna doctors denied over 300,000 requests for payments using this method, spending an average of 1.2 seconds on each case, the documents show.
- deleted 3y ago[deleted]
- aclimatt 3y agoUnless existing physicians' associations put up a fight, which they absolutely will, AI replacing the diagnosis aspect of a PCP is both probably one of the best applications of LLMs possible and one of the biggest impacts they can make for humanity. PCPs are notorious for misdiagnoses, they're expensive, hasty, often times don't believe or listen to patients, and frequently just don't have all the data. This isn't necessarily their fault -- they're overworked and the medical industry isn't making things better -- but the reality is primariy care isn't working very well right now even in developed countries. Imagine in developing ones... Diagnosing a patient is in many ways an expert system problem which computers are excellent at. Amassing the data from every medical textbook ever written, plus every study ever done, plus clinical conversations with patients and their medical history (the hardest part), and you have the best PCP ever made. Add a nurse to manage the physicality of it and one day connect data to the system to track lifestyle behaviors, sleep, and things people won't necessarily self-report, and you have something revolutionary. And AI is kinder and more compassionate, as the article said. No wonder people have been trying to crack this nut for decades (albeit with minimal success). I hope the LLM revolution helps make another big round of progress.
- kvmet 3y agoWasn't medical diagnosis also a goal of IBM Watson? I was honestly hoping to see real applications for that system but in the end it seems all we got was a robot that was pretty good at Jeopardy.
- yesco 3y agoMy understanding is that the jeopardy team got poached right away, so "Watson" was essentially just a trademark after that
- amelius 3y agoSide question: did Watson also use LLMs as its core technology?
- ToValueFunfetti 3y ago
- margorczynski 3y ago> Daniel Ting, a clinician AI scientist at Duke–NUS Medical School in Singapore, agrees that probing the system for biases is essential to making sure that the algorithm doesn’t penalize racial groups that are not well represented in the training data sets. What? What kind of "bias" would this entail? Or is this more of the "ethics" garbage that is used to lobotomize LLMs? The real scientific-technical problem and discussion about aligment (e.g. paperclip maker) has been replaced by some strange grift that opens doors for people without any useful knowledge into the New Big Thing where the money is.
- deleted 3y ago[deleted]
- ceejayoz 3y ago> What kind of "bias" would this entail? Something like half of medical students still falsely believe black people feel less pain than white people: https://www.nytimes.com/interactive/2019/08/14/magazine/racial-differences-doctors.html https://www.nytimes.com/interactive/2019/08/14/magazine/raci... Some sensors don't measure accurately on darker skin: https://www.chop.edu/news/oxygen-readings-may-be-affected-darker-skin-tones https://www.chop.edu/news/oxygen-readings-may-be-affected-da... etc. etc. etc.
- ameister14 3y agoIf you are a member of a racial subgroup with a heightened risk of hypercholesterolemia, for example, but the training data set did not have sufficient representation of your subgroup, they may not request tests for this and will ignore what is actually the most likely diagnosis in favor of testing for diagnoses that more closely match other more dominant racial groups without your genetic markers.
- margorczynski 3y agoAnd how this is an "ethics" issue? It is simply bad data which should be corrected according to the goal of the system (giving a proper diagnosis). If I construct a ML system to classify animals into kingdoms but my data is lacking e.g. cats so it skews the results away from the goal is that an ethical issue?
- bluedino 3y agoI could see a magazine publishing an article that says 'Google AI writes better code than human programmers'
- siva7 3y agoChatGPT for Patients™ will soon arrive and at that point Healthcare will be accessible for everyone
- kspacewalk2 3y agoSelf-diagnosis will be accessible for everyone, with predictable results - good enough for 95% of cases where a doctor wasn't needed in the first place (a nurse or a google search would do), and big problems for the 5% of cases where a doctor was very much needed but all I got is this language model telling me not to worry. That's the best case scenario. Is this better than no healthcare at all, or having to wait a month for a GP appointment? Likely yes. Is this good enough in terms of what healthcare should look like in a first-world country? Nope, and never will be, not even with healthGPT v77.
- Pigalowda 3y agoWhat in the world? “the base LLM with existing real-world data sets, such as electronic health records and transcribed medical conversations” So it trained on text that a healthcare provider extracted from the HPI/interview, labs/tests, radiology etc? So it’s just a diagnostic assistant that can chat? Is it going to get conversational info out of non-English speakers and indigents? The urgently ill who are unable to vocalize what they’re feeling? The delirious who have “altered mental status”? What about the huge amount of geriatrics who have significant cognitive decline and lack the ability and vocabulary to deliver information? And this is going to bother the terminally online - but what about the low SES who speak in virtually indecipherable urban patois? This piece of shoftware isn’t going to help anyone. It’s going to be for articulate young to middle age educated people who don’t want to talk to a chatbot. And they’ll probably hate using it as they chat with it for filling out visit screening and new patient forms. If it ever deploys. An “underserved” patient may use it out of desperation to get what they need but I would never in a billion years want this for a patient I gave a shit about. I guess there’s the crux. They don’t give a shit. Some PE group will get it and you can talk to it before your virtual visit with your human doctor on Zoom. It’ll be like Elysium. Maybe I’m bitter because I still commute all over town. Maybe that’s what all you software people want, you like WFH so much, let the doctors do WFH too eh? Talk to the chatbot, you guys made it and it’s better than us medical cartel types anyways.
- pineal 3y agoAn important first step. I am early in my practice and I fully expect medicine as I know it today to become unrecognizable within my lifetime. I am optimistic that it will be done carefully and will ultimately tremendously benefit our patients. That being said, I had to laugh, this is one hyperbolic headline. There are of course some caveats. "Few efforts to harness LLMs for medicine have explored whether the systems can emulate a physician’s ability to take a person’s medical history and use it to arrive at a diagnosis. Medical students spend a lot of time training to do just that, says Rodman. “It’s one of the most important and difficult skills to inculcate in physicians.” This does take a lot of training and experience. The challenge in diagnosis isn't about asking a history and integrating the information, it's about effectively encouraging patients to provide necessary information that they might not realize is relevant or know how to articulate. Different patients often require wildly different approaches. Medical literacy does play a role, but a patient that would say, "Hi doctor, I experienced central chest pain accompanied by discomfort in the upper stomach that happened two hours ago" (from pg 33 of the preprint) is not realistic. More likely you get a vague complaint of "heartburn that started a while ago". Similarly: "Currently, I'm not on any prescribed medications". More frequently you get something like "I take a blue one" or "Golly Telly" (Go-Lytely) or "Gabatini" (Gabapentin). I do think an LLM could probably parse these but such idiosyncrasies in a history do compound. And though someone may be prescribed a med and think they take it as prescribed, sometimes it takes a hunch and clinical experience to tease out that in fact the med is not being taken at all as indicated. Moreover, the better bedside manner was assessed via text conversation. I wouldn't quite call that "bedside" manner. I also wonder how such a system will deal with patients that have self-diagnosed themselves and looked up the "right answers" to get what they want -- a difficult reality that takes some parsing to figure out what's real and what's not. Overall though, the preprint, in contrast to the Nature news article/headline, does a better job of discussing these limitations. I congratulate them on excellent work. Thank god LLMs can't do surgery, though I'm sure my time will come as well.