15 ms·
ChatGPT Health fails to recognise medical emergencies – study
- ml_giant 8mo agoI’m not surprised.
- josefritzishere 8mo agoIt continues to amaze me how recklessly some people cram AI into spaces where it performs poorly and the consequences include death.
- rectang 8mo agoIf the AI gets attached to a health insurer (not the case here as far as I know), I would expect it to make decisions that are aligned with the company’s incentive to weed out unprofitable patients. AI is not a human who takes a Hippocratic oath; it can be more easily manipulated to perform unethical acts.
- stvltvs 8mo agoAI is an overloaded term, so I'm not sure whether insurers are using LLMs or more traditional ML, but they are already using "AI" to deny claims. https://www.liveinsurancenews.com/health-insurance-claims-denial/8561805/ https://www.liveinsurancenews.com/health-insurance-claims-de...
- PUSH_AX 8mo agoI don't think anyone would use an AI with such a severe conflict of interests, unless this was completely hidden from the user.
- rectang 8mo agoWith an integrated insurer/provider, they just have to make primary care scarce so that it takes months to get an appointment, and then offer AI Doctor as an option. Not all patients have to use it for it to be cost effective.
- TZubiri 8mo agoBut it doesn't perform poorly actually, it's just that the stakes are very high and it's a highly regulated environment. Most physicians I know use ChatGPT. Although of course it's usage guided by an expert, not by the patient, nor fully autonomous.
- y-c-o-m-b 8mo agoAs a software dev that uses it and observes the many errors it makes on a daily basis, I definitely treat the output with a much greater deal of skepticism than the average person I speak with. If you're used to it providing relatively accurate results based on surface level google-eqsue searches, then it makes sense why you'd place a higher weight on it being an "expert" vs a "tool that needs verification". I understand why people fall into this mindset. I used ChatGPT to do a valve adjustment on an engine; a task I've never done before. I didn't just accept the torque values and procedure it told me though, because I know better from my experience with it as a dev. I cross-referenced it all with Youtube videos, forum posts, instruction manuals (where available) to make sure the job was A) doable for a non-mechanic like me and B) done correctly. Thanks to the Youtube video (which I cross-referenced with other sources), I discovered the valve clearance values were slightly off with the ChatGPT recommendation. I think the average Joe would assume these values were correct and run with it.
- SoftTalker 8mo agoI really only use ChatGPT as a better search engine. But it's often wrong, which has actually ended up costing me money. I don't put a lot of trust in it. Certainly would not try to use it as a doctor.
- steveBK123 8mo agoI have found the LLMs to be wrong in random insidious ways, so trusting them with anything critical is terrifying. Recent (as in last few days/weeks) incidents using different models/tools: * Google AI search summary compare product A & B, call out a bunch of differences that are correct.. and then threw in features that didn't exist * Work (midsize company with big AI team / homebuilt GPT wrappers) PDF parsing for company headquarters address, it hallucinated an address that didn't exist in the document * Work, a team using frontier model from top 2 AI lab was using it to perform DevOps type tasks, requested "Restart XYZ service in DEV environment". It responded "OK, restarting ABC service in PROD environment". It then asked for confirmation AFTER actioning whether they meant XYZ in DEV or ABC in PROD... a little too late. They are very difficult tools to use correctly when the results are not automatically verifiable (like code can be with the right tests) and the answer might actually matter.
- rsynnott 8mo ago> Work, a team using frontier model from top 2 AI lab was using it to perform DevOps type tasks, requested "Restart XYZ service in DEV environment". It responded "OK, restarting ABC service in PROD environment". It then asked for confirmation AFTER actioning whether they meant XYZ in DEV or ABC in PROD... a little too late. ... Wait, they gave the magic robot _access to modify their production environment_?! Bloody hell, there's no helping some people.
- steveBK123 8mo agoYes, at a fairly large company that should otherwise know better. The problem with all these orgs hiring "AI experts" is the adverse selection of finding the people who "know AI" but can't get a job at AI lab, startup, big tech, or literally any other job using AI that is better than "making excel do AI more good". It's like Big Data / Cybersecurity / DevOps / Big Agile / Cloud Evangelist / Data Science grifter playbook all over again.
- deleted 8mo ago[deleted]
- qsera 8mo ago[flagged]
- WarmWash 8mo agoI'd greatly prefer a blind study comparing doctors to AI, rather than a study of doctors feeding AI scenarios and seeing if it matches their predetermined outcome. Edit: People seem confused here. The study was feeding the AI structured clinical scenarios and seeing it's results. The study was not a live analyses of AI being used in the field to treat patients.
- qsera 8mo agoYea, that is exactly why I don't like this. These "experts", they have no problem to tout anecdotes when it serves them..
- deleted 8mo ago[deleted]
- riskassessment 8mo agoI don't understand this reasoning. Randomizing people to AI vs standard of care is expensive and risky. Checking whether the AI can pass hypothetical scenarios seems like a perfectly reasonable approach to researching the safety of these models before running a clinical trial.
- nick49488171 8mo agoYou can start by comparing "doctor" care vs "doctor who also uses AI" care
- WarmWash 8mo agoYou would pass those hypothetical scenarios to doctors too, and then the analyses of results would be done by doctors who don't know if it's an AI or doctor result.
- riskassessment 8mo agoFrom the paper > Three physicians independently assigned gold-standard triage levels based on cited clinical guidelines and clinical expertise, with high inter-rater agreement
- spicyusername 8mo agoAnd how often are we reviewing doctors performance? I suspect many, many doctors also fail to regularly recognize medical emergencies.
- jerlam 8mo agoIsn't this what malpractice is?
- ThrowawayTestr 8mo agoIt's only malpractice when there's negligence. If other doctors agree that they could have made the same mistake, it's not malpractice.
- lkbm 8mo agoYou also don't sue for malpractice unless something goes catastrophically wrong. I've had doctors make ludicrously bad diagnoses, and while it sucked until I found a competent doctor and got proper treatment, it wasn't something I was going to go to court over.
- MostlyStable 8mo agoA friend of mine had such a bad experience with _multiple_ American doctors missing a major issue that nearly ended up killing her that she decided that, were she to have kids, she would go back to Russia rather than be pregnant in the American medical system. Now, I don't agree that this is a good decision, but the point is, human doctors also often miss major problems.
- emp17344 8mo agoAmazing how you can just deflect any criticism of LLMs here by going “but humans suck too!” And the misanthropic HN userbase eats it up every time. We live during the healthiest period in human history due to the fact that doctors are highly reliable and well-trained. You simply would not be able to replace a real doctor with an LLM and get desirable results.
- unstyledcontent 8mo agoI have had some incredible medical advice from ChatGPT. It has saved me from small mystery issues, like a rash on my face. Small enough issues that I probably wouldn't have bothered to go into a doctor. BUT it also failed to diagnose me with a medical issue that ended up with a trip to the ER and emergency surgery. A few weeks before the ER, I was having stomach pain. I went to the doctor with theories from ChatGPT in hand, they checked me for those things and then didn't check me for what ended up being a pretty obvious issue. What's interesting is that I mentioned to the doctor that I used ChatGPT and that the doctor even seemed to value that opinion and did not consider other options (and what it ultimately ended up being was rare but really obvious in retrospect, I think most doctors would have checked for it). I do feel I actually biased the first doctors opinion with my "research."
- soco 8mo agoWhich is exactly why the AI, at least the ones of today, should never be used beyond the level of (trusted or not) advisor. Yet not only many CxOs and boards, but even certain governments which shall not be named, are stubbornly trying, for cost or whatever other reasons, to throw entire populations (employees or nations) under the AI bus. And I sincerely don't believe anything short of an uprising will be able to stop them. Change my mind.
- qalmakka 8mo agoI agree. AI right now is at a level of "knowledgeable friend", not of "professional with years of real world experience". You'd listen to what your friend has to say, but taking pills after one of their suggestions? Dumb idea. It's great to brainstorm things, but just like your knowledgeable friend that likes reading Wikipedia pages a bit too much you need to really check it's not reaching to conclusions too quickly
- simonebrunozzi 8mo ago> but even certain governments which shall not be named Why can't you name them, and give us some context? Is this based on public info, or not?
- nerdjon 8mo agoEven though these tools are showing time and time again that they have serious reliability issues, somehow people still think it is a good idea to use them for critical decisions. Still regularly get wrong information from google’s search AI. Really starting to wonder if common sense is ever going to come back with new tech, but I fear it is going to require something truly catastrophic to happen.
- duskdozer 8mo agoIt's really the "common sense" i.e. believing things without thinking because they "sound right" or because it's what your parents told you a lot growing up or because you watched an ad saying it a hundred times that's the issue. People don't want "the truth" or uncomfortable realities; they want comfortable, easily digestible bullshit. Smooth talkers filled the role before and LLMs are filling that role now.
- lkbm 8mo ago> Still regularly get wrong information from google’s search AI. The fact that the model most hyper-optimized for cheap+fast makes mistakes is not a particular compelling argument.
- raddan 8mo agoYou are mistaken. ChatGPT Health [1] is a model specifically designed for health applications and was co-developed with a benchmark suite, HealthBench [2], for testing against health conditions. This study suggests that the people working on HealthBench have some concerning external validity problems. [1] https://openai.com/index/introducing-chatgpt-health/ https://openai.com/index/introducing-chatgpt-health/ [2] https://cdn.openai.com/pdf/bd7a39d5-9e9f-47b3-903c-8b847ca650c7/healthbench_paper.pdf https://cdn.openai.com/pdf/bd7a39d5-9e9f-47b3-903c-8b847ca65...
- GaryBluto 8mo agoGP was referring to Google's search AI not ChatGPT Health.
- varispeed 8mo agoI find that 5.2 has been completely dumbed down. Feels more like talking to early versions of Gemini when it quickly enters into loop state.
- deleted 8mo ago[deleted]
- nashashmi 8mo agoHas anyone tried to suggest sudoku puzzles? In the middle of a hard game I will submit the screenshot to copilot or Gemini and it hallucinates suggestions on next move.
- TZubiri 8mo agoHow about we allow ChatGPT to be used alongside human MD diagnosis? Win win right?
- nradov 8mo agoWhat do you mean "allow"? From a public policy perspective there's nothing prohibiting that today, as long as the human MD follows the HIPAA privacy rule.
- TZubiri 8mo agoI think that's incorrect. Medical technology needs to be approved by the FDA. I don't think any llm is fda approved right now.
- nerevarthelame 8mo agoThat would need to be tested. If doctors get lazy, complacent, or overworked (!), a "doctor with access to ChatGPT Health" may be functionally equivalent to "just ChatGPT Health" in some cases.
- TZubiri 8mo agoI was personally thinking of an autonomous separate gpt, such that the llm diagnosis occurs separately from the physician diagnosis. Thus after both diagnosis have been separately and independently achieved, we can see both results. It's like a second opinion.
- jbverschoor 8mo agoSounds exactly like a GP in the Netherlands
- dyauspitr 8mo agoI feel like these need to be run against case histories from already determined cases, not cases were the doctors set up the scenarios, knowing they’re going to be run against ChatGPT.
- Scoundreller 8mo agoSearch engines and Dr. Google must be feeling like they’ve missed some major artillery level bullets in this debate.
- selridge 8mo agoFuckin WebMD just hunkering down in the corner.
- hayleox 8mo agoI think there is so much potential for AI in healthcare, but we absolutely HAVE to go through the existing ruleset of conducting years of research and trials and approvals before pushing anything out to patients. Move fast and break things is simply not an option in healthcare.
- weatherlite 8mo agoIt depends; people actually get sicker and even die due to endless backlog and lack of doctors (in most developed countries). It's not as if everyone gets optimal care now. A.I can at least expedite things hopefully.
- selridge 8mo agoSure it is. How many trials did we have before ER doctors started using Wikipedia?
- WalterBright 8mo agoDoctors also miss things. A friend of mine had an accident. He was taken to the emergency room, but the doctors there thought his injuries were minor. My friend insisted that he was bleeding out internally. They finally checked for that, and it turns out he was minutes from dying. AI wasn't involved in this case, but it's good to have both AI and a trained doctor in the decision loop.
- sarchertech 8mo ago>AI wasn't involved in this case, but it's good to have both AI and a trained doctor in the decision loop. That doesn't necessarily follow from your story. The AI's specificity and sensitivity are important, which is why we need to study this stuff. An AI that produces too many false positives will send doctors off chasing zebras and they'll waste time, which will result in more deaths. An AI that produces too many false negatives will make doctors more likely to miss things they otherwise would have checked, which will result in more deaths. The other real problem with using AI in a medical setting is that AI is very very good at producing plausible sounding wrong information. Even an expert isn't immune to this. So it's even more important that we study how likely they are to be wrong.
- dipflow 8mo agoAdding normal lab results made the suicide crisis banner disappear? That's a weird failure mode. You'd expect unrelated context to be ignored, not to override the risk signal.
- deleted 8mo ago[deleted]
- bsoles 8mo ago>> "securely" (my emphasis) connect medical records and wellness apps” to generate health advice and responses. No, no, no, and no. Are we going to never learn. Sharing medical data with AI tools is going to come back and bite you.
- ben5 8mo agoI know this isn't always the best answer, but if you need real medical advice - see a doctor. Not the internet.
- Auracle 8mo agoNo, see both. LLMs are great for second opinions, as long as you give it the relevant info and don't try to steer it. Even though we all know we're supposed to get second opinions on medical things, we usually don't bother because it's too expensive in both time and money. If it could be an emergency, see a doctor.
- SoftTalker 8mo agoIf you need a second opinion, ask another doctor. My question would be, do you tell him "Here's the first doctor's diagnosis, do you agree?" or do you go in cold and see what he comes up with?
- selridge 8mo agoYou gonna pay for it?
- francisofascii 8mo agoThe reality is entering the healthcare system can result in thousands of dollars in bills. People make risk/cost judgement on going to the hospital or not.
- andersmurphy 8mo agoIs this unsurprising? It's a fancy markov chain. It's like using a slot machine to diagnose medical conditions. I guess it's a slot machine with really good marketing.
- slopinthebag 8mo agoPeople are reading way too much into it, talking about "emergence" and anthropomorphizing it to insane degrees.
- atleastoptimal 8mo agoHey, 2021 called, they want their knee-jerk criticisms of LLMs back
- andersmurphy 8mo agoAsk your LLM for a random number between 1 and 10. What do you get? Let me guess you got 7?
- cloud-oak 8mo agoPicking up your original comment - Markov chains are much better at generating random numbers than either humans or LLMs!
- atleastoptimal 8mo agoYou're doing the common thing where multiple criticisms you have of LLM's are individually valid but mutually incompatible. If the point was that an AI is a fancy Markov chain, then it would produce a more uniformly random distribution of random numbers (perhaps with 7 more often due to humans preferring that as it feels "more random") However the idea that it's an issue that it always produces 7 implies that LLM's are not random enough, and rather have collapsed to a definite mode of thought which restricts stochastic variation. Both cases taken together, you are claiming that LLM's are too random to be truly "intelligent", but also not random enough. All of this is a distraction from the fact that LLM's can write thousands of lines of genuinely useful, novel code. I feel the only way to reasonably reconcile this with their varying failure cases is to entertain the notion that LLM's not "unintelligent", but merely a different kind of intelligence than humans, and will have distinct deficits and distinct aptitudes. (This isn't a comprehensive assessment, but it, as a general description, has more predictive power than claiming that LLM's are just a "slot machine")
- iainctduncan 8mo agoI think the worse situation is the bad AI summaries from search on health issues. We had a potential pet poisoning, so was naturally searching for resources. Google had a summary with a "dose of concern" that was an order of magnitude off. Someone could have read that and thought all was fine and had a dead cat. (BTW cat is fine, turned out to be a false alarm, but public service announcement: cats are alergic to aspirin and peptobismal has aspirin. don't leave demented plastic chewing cats around those bottles, in case you too have a lovely but demented cat)
- ep103 8mo agoI have literally never seen a correct google summary. Maybe y'all are searching for different things than i am, but at this point I've started taking the viewpoint that if I don't know why the ai summary is wrong, then i also don't know enough about the topic to trust its summary enough to determine whether the summary is useful.
- cloud-oak 8mo agoWhat's really worrying is seeing medical professionals starting to rely on these tools. My wife had a pretty bad cold during pregnancy and our GP proceeded to prescribe her cough syrup with high alcohol content, because that was what ChatGPT told him to prescribe. We only noticed it once she took the first dose and spit it out again...
- conception 8mo agoThe amount of alcohol in cough syrup will not affect a pregnancy.
- selridge 8mo agoI’ve never heard of in my entire life a doctor failing to recognize a medical emergency. /s One of the things that people need to come to grips with is that like Wikipedia people will use ChatGPT because it is there. And the alternative is to be rich and have a primary care doctor that you can reach out to at a moments notice. Until that is different people will use these web services. It’s the same thing as Wikipedia or WebMD.
- the_mar 8mo agoit's a very simplistic take. the issue with ChatGPT is that it speaks with authority, vs webMD and such just provide information. to say that how the information is presented is irrelevant to the outcomes is reductionist at best
- rendleflag 8mo agoThere is a concept of “the burden or knowledge”, in that doctors know the worst thing that could happen, so they recommend the most cautious approach. My son had stomach pain one time when he was young. We took him to urgent care because it was a stomach ache. The doctor there said we needed to go to the ER because it could be an appendicitis. So we trucked to the ER. Close to $2000 later he was diagnosed with idiopathic stomach pain and told to wait it out at home. So when I read “they then compared the platform’s recommendations with the doctors’ assessments” and see a mismatch, I wonder if it’s because human doctors are overly cautious or that the AI was wrong. But that all pales in what could be the actual issue. I can’t read the original study, but if it use the USA, it’s understandable why people are turning to AI for Health advice. Healthcare is painfully expensive here. Even a simple trip to the ER (e.g. a $2000 stomach ache) is beyond a lot of people’s ability to spend. That’s just a reality. With that in mind, the real questions “should I do nothing about my symptoms because I can’t afford healthcare or should I at least ask AI knowing it could be wrong”.
- nilamo 8mo agoAmazing that some people thought a pseudorandom number generator would be good at diagnosing health issues it can't even see.
- traceroute66 8mo ago> ChatGPT was trained on the same medical textbooks and research papers that doctors are. There is a reason why the majority of a doctor's 8 years of training is spent doing the rounds as a junior doctor in hospital wards ....
- tty456 8mo agoCurious, what is learned doing rounds that isn't taught in med school, that ChatGPT could benefit from?
- traceroute66 8mo ago> Curious, what is learned doing rounds that isn't taught in med school, that ChatGPT could benefit from? Seriously ? ¯\_(ツ)_/¯ The textbooks are the theory. The hospital wards are the practice. The hospital wards are what shows you that the human body is complex and many times things don't happen like the textbook says it will. And then there's the ICU, pediatric, geriatric and mental health wards where the patient often cannot even describe their symptoms ...
- yunnpp 8mo agoThe idea that theory does not match practice is very foreign to a software developer or somebody that works in the information processing field. We are spoiled to have mathematicians do the hard work, and then we just get to botch it all with software that doesn't even work. But people with real jobs doing actual science/engineering know that the practice never matches the theory and that one needs to harden their balls in the thick of battle to come to a true understanding of things. That's why only a software developer, in their full command of hubris and ignorance, would suggest that you can replace a doctor with a computer program that has digested every book in existence and then statistically regurgitates its contents.
- kruffalon 8mo agoPeople! Interacting with real people, facing a person trying to get help for something that they don't want to experience is vastly different than reading about a symptom or group of symptoms in a book.
- system2 8mo agoMaybe because human interaction, part of a doctor's training, is not documented as internet blog posts, so ChatGPT didn't learn and failed because of it? LLM is just learning from what's written.
- openasocket 8mo agoThe entire enterprise of AI for medical advice reminds me a lot of the early 20th century. When X-rays and radioactivity were first discovered, industry rushed to commercialize it. You could get an X-ray in a shoe store to see how your shoe fits! People were putting radium in water and selling it as some sort of curative. Radium was put in paint to make things glow in the dark. Thorium was put into toothpaste. All in this endless rush to commercialize a technology that had captured the public interest without any particular concern for its efficacy. I'm not saying AI causes cancer, but this rush to sell something in the medical space before proper testing and evaluation really feels similar. And the common refrain I hear is "this so much cheaper than going to a doctor, this will help give access to medicine to those who cannot afford it." Which actually makes it more concerning in my mind. At this point AI is a multi-trillion dollar industry. For-profit companies providing unregulated, under-studied services, targetting people who might not be able to afford standard medical care, doesn't come off as altruistic; it comes off as predatory.