17 ms·
OpenAI’s o1 correctly diagnosed 67% of ER patients vs. 50-55% by triage doctors
- llbbdd 5mo agoCan't happen soon enough. If the bar was as high as it needed to be, there'd be like one qualified doctor on Earth so far.
- beering 5mo agoo1 is several generations old and was released in 2024. Is this some quite old research that took a long time to get published?
- SpicyLemonZest 5mo agoYes, the preprint of the same paper (https://arxiv.org/abs/2412.10849 https://arxiv.org/abs/2412.10849) was first written in December 2024.
- nhinck2 5mo agoIt's also important to note that it beat doctors in diagnosing in a way doctors do not diagnose.
- oofbey 5mo agoMedical research moves. Very. Slowly.
- bluefirebrand 5mo agoThat's a good thing The medical equivalent to "move fast and break things" would be "move fast and kill people"
- oofbey 5mo agoPeople die either way. People die because treatments sit in the lab for years instead of in clinics. Consider the FLASH oncology treatment. In 1995 Dr Fauvadon figure out how to make radiation therapy much much safer. He spent 14 years before sharing it in 2009. In 2014 it was actually published. [1] Know anybody who died of cancer in those DECADES? [1] https://institut-curie.org/news/dr-vincent-favaudon-receives-estro-2025-lifetime-achievement-award https://institut-curie.org/news/dr-vincent-favaudon-receives...
- aurareturn 5mo agoIt's hard to draw any conclusion from this study precisely because of this. Since 2024, we went from AI being able to do a few minutes of coding work to now a few weeks autonomously. That's like going from an intern to staff engineer level.
- Bender 5mo agoHumans could not diagnose and treat me correctly. They almost killed me. Curious where I could feed my symptoms and the same data I gave to an ER to an AI to test it.
- causal 5mo agoChatgpt.com?
- deleted 5mo ago[deleted]
- Bender 5mo agoAll the AI's are able to guess what is going on based on what information I gave the ER. I was under the impression that there is a different interface that does not redirect people to a real doctor and will try to act like a doctor which AI does not.
- jacekm 5mo agohttps://aistudio.google.com/ https://aistudio.google.com/
- jmpman 5mo agoBesides for myself and wife, I've also used LLMs to diagnose my dogs. Convinced there's a huge opportunity for AI based veterinary, especially one which then performs bidding across the local veterinary clinics to perform the care/surgeries. I've noticed that local vets vary in price by more than an order of magnitude. My 80 year old mother and mother inlaw have been regularly scammed by over charging vets, and with their dogs being a major part of their lives, they extremely susceptible to pressure.
- contagiousflow 5mo agoWhat makes you think that LLM vet companies wouldn't bend to the same forces of "over charging"
- NiloCK 5mo agoThe general trend is that cost of entry in a lot of domains is collapsing. Every sniffed out systematic service overcharge can be aggressively undercut by competition. "Your margin is my opportunity", etc.
- contagiousflow 5mo agoIn this race to the bottom, how is quality compared between vendors by consumers? What are the market forces that will cause thousands of virtual vet vendors that give incorrect information, to beat out the vendors that provide quality work?
- Aboutplants 5mo agoNow show me the result of Triage Doctors with aided AI help
- bluefirebrand 5mo agoUnfortunately, from my understanding Doctors don't necessarily diagnose for accuracy, they often diagnose to limit liability. They aren't going to take a stab at an uncommon diagnosis even if it occurs to them, if they might get sued if they're wrong. Edit: I'm not trying to say Doctors deliberately diagnose wrong. Just that if there are two possible diagnoses, one common that matches some of the symptoms and one rare that matches all symptoms, doctors are still much more likely to diagnose the common one. Hoofbeats, horses, zebras, etc
- taurath 5mo agoI’d love to see a follow to that radiologist evaluation, where it failed so miserably on the thing it was supposed to be the best at that now there’s a shortage of radiologists.
- gpm 5mo agoI'd be very very hesitant to trust studies like this. It's very easy to mess up these benchmarks. See for example this recent paper where AI managed to beat radiologists on interpreting x-rays... when the AI didn't even have access to the x-rays: https://arxiv.org/pdf/2603.21687 https://arxiv.org/pdf/2603.21687 (on a pre existing "large scale visual question answering benchmark for generalist chest x-ray understanding" that wasn't intentionally messed up). And in interpreting x-ray's human radiologists actually do just look at the x-rays. In the context the article is discussing the human doctors don't just look at the notes to diagnose the ER patient. You're asking them to perform a task that isn't necessary, that they aren't experienced in, or trained in, and then saying "the AI outperforms them". Even if the notes aren't accidentally giving away the answer through some weird side channel, that's not that surprising. Which isn't to say that I think the study is either definitely wrong, or intentionally deceptive. Just that I wouldn't draw strong conclusions from a single study here.
- pixel_popping 5mo agoI agree with you on this specific study, however, I can't really wrap my head about the fact that doctors will be better than AI models on the long-run. After all, medicine is all about knowledge, experience and intelligence (maybe "pattern recognition"), all those, we must assume that the best AI models (especially ones focusing solely in the medical field) would largely beat large majority of humans (aka doctors), if we already have this assumption for software engineers, we should have it for this field as well, and let's be realistic, each time I've seen a doc the last few months (and ER twice), each time they were using ChatGPT btw (not kidding, it chocked me). So I’m genuinely curious: What is the specific capability (or combination of capabilities) that people believe will remain permanently (or at least for decades) where a top medical AI cannot match or exceed the performance of a good human doctor? Let's put liability and ethics aside, let's be purely objective about it.
- nkrisc 5mo ago> What is the specific capability (or combination of capabilities) that people believe will remain permanently (or at least for decades) where a top medical AI cannot match or exceed the performance of a good human doctor? Let's put liability and ethics aside, let's be purely objective about it. Being a human when a patient is experiencing what is potentially one of the worst moments of their life. AI could be a tool doctors use, but let’s not dehumanize health care further, it is one of the most human professions that crosses about every division you can think of. I would not want to receive a cancer diagnosis from a fucking AI doctor.
- SkiFreeWin3 5mo agoYes, but what was the overlap
- wg0 5mo agoThe Guardian needs to raise their bar on what to report and how to give readers full context on the ongoing NFT AI trust me bro crypto scam and that context would be that it is a mathematical model of human language and not medical expert or replacement for one.
- sigmar 5mo ago>The Guardian needs to raise their bar on what to report and how to give readers full context Should they not report on peer reviewed articles published in Science? or only report published articles that fit your priors?
- wg0 5mo agoFair enough. But there's lot of faulty and wrong peer reviewed research as well. One such paper comes to mind which is probably cited some 7000+ times in other papers but itself is wrong.
- tene80i 5mo agoIt’s a peer reviewed study in one of the world’s top science journals. It’s not some random person on a podcast.
- pixel_popping 5mo agoSo we can eventually classify AI models as Software experts, but not as Medical experts, why so?
- jval43 5mo agoWe can't. It's just that everyone and their dog has an interest in selling you that lie because money. Stochastic parrots can code yes, but that does not make them experts. Don't trust them with your life.
- wg0 5mo agoI don't classify them as software experts either. Anyone doing so is probably not an expert themselves. I take them as those code generation command line tools like create react app and such.
- LeCompteSftware 5mo agoIt is easy to overinterpret this based on the headline, the doctors were actually at a slight disadvantage. This isn't how they normally work, this is a little more like a med school pop quiz: An AI and a pair of human doctors were each given the same standard electronic health record to read – typically including vital sign data, demographic information and a few sentences from a nurse about why the patient was there. The AI identified the exact or very close diagnosis in 67% of cases, beating the human doctors, who were right only 50%-55% of the time.... The study only tested humans against AIs looking at patient data that can be communicated via text. The AI’s reading of signals, such as the patient’s level of distress and their visual appearance, were not tested. That means the AI was performing more like a clinician producing a second opinion based on paperwork. "I don't know, let's run more tests" is also a very important ability of doctors that was apparently not tested here. In addition to all the normal methodological problems with overinterpreting results in AI/LLMs/ML/etc. Sadly I do think part of the problem here is cynical (even maniacal) careerist doctors who really shouldn't be working at hospitals. This means that even though I am generally quite anti-LLM, and really don't like the idea of patients interacting with them directly, I am a little optimistic about these being sanity/laziness checkers for health professionals.
- bux93 5mo agoAlso, this is not how ER doctors work? They are not trained for this, nor does it reflect their day-to-day performance. If they would work like this, perhaps they would know a bit more about the nurse writing down those notes, and the kinds of things that particular nurse is likely to miss or overemphasize - just as an example. The article gives a neat example: In one case in the Harvard study, a patient presented with a blood clot to the lungs and worsening symptoms. Human doctors thought the anti-coagulants were failing, but the AI noticed something the humans did not: the patient’s history of lupus meant this might be causing the inflammation of the lungs. The AI was proved correct. Which is nice and all, but in the presence of a blood clot, I can understand that treating inflammation instead is not the first thing on a doctor's mind, what with blood clots being potentially life threatening and all. It raises the question; was this a real-life case, and what happened to that patient? Since this is a case for which the correct diagnosis is known, it was eventually correctly diagnosed - presumably then the patient did not die of a blood clot, nor of an uncontrollable fever. Also, how representative is a patient with Lupus? According to House, MD, it's never Lupus.
- SilverElfin 5mo agoI’ve had much better luck with diagnosis of my own family’s issues than with doctors. Usually now, I’m feeding them more information to begin with, so that their 30 minute office visits are not wasted, requiring another expensive follow up appointment. While I’m sure there can be ways in which such studies are wrong, it’s very obvious that AI can accelerate work in many of these areas where we seek out professional help - doctors, lawyers, etc.
- kakacik 5mo agoIt can speed up some aspects of work, but please don't trust some llm with variable quality of output more than professional. If you don't like current doctor try another, most are in the business of helping other people. If you have string of issues with 10 last doctors though, then issue is, most probably, you... My wife is a GP, and easily 1/3 of her patients have also some minor-but-visible mental issue. 1-2 out of 10 scale. Makes them still functional in society but... often very hard to be around with. That doesn't mean I don't trust your words, there are tons of people with either rare issues or even fairly common ones but manifesting in non-standard way (or mixed with some other issue). These folks suffer a lot to find a doctor who doesn't bunch them up in some general state with generic treatment. There are those, but not that often. It helps both sides tremendously if patient is not above or arrogant know-it-all waving with chatgpt into doctor's face and basically just coming for prescription after self-diagnosis. Then, help is sometimes proportional to situation and lawful obligations.
- llbbdd 5mo agoRespectfully, as someone with a family with plenty of medical issues and having experienced plenty of useless doctors, the onus is now on medical professionals to prove their worth. They are a second option and most of their remaining value is in the license to prescribe medication, after being told by laymen what medication is appropriate. They're using the same tools I am and they're worse at evaluating them. Doctors thinking patients are arrogant is an age old problem.
- aduwah 5mo agoIt makes me so upset when anyone even tries to defend the GPs. I admittedly I have a bunch of medical issues and these gems are my favourites from the GPs. 1. I cannot see the tonsil on the left side, so it is OK. (there was a 6cm!!! cyst in front of it) 2. After missing sky high TSH measures consistently for 2 years (4 testst) : "It must have been a few one offs" (no it wasn't and it is not even possible) 3. "Blood pressure has nothing to do with weight" These %#£&* so called medical professionals are still working and most likely killing people legally. These days I research and read studies, arm myself with knowledge, cross check with multiple LLMs and go in with a diagnosis and request a specific prescription. After 5 years with my health in the gutter I had my first comprehensive private blood test coming back with no issues. So no, do not try to call me arrogant. I am not arrogant, I am defending myself from these "GPs" so they won't put me in an early grave by making fatal mistakes.
- SpyCoder77 5mo agoThis is a rather new article about an old model...
- sigmar 5mo agoStudy design, data collection, analysis, and peer review take time. O1 came out a little over 1.5 years ago
- cubefox 5mo agoAt this point the study is already mostly irrelevant because the model in question has long been far surpassed by new models. It seems traditional publishing doesn't work for really fast moving fields.
- basyt 5mo agoi would rather be incorrectly diagnosed by a doctor than have chudgpt treat me.
- xaxfixho 5mo ago[dead]
- journal 5mo agowould it ever diagnose incorrectly to save more lives? kinda weird an ai would decide who die so others may survive, but i guess whatever.
- HWR_14 5mo agoNot only should AI misdiagnose to save lives, but a human should too. You walk in with symptoms that most likely is a harmless virus that clears up on its own or 5% of the time is a deadly bacteria. The correct course of action is to try to test if it is the 5% case (most often the wrong diagnosis), not send people home because they are most likely fine. Many cases have a similar low but not 0 risky diagnosis.
- Noahxel 5mo ago[flagged]
- theshrike79 5mo agoI'll repeat my idea on how this MUST be done: 1. AI gets data about the patient and makes a diagnosis. This is NOT shown to doctor yet. 2. Doctor does their stuff, writes down their diagnosis. This diagnosis is locked down and versioned. 3. Doctor sees AI's diagnosis 4. Doctor can adjust their diagnosis, BUT the original stays in the system. This way the AI stays as the assistant and won't affect the doctor's decision, but they can change their mind after getting the extra data.
- stuxnet79 5mo ago5. Private Equity uses this valuable data to stack rank doctors based on how correct / AI-aligned their diagnoses are over time 6. Rankings are used to periodically "trim the fact" thus delivering more optimized cash flows to clinics that have been saddled with toxic debt 7. Sensing an opportunity AI providers start selling a $200 / month Data Leakage as a Service subscription to overworked physicians so that they can avoid the PE guillotine
- fc417fc802 5mo agoA more realistic step 7 is that physicians gradually align their diagnoses with the LLM as they sacrifice to Moloch in order to (temporarily) game the metric. Eventually the humans become little more than an imperfect proxy for the LLMs and are eliminated. I agree with GP's solution but we'd need regulation to prohibit what you describe.
- cindyllm 5mo ago[dead]
- avidiax 5mo agoWhy would private equity want more competent doctors? Incompetent ones order unnecessary tests and exhaust treatment possibilities, which drives up cost billed to insurance. Only the insurance industry and perhaps licensing bodies can pressure to keep the quality floor high, at least in terms of accurate diagnosis and prevention of overtreatment.
- droidjj 5mo agoThe paper: https://www.science.org/doi/10.1126/science.adz4433 https://www.science.org/doi/10.1126/science.adz4433 (April 30, 2026)
- colechristensen 5mo agoI think this is more a commentary on how bad ER diagnosis is.
- davycro 5mo agoThe emergency room should be good at diagnosing emergencies, but most ailments aren’t.
- creativeSlumber 5mo ago> "An AI and a pair of human doctors were each given the same standard electronic health record to read" This is handicapping the human doctors abilities. There is a lot more information a human doctor can gather even with a brief observation of the patient.
- jrm4 5mo agoThis feels like a deeply important observation. Now also, would be interesting to include e.g. a short video or photograph for the AI to use as well.
- cogman10 5mo agoAgreed. I think the best use of this sort of tech is to use both to their strengths. Use AI to go over the record and suggest diagnoses which you have the doctor review after observing the patient. The other thing is that common issues are common. I have to wonder how much that ultimately biases both the doctor and the LLM. If you diagnose someone that comes in with a runny nose and cough as having the flu you will likely be right most of the time.
- kqr 5mo agoOn the other hand, > there are few things as dangerous as an expert with access to open-ended data that can be interpreted wildly, like a clinical interview. https://entropicthoughts.com/arithmetic-models-better-than-you-think https://entropicthoughts.com/arithmetic-models-better-than-y...
- deleted 5mo ago[deleted]
- deleted 5mo ago[deleted]
- delfinom 5mo agoBonus, health networks now push doctors to use AI transcription software for the EHR entries. Doctors and nurses like it because they don't have to type it up. But it is a complete shitshow on whether the records are reviewed for transcription errors which happen quite often Now feed a flawed transcripted into an AI diagnosis system and bam-o. The AI will treat it as gospel, while the doctor may go wait what.
- OptionOfT 5mo agoAs a 37 year old male with 2 THRs I'm glad the AI was NOT used in my diagnosis. All the models that I used to look at my x-rays said nothing was wrong, even when adding symptoms. When adding age it said the patient was too young. (I was ~3 months away from wheelchair bound in those x-rays). The worst one was Gemini. Upload an x-ray of just the right hip, and it started to talk about how good the left hip looked like. I think with AI taking over it's gonna be harder to get a solution when your problem isn't the run-of-the mill.
- jeffbee 5mo agoAll versions and levels of Gemini have terrible spatial reasoning. I don't know why. That kind of task seems to be simply outside of the abilities of the model.
- cyberax 5mo agoThe general AI models are useless if you need precision. They are designed to create/analyze pretty pictures. But specialized models can be inhumanly good. I know, our main product is a model that does _precise_ analysis :)
- xaxfixho 5mo ago[dead]
- lukko 5mo agoI'm surprised at both the article and the paper - both seem very hyperbolic. This is LLMs competing against doctors in a way that is heavily weighted in the LLMs favour, which does not represent clinical practice. These reasoning cases are not benchmarks for doctors, they are learning tools. I think it's important to note that diagnosis also relies on accurate description of the patient in the first place, and the information you gather depends on the differential diagnosis. Part of the skill of being a doctor is gathering information from lots of different sources, and trying to filter out what is important. This may be from the patient, who may not be able to communicate clearly or may be non verbal, carers and next of kin. History-taking is a skill in itself, as well as examination. Here those data are given. For pattern recognition from plain text, especially on questions that may be in the o1's training data, I'm not surprised at all that it would outperform doctors, but it doesn't seem to be a clinically useful comparison. Deciding which investigations to do, any imaging, and filtering out unnecessary information from the history is a skill in itself, and can't really be separated from forming the diagnosis.
- lokar 5mo agoAlso, you need to see an analysis of the incorrect calls. The goal of a human Dr is not to get the highest accuracy, it's to limit total harm to the patient. There can be cases where the odds favor picking X (but it may not be by that much), but the safe thing to do is to rule out some other option first, or start a safe treatment that covers several other possible options. Simply getting the "high score" on this evaluation is not necessarily good medical treatment.
- lukah 5mo agoExactly this. Most diagnosis isn’t about pinpointing the underlying exact cause, it’s ruling out the really bad stuff and minimising harm. Differential diagnosis just isn’t real world medicine.
- IshKebab 5mo agoYeah 100% this. We've all used AI. It's obvious that it can sometimes outperform humans in a "did it get the right answer" benchmark while being wildly worse overall because of worse failure modes. I bet the AI's incorrect answers are less "I don't know, let's get a second opinion" and more "you're perfectly fine, 0% chance this is cancer".
- mawadev 5mo agoI don't think AI is a good use case for such critical situations. Maybe in a decade we have AI help out doctors with doing a pre check. What if Ai finds nothing and the doctor does not bother to look into it further? It is this small question which breaks the technology from any angle later down the road from my POV. AI has to stay optional here. Even if AI is used to sample or summarize a lot of data that a human couldn't do in time: What if it misses something that a human won't? What if a human inversely misses something that AI won't? Would you rather trust the machine or the human? (Especially if the human is held accountable.)
- henry2023 5mo agoYou can replace AI with blood tests in you comment and the same questions are relevant today.
- chromacity 5mo agoAll the other points raised in this thread aside, it seems like an odd thing to benchmark because a significant proportion of ER practice is dealing with emergencies, often accidental injuries. There's not a whole of diagnosing going on if you show up to ER with a gash on your forehead or a missing finger.
- tedggh 5mo agoBelievable and not shocking. LLMs literally may have saved my sons and potentially her mother too by allowing us to fact check a lot of non sense data and scare tactics by a group of at least 5 different doctors ambushing us to make a life changing decision in minutes. The problem is doctors, at least in the US, prioritize liability exposure over patients long term outcomes. Let’s say you need an intervention where two options A and B are available to you. A carries 1% risk of complications but a great outcome. Option B has 0.1% risk of complications but once you are discharged the short term effects are challenging and long term effects not well understood. Well, 10/10 times doctors will suggest option B and will do anything they can to nudge you into making that choice, like not telling you the absolute numbers and constantly using the word “death”. They also lie about the outcomes, because again, once you accept the procedure, sign and are sent home, they have nothing to do with you.
- voxl 5mo agoNeedless conspiracy bullshit without sharing specifics
- wiseowise 5mo agoThe Pitt third season leak? All of the ER is fired and Robbie is fighting schizophrenia with 15 agents and Dana?
- Lihh27 5mo agoradiology already had its "AI beats doctors" moment. radiologists are still here. what changed first was the workflow, not the specialty. er is probably next.
- husarcik 5mo agoI don't think radiology has had that moment at all. Computer programming is much closer, if not, at that moment right now.
- Madmallard 5mo agono programming it's still just tool use for CRUD applications with react and tailwind complex systems programming is just so unreliable and foolish to use LLMs to do anything important companies adopting it for more safety critical systems are just already seeing the problems pile on and we're seeing news about it almost every day on Hacker News If the tool can make something look smart but isn't necessarily correct, lazy employed humans will just defer to it, especially when their lazy greedy bosses tell them to, and everybody loses over time (except the stakeholders that just jump companies anyway after they made their money) It's just sad to see these really unwise and inexperienced sentiments repeated ad nauseam
- jmcgough 5mo agoLLMs can be a useful second opinion for a highly educated patient with good insight into their health and body, but this is not the average patient I see in an urban emergency department. Many patients can't give a cohesive history without a skilled clinician who can ask the right questions and read between the lines. I am very skeptical of studies like this that don't adequately reflect real world conditions, but when I was a software engineer I probably wouldn't have understood what "real" medicine is like either.
- matheusmoreira 5mo agoYou went from software to medicine? Pretty cool to discover I'm not alone in this world. > LLMs can be a useful second opinion for a highly educated patient with good insight into their health and body I have the same opinion. It's just like software in this regard. A person who's already knowledgeable can prompt well and give detailed context, and tell when the LLM is confidently bullshitting or just plain being lazy. That is not the reality of the average person. I tried using Claude to help with some hard cases a couple of times and it was very prone to jumping to conclusions based on incomplete information. It was excellent as a research buddy though. I'm using it to great effect to keep myself up to date.
- lvl155 5mo agoI’ve some family in medicine and it scares me how much they now rely on AI. Some even quote it like Bible.
- swisniewski 5mo agoLet’s assume the AI does out perform the DR. I still want humans in the loop, interpreting the LLMs findings and providing a sanity check. You can’t hold an LLM accountable. That’s the min responsible bar for LLM authored code, which normally doesn’t really matter much. For something as important as ER diagnostics, having a human in the loop is crucial. The narrative that these tools are replacing human intelligence rather than augmenting it is, quite frankly, stupid. We should embrace these tools. But, “eliminating DRs”… hardly.
- appz3 5mo ago[flagged]
- jmathai 5mo agoI advise a medical non profit and we ran a series of tests against cases doctors input to our system looking for specialist recommendations. Our findings found that gpt-5-mini performed better than gpt-5, sonnet 4 and medgemma. I think these studies are very hard to accurately score. But in any case, AI seems to do a very good job compared to humans. Unsurprising, really.
- afro88 5mo agoI wonder about the nuance within the data. Like does AI do much worse with children than adults, but still better overall for example. Or biological male vs female. I think we'd want it to do better across all groups, ages etc so we're not introducing some kind of horrible bias resulting in deaths or serious health consequences for some groups
- DeepYogurt 5mo agoWho's accountable for the 33%?
- Aurornis 5mo agoGell-Mann Amnesia kicks in hard as soon as the LLM topic changes to a profession other than our own. It’s much easier to believe an LLM can outperform someone else doing their job than to believe that it’s a good idea to replace your own work with an LLM. The number in the headline isn’t even a good comparison because they asked doctors to make a diagnosis from notes a nurse typed up. Doctors are trained to be conservative with diagnosing from someone else’s notes because it’s their job to ask the patient questions and evaluate the situation, whereas an LLM will happily leap to a conclusion and deliver it with high confidence When they allowed both humans and doctors access to more information about the case, the difference between groups collapsed into statistical insignificance: > The diagnosis accuracy of the AI – OpenAI’s o1 reasoning model – rose to 82% when more detail was available, compared with the 70-79% accuracy achieved by the expert humans, though this difference was not statistically significant. Talking to my medical professional friends, LLMs are becoming a supercharged version of Dr. Google and WebMD that fueled a lot of bad patient self-diagnoses in the past. Now patients are using LLMs to try to diagnose themselves and doing it in a way where they start to learn how to lead the LLM to the diagnosis they want, which they can do for a hundred rounds at home before presenting to the doctor and reciting the script and symptoms that worked best to convince the LLM they had a certain condition.
- 01100011 5mo agoI wouldn't put much weight in this study, but I think a lot of us can still attest to the usefulness of LLMs in self-diagnostics. The reality in the US is that it is difficult to get the attention and care of a doctor so we're left having to do it ourselves. 10 years ago you'd hear docs complaining about patients coming in with things they found on google but now I don't think there's an alternative. Case in point, I went to a podiatrist for foot and ankle issues. He diagnosed my foot issues from the xray but just shrugged his shoulders for the ankle issues and said the xray didn't show anything. My 15 minute allocation of his attention expired and I left without a clue as to the issue or what corrective actions to take. 5 minutes with an LLM and I had a plausible reason for the ankle issues which aligned with the diagnosis in my foot.
- NegativeK 5mo agoI don't think that using LLMs for medicine is an appropriate fix for the US's healthcare issues. Unless healthcare businesses decide to improve patient care with AI instead of increasing patients per day, I think it's going to make things even worse.
- 01100011 5mo agoI'm not suggesting it as a fix. I'm saying it's the only option to get medical answers for many people.
- vjvjvjvjghv 5mo agoDoctors using AI will probably just increasing the number of patients they see. But for me as patient AI is super useful to get a good handle on the situation before I see a doctor.
- guidedlight 5mo agoI agree. I think the issue with LLM’s are not with the correct diagnoses’s but rather the incorrect ones. Real doctors tend to have a degree of cautiousness. I would rather a real doctor be hesitate and seek more information, than an alarmist LLM suggesting I have cancer.
- kian 5mo agoBut what was the overlap?
- adamtaylor_13 5mo agoDespite what I suspect the general consensus on HN may be, this does not surprise me at all. My wife was recently diagnosed with Mast Cell Activation Syndrome (MCAS) after a pretty scary series of ER visits. It's a very strange and stubborn autoimmune disease that manifests with a number of symptoms that, taken individually, could indicate damn near anything. You could almost feel the doctors rolling their eyes as she explained her symptoms and medical history. Anyway... it lit a bit of a fire in me to dig deeper, and one day Claude suggested MCAS. I started plugging in more labs, asking for Claude to cross-reference journals mentioning MCAS, and sure enough: it's MCAS. idk what the moral of the story is except our current medical system is a joke. The doctors aren't the villains, but they sure aren't the heroes either.
- seanmcdirmid 5mo agoThe quality of doctors is really uneven, and the amount of things they can and have to pattern match on grows each year. I definitely hope they at least adopt AI tooling to ease their pattern matching burden. There is no reason AI needs to replace doctors, I think as it is in SWE doctors are still needed to guide and check the AI in its search for solutions. Of course, there are plenty of places on earth that are extremely under doctored, and AI will definitely be better than nothing in poor regions of Africa if all it needs is a network connection and someone to donate the tokens.
- gizmodo59 5mo agoThe negative reactions here are baffling me. The fact that we can even get to say 30% with computer is amazing. So much hatred towards AI and anything from the frontier labs like OpenAI (or Goog for that matter) makes no sense.
- an0malous 5mo agoI for one am delighted for my acquaintances in the medical field with their cushy, cartel-supported salaries to feel the existential dread of AI coming for their jobs like I have
- krupan 5mo agoI'm sorry that you are feeling existential dread about your career. It could help to stop listening to the hype that the people selling AI are spewing and take a hard look at the tools themselves. Like most products, they aren't as good as the salespeople say they are. Also, take any predictions for how these products will do in the future with a huge grain of salt. Predicting the future is very difficult. It's taken us 70 years of computer and AI research and development to get to this point. It's likely that the rate of improvement will not change drastically. Yes, things are changing, but the singularity (still) is not coming tomorrow
- 12345ieee 5mo agoOh no, imagine the people that save human lives having high salaries, the horror. If you, like me, are in the software field, know that this is likely the most comfortable job even invented by humanity, we should really be paid just above the poverty line in exchange.
- an0malous 5mo agoGive me a break, most of them are glorified drug dealers. Their salaries are inflated by an artificially capped supply of doctors, at the cost of patients. I had to leave my job this year because of burnout when the execs mandated that we use AI tools, become our own designers, PMs, and QA, and double our velocity. They run through a decision tree they leaned in residency every day and I’m learning how to do 3-4 other people’s jobs on top of whatever the new AI thing is. I was working nights and weekends while my friends in medicine are planning their 3rd vacation this year to Tuscany.
- david_mchale 5mo agohaving been in ERs too many times when they are beyond capacity, something like this would be better than patients slipping through the cracks, at least you get a chance.
- tsoukase 5mo agoThis reminds me GPT-4 era studies where the LLM was better in a Law school exam than a student. We are not in 2023 anymore, or in the case of medicine, are we? If yes, this is bad news for health related applications as the low hanging fruits in LLM have been cut off.
- getnormality 5mo agoWow, amazing. They had an AI robot running o1 look at live ER patients coming in just like a real doctor and they did that much better? Incredible! (literally)
- 1980phipsi 5mo agoHow much time do the doctors spend to diagnose versus o1?
- PAndreew 5mo agoI mean an LLM is a slightly stirred up soup of current human knowledge. It has an advantage in quantity of accumulated data and maybe connecting seemingly less connected parts of that data - but not reliably. The human has an advantage (for now) in data collection (seeing, hearing sensing the patient), actual agency, real world experiences and getting the useful data out of the stirred up soup. Both human and LLM are susceptible to bias and harmful influence. Let’s simply isolate them in the diagnostic process and then compare their output. Human collects data -> both human and LLM evaluate independently -> compare the results -> human may get new insights -> final diagnosis by human.
- arkt8 5mo agohow much confidence is 67%? does it was at the same patients with the same info? If not it is just selling bait.
- arkt8 5mo agoHow much far is 67% against 55%? Does the research considered same patients as the doctors? How much it can be effective for science if it is not compared side by side how each scenario was evaluated by both and how it came to different conclusions. Who can ensure a doctor couldn't spot some blind point AI couldn't at the remaining 43%. Tools are not for replacement but combining efforts. Throw such % to the public is a lot of irresponsibility.
- gamerslexus 5mo agoHold on. Does this mean ER diagnoses are marginally better than pure chance?
- n2d4 5mo agoNo, because randomly guessing from a list of diagnoses is not 50/50
- notahacker 5mo agoAnd ER generally does not involve key decisions being made by someone isolated from the patient given only an incomplete set of notes to make their diagnosis
- gamerslexus 5mo agoGood point.
- Kuyawa 5mo agoAs a 60yo I developed my own AI medical assistant [1] and I've used it extensively for many conditions, I can't be happier. After analyzing some lab tests it even recommended a marker that was not considered first by the doctor, so yes, it won't replace doctors but it is a very helpful tool for self-diagnosing simple conditions and second opinions. [1] https://mediconsulta.net https://mediconsulta.net (DeepSeek)
- nickvec 5mo agoVery cool! Just a heads up, the "Pricing" button in the navbar currently has no redirect.
- Kuyawa 5mo agoIt used to have a pricing section but I removed it to make it free for everybody. I'll fix that, thanks.
- Flere-Imsaho 5mo agoInteresting. From your website I couldn't see where you are based. The reason I'm asking is that I'd only consider using these types of services if they are European/UK based.
- Kuyawa 5mo agoI live in Venezuela, hosting in US, AI in China, if that helps.
- xaxfixho 5mo ago[dead]
- biglost 5mo agoMe da curiosidad, me gustaría saber si ese 33% es un subconjunto del 50-45% Si no es un subconjunto, entonces que tan grave fue ese error? Más muertes? Más tiempo de recuperación? En qué se tradujo esa diferencia?
- Tenobrus 5mo agoo1 has a METR time horizon of around 40 minutes, opus 4.7 has an implied horizon of 18 hours based on its ECI score. this study is on a model that's several generations behind wrt the kind of tasks it can complete. it would be shocking if this number were anywhere near as low with GPT 5.5, to the point it seems nearly totally irrelevant to talk about these results
- manmal 5mo agoI know a cardiologist who founded a training & knowledge base startup for doctors. He once told me (that was before LLMs), that it’s super common to tell a patient that the doc needs to look up sthg in their patient history, to then instead google the symptoms. Or, even more often, quickly text a colleague. I have no way of knowing if this is true. But I‘d rather had a complete, guided prompt be the basis of a diagnosis, than a 2m google search.
- warmwaffles 5mo ago> quickly text a colleague. This is still common and useful to gut check and make sure you aren't missing something. Source: wife is a doctor.
- manmal 5mo agoDoes she think this really does the complexity of each case justice though? I doubt you can compress an anamnesis into a two-liner without losing essential data.
- simoncion 5mo ago> Does she think this really does the complexity of each case justice though? Do you believe that -prior to the 2020-ish mass evacuation of doctors from the profession- the typical specialist would misrepresent the facts of a case when asking for a cross-check? Related: Have you ever worked as "the guys who actually work on the thing"-level tech support for a nontrivial Enterprise Software Product (or System)? If you have, did you never send a quick message to a knowledgeable coworker to double-check something that you were pretty sure was correct, but weren't 100% certain about?
- manmal 5mo agoAn enterprise product is not comparable with the human body at all. A single cell contains hundreds of times more information/entropy in its state than an operating system.
- lowbloodsugar 5mo agoComputers have been better at this since the 80s. But the doctors have a really good union, and they’re smart enough not to call it a “union” so it sounds like it’s about standards and ethics.
- ivolimmen 5mo agoI can't help to visualize the scene in Idiocracy where there is an examination. The guy gets multiple wires that gets put in his hands, mouth and rectum. The guy that assists (aka the doctor) switches the wires after each person. If we trust machines to much...
- thih9 5mo agoOff topic, is a “reject all and subscribe” cookie popup button legal? I thought websites have to make it as easy to give consent as withdraw consent[1] - and here one cannot withdraw consent without an extra step (subscribing). Instead I would expect access to the article, with same ads as in the “user consented” path, just not personalized. [1]: “The GDPR is specific that consent must be as 'easy to withdraw as to give'”, https://en.wikipedia.org/wiki/HTTP_cookie https://en.wikipedia.org/wiki/HTTP_cookie
- bux93 5mo agoNo, typically it is not. https://en.wikipedia.org/wiki/Consent_or_pay https://en.wikipedia.org/wiki/Consent_or_pay
- lqstuart 5mo agoNot long ago I started having an issue with my eye. I called around and they said I should get seen ASAP, same day if possible, but it wasn’t worth the ER and it was a five day wait for an appointment. I was pretty freaked out. During that time, I tried diagnosing it with AI. When I finally got to the appointment, the actual doctor sat down, looked at all the unremarkable images, asked me one (1) question, ordered another image and diagnosed the issue. When I looked back, in all that time, the AI had mentioned it exactly one time early on, ruled it out immediately based on a flawed understanding of the symptoms, and never brought it up again. Just my anecdotal evidence, but I’d never trust any AI on its own. My doctor can use it if they want, I can’t.
- bando00 5mo agoIt would have been interesting to see how a doctor with access to LLMs would perform, compared to only LLMs and only doctors. If doctors with LLM access still score 67%, then someone with no medical knowledge could potentially score the same, which would make ER triage a replaceable task by AI. But I am sure that is not the case. Competent doctors with the background they have can use LLMs to brainstorm and analyze different paths and score higher.
- ZiiS 5mo agoTriage deliberately diagnoses rarer conditions that would be more serious or require more urgent treatment so they can be ruled out.
- deleted 5mo ago[deleted]
- epmaybe 5mo agoI’m in ophthalmology where AI diagnostics have been promised for almost a decade. We have FDA approved diagnostics for diabetic retinopathy screening that has been commercially available since 2018, and papers claiming board certified ophthalmologist level classification accuracy as far back as inceptionv3. Maybe it’s just an economic barrier but these tools still haven’t made any meaningful impact in the US. Other countries without healthcare access? It’s helpful for culling the herd, but it doesn’t fix the last mile problem of what you do when you find referable disease that needs treatment. My philosophical take: if AI can outperform the average, it’s probably a net benefit for society that I won’t have a job. Until then, I’m going to take my income and save up for an early retirement.
- alansaber 5mo agoAI diagnostics is maybe 60% the way there. Robotics is maybe 20% the way there. You'll have a job as a doctor for a good long while.
- hansmayer 5mo agojfc, when does this ai boosting finally stop.
- Hobadee 5mo agoObviously annecdotal, but a couple years ago my friends kid was sick, and doctors were trying to figure out what was going on. My friend threw the symptoms and test results into ChatGPT, and it said the likely cause was leukemia. A few hours later the doctors handed them an official leukemia diagnosis. I think AI, like in all other fields, will become a great tool to help augment. Throw the patient data in and get a response and that can be the first thing the doctor checks for, but they shouldn't simply take AI as truth. P.S. friends kid is doing great - it was caught early enough. They are due to be completely done with treatment in just a couple months!
- nikhilpareek13 5mo ago[flagged]
- noashavit 5mo agoIf this is repeatable and holds true across testing groups and practitioners that would be amazing! Doctors could finally spend time with patients rather than rushing to probe, document, test and diagnose. They are so pressed to maximize their time that any time back could go straight into real care. Am I being blindly optimistic here?
- plexescor 5mo agoOne shouldnt trust AI regarding medical matters, things can go downhill you know
- economistbob 5mo agoWhat we need is completely walled garden during the ER sign in process where the patient tells what they think the problem is. The things proceed as normally. We need some data to know if the patients are leas than fifty percent accurate or not. Fifty percent accuracy. That's terrible.
- hereme888 5mo agoHyped title. It was exclusively text-based diagnosis after physicians did the whole interview, exam, labs, etc. Also, later in the encounter, with more chart information, AI scored 82%, physicians 70–79%; that difference was reportedly not statistically significant. So current AI can aid in diagnosing like we've all known.
- programmertote 5mo agoMy spouse is an hematologist+oncologist. She and all of her coworkers use ChatGPT. Before then, they look stuff up on UpToDate [ https://www.uptodate.com/login https://www.uptodate.com/login ] (they sometimes still do). I went to medical school for three years and quit because I couldn't stand the rote memorization part of the studies. Too many facts to remember IMO. Even as an AI-neutral person, I'm very confident that AI/ML based computer systems, once trained specifically for medicine, will consistently do better than human doctors because believe it or not, there are a lot of human errors made in medicine field (doctors just don't admit that and we don't know) due to lack of time by doctors or incompetence or simply forgetting a fact or two that they should have checked when diagnosing or coming up with a treatment.
- burnte 5mo agoUpToDate is SUCH an awful company, pure rent taking. For site licenses, you just give them your sites' IP addresses and they program them into their firewall. No account management at all. INSANELY high prices. We replaced them with OpenEvidence.
- donchuru 5mo agoI have a lot of doctor friends who tell me they all use OpenEvidence [1] in their practice. They've done a good job of capturing the doctor market while offering a useful product. [1] https://www.openevidence.com/ https://www.openevidence.com/
- zahlman 5mo agoSince when do "triage doctors" attempt diagnosis, or have the expectation of doing so? They're just trying to figure out who needs to see the actual doctor first.
- ButlerianJihad 5mo agoYeah, triage is fundamentally a process of arranging patients in a priority queue, so that the most critical cases can be addressed in minutes, and the other resources are assigned to people who aren’t immediately dying or going into shock. Triage in disaster/crisis response can even be about figuring out which patients are already dead, or cannot be helped before dying, so you mark them or assign them with a toe-tag, and focus your resources on preventing that number from increasing.
- yfw 5mo agoSensitivity vs specificity
- ArjunPatel64 5mo agoAI beating doctors on text-based diagnosis isn't surprising. LLMs are pattern-matching machines trained on millions of medical cases. But the real test is the physical exam. Can AI tell the difference between someone faking pain and someone with a ruptured appendix? Between a panic attack and a heart attack? That's what doctors do that AI can't.
- ArjunPatel64 5mo agoThe x-ray study where AI beat radiologists without access to x-rays is hilarious and terrifying. It means the benchmark was broken, not the AI. We're going to see a lot of "AI beats humans" studies that just mean the humans were asked to do something unnatural. Real medicine isn't a multiple-choice test.