11 ms·
GenAI and erroneous medical references
- rubatuga 3y agoThe safest way now to use LLMs is for simple entity extraction of signs, symptoms, investigation summaries, then translate them into the inputs for an understandable linear/logistic model, e.g. Wells criteria, Canadian CT head, or Centor score. I feel that a comprehensive but explainable model that supports multiple diagnoses will be developed in the future, but no such model currently exists.
- deleted 3y ago[deleted]
- LuciBb 3y ago[dead]
- larsiusprime 3y ago> Unfortunately, very little evidence exists about the ability of LLMs to substantiate claims. In a new preprint study, we develop an approach to verify how well LLMs are able to cite medical references and whether these references actually support the claims generated by the models. Is there a corresponding control group for how well an average doctor is able to cite medical references and whether these references actually support the claims generated by the doctors?
- jncfhnb 3y agoDoctors don’t cite claims on the spot
- MauranKilom 3y agoA fair comparison (on a technical level) to GPT-4 RAG would be a doctor in a relevant field who also has internet access. I think this would be indeed interesting to compare to assess the resulting quality of care, so to speak! (The other models being only partially able to source good references is unsurprising/"unfair" on a technical level, but that's not relevant for assessing their safety.)
- esoleyman 3y agoGive me a few minutes and I can pull up any number of medical studies or references to back up my claims. I don’t have them memorized to the actual URL but I have kept up to date with the latest studies and summaries that pertain to my field and my patients.
- SkyPuncher 3y agoNeither do most professionals- unless they’re doing a prepared segment. The difference is LLMs can’t back their claims.
- numpad0 3y agoSome engineers can cite RFCs and ISO standards on the spot sometimes, probably same for doctors
- Loggias 3y ago[dead]
- larsiusprime 3y agoThey often give clinical recommendations and prescriptions, usually after a quick skim of the medical history and a rushed five minute conversation with the patient. It would be nice to know how many of these typical in-office recommendations wind up being actually backed by the current state of the research, whether a citation is given in the visit or not.
- salad-tycoon 3y agoIt’s more like this, there is an algorithm / flow chart. You follow it down and do the next step in the process. You learn the reason why and then you just follow whatever the latest update to the algorithm is. Doctors are then graded by administrators as to how well they follow that flow chart. Have you done this or that screening, is BP well controlled, etc. Medicare sets many of the standards, you look up the USPTF guidelines for some more idea of what your doctor should be doing. So yeah. Kinda like factory work. “Pt male 70 wakes up urinate - does he have signs of infection > yes/ no> proceed to no > next step do this test > prescribe this med > re evaluate in 2-4 weeks. “
- __loam 3y agoWhy would we use proven processes involving experts with over a decade of education like this when we could use an LLM thrown together by a software developer who may not have even taken a biology course?
- razodactyl 3y agoSpot on
- paulddraper 3y agoNo, but studies have found that 250k+ deaths per year in the US are due to medical errors. [1] [1] https://www.hopkinsmedicine.org/news/media/releases/study_suggests_medical_errors_now_third_leading_cause_of_death_in_the_us https://www.hopkinsmedicine.org/news/media/releases/study_su...
- __loam 3y agoDoctors also go to school to be doctors for 12 extra years. It turns out that medicine is hard. Not really a good reason to turn to an LLM that will just confidently make things up.
- paulddraper 3y agoMedicine being hard is in fact an excellent reason to use AI tools.
- larsiusprime 3y agoI’m aware, I come from a medical family. You haven’t answered the question.
- esoleyman 3y agoMartin Makary’s study and the previous IOM one are based on faulty statistics used. The number is extrapolated from a small population to a larger one. I haven’t paid it any attention because of this problem. GIGO. https://www.sciencealert.com/no-500-people-don-t-die-in-the-us-every-two-days-due-to-medical-error https://www.sciencealert.com/no-500-people-don-t-die-in-the-...
- epcoa 3y agoYes perhaps read your own link: “ The researchers caution that most of medical errors aren’t due to inherently bad doctors, and that reporting these errors shouldn’t be addressed by punishment or legal action. Rather, they say, most errors represent systemic problems, including poorly coordinated care, fragmented insurance networks, the absence or underuse of safety nets, and other protocols, in addition to unwarranted variation in physician practice patterns that lack accountability.” An LLM is not going to address any of that. You are misinformed implying a significant majority of medical system errors are due to misdiagnosis.
- melagonster 3y agoI sure most of treatment are from textbooks.
- extragood 3y agoThis has been my major concern with the currently available LLMs. You can know what the input is, you can know the output, you may even be aware what it's been trained on, but none of the output is ever cited. Unless you are already familiar with the topic, you cannot confidently distinguish between fact and what sounds reasonable and is accepted as fact.
- deleted 3y ago[deleted]
- mateo1 3y agoMost doctors will be able to turn around, pick the corresponding textbook from their library and show you where they learned something. Or point you to an actual clinical case they had to handle. If they had to, somehow.
- atleastoptimal 3y ago> A new study finds that large language models used widely for medical assessments cannot back up claims. like real doctors
- rzzzt 3y agoI don't want to single you out, but what should I be taking away from the often recited "well people sometimes* can't do thing X either"-counterargument? Is all of this just fine? Can't we expect just a little bit more, I don't know, accuracy or rigor from a computer than a living person?
- chasd00 3y agoYeah I don’t buy that argument either. If excel got the math wrong replying with “well humans do to” wouldn’t fly.
- atleastoptimal 3y agoWe've already passed the point where LLM's are better than human experts for medical diagnoses. In fact, according to this study, even LLM's alone are more accurate than human experts + LLM's, meaning any input the humans added was only a detriment to the accuracy https://arxiv.org/pdf/2312.00164.pdf https://arxiv.org/pdf/2312.00164.pdf (Page 8) Computers are already perfectly accurate, and have been for decades in explicit quantifiable fields. In medicine, since a computer cannot perfectly replicate every single cell in the human body, its abstractions will be lower resolution than reality, but what matters is whether that low resolution abstraction is better than the alternative (human doctors). A human doctor couldn't bring up a list of citations in literature instantly regarding a diagnosis. A LLM can.
- rzzzt 3y agoThey could, but they don't, at least that is what I'm getting from the article. Even if they do, someone with the capability and understanding required (ie. not me) needs to bring that source up and verify that the claims align with the citation; the authors decided to use GPT-4 for this: "We adapted GPT-4 to verify whether sources substantiate statements and found the approach to be surprisingly reliable." I'm not happy with that either.
- simonw 3y ago"For the most advanced model (GPT-4 with retrieval augmented generation), 30% of individual statements are unsupported and nearly half of its responses are not fully supported" Show us the source code and data. The way the RAG system is implemented is responsible for that score. Building a RAG system that provides good citations on top of GPT-4 is difficult (and I would say not a fully solved problem at this point) but those implementation details still really matter for this kind of study. UPDATE: I found it in the paper: https://arxiv.org/html/2402.02008v1#S3 https://arxiv.org/html/2402.02008v1#S3 - "GPT-4 (RAG) refers to GPT-4’s web browsing capability powered by Bing." So that "30% of individual statements are unsupported" number was actually a test of how well ChatGPT's GPT-4 browsing mode with Bing could provide citations when answering medical questions.
- IanCal 3y agoIt's using the web search provided by openai. Importantly this doesn't actually guarantee that it does any kind of search. I'm confused as to whether they're using the API or not. Afaik only the assistant API has access to the web search, so I would expect this was manually done? But then the reason for only doing this with openai is that the others don't provide an API > GPT- 4 (RAG) refers to GPT-4’s web browsing capability pow- ered by Bing. Other RAG models such as Perplexity.AI or Bard are currently unavailable for evaluation due to a lack of API access with sources, as well as restrictions on the ability to download their web results. For example, while pplx-70b-online produces results with online access, it does not return the actual URLs used in those results. Gemini Pro is available as an API, but Bard’s implementa- tion of the model with RAG is unavailable via API.
- golergka 3y ago> Importantly this doesn't actually guarantee that it does any kind of search. What's more important is that a user _can see_ whether GPT-4 has searched for something or not, and can ask it to actually search the web for references.
- CSMastermind 3y agoThat's wildly misleading then. It would be interesting to see how GPT-4, properly augmented with actual medical literature, would do.
- hackerlight 3y ago> This problem goes away with the retrieval augmented generation (RAG) model, which first performs a web search for relevant sources before producing a summary of its findings. However, even in the GPT-4 RAG model, we find that up to 30% of statements made are not supported by any sources provided, with nearly half of responses containing at least one unsupported statement. Could you do something like this. Chunk/separate the individual claims made by the LLM, and the associated sources from RAG. Then feed them into a second LLM one by one, asking "is this claim $X_i reflective of the source $Y_i?". Whatever this second LLM says, feedback the response to the original LLM and ask it to revise what it's saying if the answer is "No". Iterate until the second LLM says "Yes" to every separate claim. Not perfect but might help.
- simonw 3y agoGoogle Gemini actually has a feature like this, though it's hard to spot. There's a colorful G icon below each Gemini response - if you click it, a second process runs which attempts to "fact check" the claims from the original prompt response - it highlights them in different colors and adds citation links to them.
- abid786 3y agoLLMs are better at this and it’ll probably marginally improve the output quality but there can potentially be hallucinations (false positives or negatives) even in this evaluation task.
- potatoman22 3y agoThis doesn't seem to say the LLM is wrong, it's just bad at giving a source to its statements.
- Legend2440 3y agoNo shocker there, even when LLMs have the correct answer they can't tell you where they got it. They simply do not know.
- rickysahu 3y agoAgreed that this is a challenging problem, but mostly because of the data used to train and the tokenization for language models. We’re working on this building what we call a large medical model (lmm) which is trained on medical event histories from 10s of millions of patients, not papers or internet text. Our tokens are medical codes facilities demographics etc. If anyone is interested we have an api + we’re hiring. https://genhealth.ai https://genhealth.ai
- srameshc 3y agoVery interesting. Who are your target customers ?
- rickysahu 3y agoIts broadly applicable, but initially health plans and provider orgs.
- noduerme 3y agoThis is fascinating... but just a superficial nit about the website and what it puts across: As someone who spent half my life designing, illustrating and art directing for brands, the choice to use anime style art as the theme for section headers is the exact type of decision I look for when reviewing portfolios that tells me a potential hire might know how to use image editing tools or be proficient in technical fields, but is not actually a designer; i.e. can create an image they "like" but lacks the knowledge and creative ability to synthesize subtexts in design and art history in original ways that are visually compelling but also create the right tone for a client. The choice of generated amime art implies its use on the page as mere decoration, rather than as a language of communication carefully selected to give viewers the right impression of the company. It argues for why diffusion models guided by engineers will not replace professional art direction anytime soon. No offense, this is a standard art crit I would give to any art student, or to anyone running a startup who had nonprofessional design.
- user3939382 3y agoTo play devil's advocate, it could turn out that only 1 person needed to be persuaded by the art, some investor that ends up leading their next round, who happens to be an anime fan and now it was the right choice. > rather than as a language of communication carefully selected to give viewers the right impression of the company You could argue it gives the impression that the culture of the company is primarily technical, given that the technical and anime communities have a huge overlap.
- Aurornis 3y agoA close friend of mine went down the ChatGPT medical rabbit hole last year. He was disagreeing with his doctors and getting deep into supplements and unproven medical theories. ChatGPT was instrumental in convincing him that he was correct and his doctors were wrong. He would post his ChatGPT conversations as proof, but we could see that his prompts were becoming obvious leading questions. He would phrase things like “Is it possible for {symptoms} to be caused by {condition} and could it be treated by {herbal product}?” Then ChatGPT would give him a wall of text saying that it’s possible, which he took as confirmation of being correct. It was scary to see how much ChatGPT turned into a mirror for what he wanted to be told. He got very good at getting the answers he wanted. He could dismiss answers that disagreed as being hallucinations or being artifacts of an overly protective system. With enough repeat questioning and adjusting his prompts he could get it to say almost whatever he wanted to hear. ChatGPT is rocket fuel for hypochondriacs. Makes WebMD look tame.
- ageek123 3y agoYou can do the same thing with regular web search.
- dartos 3y agoNot as quickly, easily, or convincingly.
- __loam 3y agoWeb search also wasn’t trained to be convincing to humans.
- visarga 3y agoWeb content was created to be enticing to humans. Most of it is just like flowers waiting for bees to come over.
- tsimionescu 3y ago
- jonathan-adly 3y agoIf anyone is interested - my startup which did exactly this was acquired 8 months ago. As many mentioned - the sauce is in the RAG implementation and the curation of the base documents. As far as I can tell, the software is now covering ~11m lives or so and going strong - the company already got the acquisition price back and some more. I was even asked to come support an initiative to move from RAG to long context + multi-agents. I know it works very well. There are lots of literature from the medical community where they don't consult any actual AI engineers. There are also lots of literature on the tech community where no clinicians are to be seen. Take both of them with a massive grain of salt.
- dartos 3y agoIf you don’t mind sharing, what lesser known RAG tricks did you use to ensure the correct information was going through? Simple similarity search is not enough.
- jonathan-adly 3y agoMain thing is to curate a good set of documents to start with. Garbage in (like Bing google results like this study did) --> Garbage out. From the technical side, the largest mistake people do is abstracting the process with langchain and the like - and not hyper-optimizing every step with trial and error.
- westurner 3y agoFrom [1], pdfGPT, knowledge_gpt, and paperai are open source. I don't think any are updated for a 10M token context limit (like Gemini) yet either. [1] https://news.ycombinator.com/item?id=39363115 https://news.ycombinator.com/item?id=39363115 From https://news.ycombinator.com/item?id=39492995 https://news.ycombinator.com/item?id=39492995 : > On why code LLMs should be trained on the edges between tests and the code that they test, that could be visualized as [...] "Find tests for this code" "Find citations for this bias" Perhaps this would be best: "Automated Unit Test Improvement Using Large Language Models at Meta" (2024) https://news.ycombinator.com/item?id=39416628 https://news.ycombinator.com/item?id=39416628 From https://news.ycombinator.com/item?id=37463686 https://news.ycombinator.com/item?id=37463686 : > When the next token is a URL, and the URL does not match the preceding anchor text. > Additional layers of these 'LLMs' could read the responses and determine whether their premises are valid and their logic is sound as necessary to support the presented conclusion(s), and then just suggest a different citation URL for the preceding text
- iamleppert 3y agoChatGPT4 correctly diagnosed my neurological condition, an infection that many doctors had missed. While I was in the hospital I asked ChatGPT the same questions as the doctors and it was nearly identical to what they were telling me every time. It also acted as a therapist and talked me down from several depressions while in the hospital, far better than any human therapist I’ve ever had. The fact that it’s an AI made me actually feel better than if the therapy was delivered by a real therapist, for some strange reason.
- dheera 3y agoMeanwhile my doctors are incompetent at diagnosing most of my symptoms, so I'll take ChatGPT over nothing until the medical system can gets its shit together.
- hooloovoo_zoo 3y agoA medical statement being supported by at least a single citation is honestly a really low bar anyway.
- xyst 3y agoFor every 1 medical case solved by web search/chatgpt/ai. There will be hundreds of false-positives. I remember shadowing in the ER in the early 10s and many people would come in with Google search trash. One kid even cited something from the psychologist handbook. Lol
- pants2 3y agoEvery medical issue I've ever had I managed to correctly diagnose by myself using Google (haven't had many issues since ChatGPT) and later confirmed by a doctor. I attribute this to my years of experience using Google to troubleshoot and debug software.
- elietoubi 3y agoAs a fun side project, We fine tuned mistral 7B and used RAG with 35M radical papers. Gives you actually really accurate answers: https://chat.scribemd.ai https://chat.scribemd.ai
- EagnaIonat 3y agoDoing a similar project it has been quite an eye opener on the results. At a first glance the answers look very good but we have noticed these things. In a number of cases the documents returned were wrong, but the LMM was able to use the contents of documents to find the right answer within the model. In our case this is a fail, as it had to cite valid documents. Not as frequently but also noticed is the sequence of returned chunks from the vector database had an impact on the answer. In one case the question had one word as a past in first question, present tense in the second question. Otherwise the same. This swapped the order of the chunks and gave an opposite answer. There is no easy way to see when these pop up. The current testing frameworks are limited in validating these.
- ilaksh 3y agoCan a 7B model really give good answers? The reasoning and overall capacity of small models seems inadequate for serious tasks.
- lrasinen 3y agoIn similar vein: I've asked ChatGPT (3.5) electronics questions. It happily generates non-existent links to schematics images, and then refers to the schematics in its answer.
- spangry 3y agoChatGPT4 has overreacted to this issue IMHO. When I try to do even slightly esoteric medical research, ChatGPT seems pretty intent on only referencing Cleveland and Mayo Clinic, the most mainstream of orthodox mainstream medical sources. Try to get it to reference even peer-reviewed medical journals requires a frustrating amount of cajoling - it seems extremely reluctant to deviate from anything that isn't 100% mainstream medical orthodoxy. This is not a good solution in the long run - ChatGPT will just reinforce existing dogmas and orthodoxies, even the ones that are (inevitably) wrong. Imagine if this approach to medical science was widespread at an earlier point in our history - we'd all probably believe that peptic ulcers are caused by 'stress' (rather than, primarily, the bacteria Helicobacter pylori). Go back even further and we'd still be lobotomising gay men to 'change their sexual orientation'. Rigidly enforcing current orthodoxies, under the premise that we're right about everything unlike those idiots in the past, will kill progress and society will stagnate. If I wanted a blindingly arrogant tech mega-corporation to decide what 'experts' I'm allowed to get information from, I'd just use Google instead. If, as many seem to believe here, OpenAI are just worried about being sued, then why don't they just create an individual 'safe GPT output' setting (like Google 'safe search') which I can disable after acknowledging disclaimers that it's dangerous to think for myself and question mainstream positions. I've grown to hate authoritarian Silicon Valley twats who arrogantly impose their politics, ignorance and, frankly, bizarre norms on the rest of us. It's highly ironic that these 'I love science!' types don't appear to understand that the scientific process involves making empirical observations, forming hypotheses consistent with those observations, and then continuously testing those hypotheses to determine which one is most robust to observed reality. They instead seem to think science is some kind of religion where you treat the views of the mainstream authorities as divine truth revealed by God and only heretics dare to question. By discouraging the formation of alternative hypotheses and rigorous questioning they are actually inhibiting scientific progress and making a mockery of the scientific method. I look forward to the day we have a model that just synthesises available information and let's us decide for ourselves what to make of it. I think people will switch to such a model in droves and the likes of OpenAI and Google will go the way of all other social conformists who attempt to enforce the reality-denying orthodoxies of their day.
- donkeybeer 3y ago
- kleiba 3y agoSome of my former co-workers founded a start-up a few years ago (before the ChatGPT age) focussing on AI tools for the medical sector. By far their biggest challenge wasn't the tech but the hoops they had to jump through to get even their smallest tools certified so that they could be used by doctors at all. Unbelievable amounts of red tape and time investment on the side of that startup. I find it disgusting to hear that ChatGPT and the likes are now regularly used by medical professionals where, obviously, these LLM tools have never gone through the same process. For two reasons - one because of the obvious medical risks involved (hallucinating LLMs) but also because of the gross commercial advantage the companies behind these LLMs get without even as much as trying to get certified. I've lost touch with my former co-workers but I would be surprised if they hadn't looked into LLMs to be included in their products. My guess is that they're still somewhere in the administrative hell of trying to get their latest tools certified while meanwhile, doctors are happily using ChatGPT and GenAI. Must be frustrating.
- harvey9 3y agoIf doctors consult chatgpt (unregulated, uncertified) that's on them, not the platform. I hope your old team keep working on high quality stuff but I can understand if they walk away. I have seen some chatbot type products aimed at the admin side of healthcare, with the salesmen telling me about the 'guardrails'. Given some guy recently got the bot at his utility supplier to start criticising the company and using foul language then I do not think even this less critical work is ready for general use.
- kleiba 3y agoWell, I think it's worse than you may think: the startup is not even allowed to market their products as medical tools unless they're certified. ChatGPT is used today by doctors without being certificied. How is that fair?
- harvey9 3y agoI don't think it is fair and I do know the difference between a regulated product and an ad hoc (mis)use of ChatGPT. I just do not think you can blame OpenAI for doctors (or other professionals) ignoring the warning they are presented with. On the other hand I do think professional bodies should be inclined to strike off members who do this kind of thing.
- castles 3y agoSegment ids are all you need (probably)
- robviren 3y agoI am attempting to come to grips with this problem actively. What are people's thoughts about using an LLM as a tool for linking out information? Traditional search is not proving enough in connecting patients and providers with the absolute wealth of information on grants, best practices, etc. There is simply too much content in too many places. I dream of something like "Cancer Bot 9000" that would be able to connect to resources pulled from a RAG, not necessarily answer the questions directly but interpret the questions and connect the person with the most likely resources. Bonus points for additional languages or accessibility which I constantly see as a barrier.
- 082349872349872 3y agoLooking on the bright side: will GenAI be what convinces the general public to finally get into our habit of checking if references actually support claims?
- btbuildem 3y ago> even in the GPT-4 RAG model, we find that up to 30% of statements made are not supported by any sources provided I wonder how much of that is due to specifics of the paper's authors' RAG implementation -- and how much is due to the guard rails put into GPT by OpenAI (not unlikely, given how medical advice would be a hot button topic for lawyers).
- throwaway118899 3y agoI’m going to make a career out of tricking chatbots into giving wrong advice and suing the companies using them. Like what happened with the Canadian airline but on an operationalized scale. Imagine a small call center dedicated to destroying companies who get rid of their human customer support.