7 ms·
Doctor GPT: A Large Language Model That Can Pass the US Medical Licensing Exam
- throwuwu 3y agoAmazing that you could use the 7B model for this. Going to try it out tomorrow. How good are the answers it gives in your experience?
- classichasclass 3y agoImpressive, but I know plenty of doctors who passed the USMLE and shouldn't be allowed within miles of a living human being. (Also, which step, 1, 2 or 3?)
- da39a3ee 3y agoWhy not log in anonymously to a single shared service that can be maintained more easily than N instances for N patients?
- ubj 3y agoI don't see a description on the GitHub page of any experiments performed showing the model can pass an exam. Where are these details, and how was the testing performed? EDIT: The git commit was 4c6d52a when I originally posted this comment.
- bomewish 3y agoAgree. Seems highly irresponsible to make such claims without demonstrating and validating them. Would also like to see how much it hallucinates or how closely the generations are to source data.
- avereveard 3y agoin the notebook there is a passage where it goes trough the exam dataset and use an embedding model to compare answer. I don't think that's the accepted methodology (I think they ought to compare the logist weight on the answers keys), on top of that, the embedding model they use for similarity is somewhat weak. also passing in their own graph means 61% right or something like that.
- syntaxing 3y agoSince its trained from LLaMA-2 7B, you in theory can just straight up load the weights [1] with text-gen-ui or llama.cpp. Hoping TheBloke converts this to GGML + safetensors so I can try it out. [1] https://huggingface.co/llSourcell/medllama2_7b/tree/main https://huggingface.co/llSourcell/medllama2_7b/tree/main
- formvoltron 3y agoHow can we use this to push doctors to be better doctors? Maybe if somehow and AI reviewed the doctors decisions? Or maybe if doctors are required to provide a transcript of their observations and decisions and gives that to the patient who can then fact check it with gpt?
- ch4s3 3y agoThis is an amusingly naive take. Clinical decision support software has existed for 20 years, and some types of healthcare groups mandate the use of those tools. Moreover doctors all take notes and in many places those are available through patient portals already.
- Tao3300 3y ago> maybe if doctors are required to provide a transcript of their observations and decisions and gives that to the patient who can then fact check it with gpt? When you try to brainstorm so hard that a brainfart slips out.
- nvy 3y agoImagine a future where GPT can fact check something. What a beautiful dream that would be.
- syndicatedjelly 3y ago> How can we use this to push doctors to be better doctors? By never showing this to a medical professional, who will laugh you out of the room and never take AI seriously again.
- TuringNYC 3y agoThank you for the submission @mutant_glofish and efforts @llsourcell if you are reading this. Could you elaborate how much compute (esp cost) you incurred for the fine-tuning? I think there is an entire blog post there that would be of great interest to numerous folks Also could you evaluation on your evaluation criteria? How did you test this? How did you ensure the exams were not leaked into the training et?
- empath-nirvana 3y agoI think it would be a good idea to consult with an attorney about your product description. You have to be extremely careful about making medical claims. The fact that you're so careless about this makes me concerned about your ability to judge its medical advice.
- tansan 3y agoWhat issue did you have with it?
- Tao3300 3y agoTell me you're giving medical advice without telling me you're giving medical advice. > an open-source project with a mission to provide everyone their own private doctor Perfect.
- bobboies 3y agoTo be fair to the author—they state it can “pass the medical exam.” Which is different than if I should ask it for medical advice. I mean it can’t examine me physically therefore it can’t be my doctor. On the other hand, there aren’t enough doctors where I live so I guess I’ll take it! Lol
- BlackjackCF 3y agoI think this is the wording I have beef with: “DoctorGPT is a Large Language Model that can pass the US Medical Licensing Exam. This is an open-source project with a mission to provide everyone their own private doctor.” The “provide everyone their own private doctor” part definitely makes it sound like it can replace a doctor. I’m just worried about the combination of medical misinformation and the people who don’t understand how LLMs work who will take everything a LLM spits out as truth. There are already a ton of examples in the news of people who really don’t understand how they work (like that one professor who failed an entire class because ChatGPT told him that all the papers that the class wrote were plagiarized.) We also just saw how much harm can be done with medical misinformation with COVID. Maybe I’m just being pessimistic and paranoid, but god am I scared of how things could cause people to inadvertently harm themselves and the people around them.
- nine_zeros 3y agoCan someone make a LeetcodeGPT? We need to trivialize leetcode interviews to oblivion.
- verandaguy 3y ago_Wow_ this is a dangerous precedent and tagline. @op: please update this project's description to explain that it's not a substitute for seeing a real doctor. People can get hurt with tools like this. To be absolutely clear: as you currently describe it, implicitly or explicitly, this project is massively irresponsible.
- themitigating 3y agoA doctor is simply a database of past cases, information they've read, and medical knowledge. They take in symptoms, perform tests, analysis output, repeat
- princetman 3y agoA programmer is simply a database of past projects, information they’ve read, and programming knowledge. They take in requirements, perform coding, analyse output, repeat.
- themitigating 3y agoNo because programmers can produce new code/logic
- princetman 3y agoYes, field of diagnosis and experimental medication/treatment requires no new logic.
- bugglebeetle 3y ago> The total training time for DoctorGPT including supervised fine-tuning of the initial LLama model on custom medical data, as well as further improving it via Reinforcement Learning from Constitional AI Feedback took 24 hours on a paid instance of Google Colab. The most impressive part of this project is that they got Google Colab to not close the session for 24 hours straight.
- malablaster 3y agoGP to a tee.
- eftychis 3y agoNot legal advice. You almost certainly need legal advice on this topic. > DoctorGPT is a Large Language Model that can pass the US Medical Licensing Exam. This is an open-source project with a mission to provide everyone their own private doctor I can think of the top of my head a number of issues, depending on which country or state you are in. If you are in California U.S.A. for example... Given that this references USMLE (U.S. Medical License Exam) Regardless, a AS IS warranty is in order and HUGE banner that this is not meant to be a doctor in any way shape or form. Which you kind of claim it is. And that is a problem. [And change the name to not state in any way doctor in there. ]
- beders 3y agoYou want to trust a text generator with a known tendency to hallucinate?
- dr_kiszonka 3y agoI was excited to try it but some errors and comments in the notebook gave me pause.
- Animats 3y agoThe scary thought is that once this sort of thing starts to work, it will be used by call centers for low-end medical services.
- thornjm 3y agoOff topic because I am skeptical of the claims in the first place. As a general principle: the licensing exam is not the benchmark AI should be evaluated against to then go about calling itself a doctor. I'm from a different country but these exams are the minimum standard to demonstrate a doctor is safe prior to interacting with patients. To be really explicit, the core competency being assessed is identifying potentially serious situations and answering the same way every time: "I WOULD CALL FOR HELP" +/- principles of basic care. The benchmark for doctors actually making decisions about patient care are the assessments to become fully qualified consultants in a each specialty. Again, to be really explicit don't confuse a test for "doctor won't immediately kill someone and commence reasonable first steps" with an actual "doctor with years of experience and subspecialty training who regularly makes decisions about patient care".
- npsomaratna 3y agoNon-US doctor here. Completely agree. Licensing exams mainly confirm that the aspiring doctor won't harm/kill patients by mistake. Specialist-level exams assess standard of care.
- alexnewman 3y agoJust reframe it as education
- prashp 3y agoAre we all aware that the repository owner is Siraj Raval? The guy infamous for claiming others' work as his own? I think the ML community disowned this guy a long time ago for peddling snake oil
- minimaxir 3y agoTechnically, he was disowned for a) plagiarizing a ML paper badly, creating the "complicated doors" meme b) taking money for an online course that he blatantly misrepresented and c) doing both a and b at the same time and therefore getting too much heat to ignore. He did not get as much heat for just peddling AI snake oil. Atleast AI scrutiny has evolved since then, and I'm surprised this is the first time I've heard about Siraj in quite awhile.
- judge2020 3y ago> a) Have a reference for this meme? Can't find it from a google search
- minimaxir 3y agoHere's one of the original discussions: https://www.reddit.com/r/learnmachinelearning/comments/dh38x9/siraj_raval_has_a_new_paper_the_neural_qubit_its/ https://www.reddit.com/r/learnmachinelearning/comments/dh38x... It appears I was wrong on the specifics of the meme: > He changed "logic gate" to "logic door" and "complex Hilbert space" to "complicated Hilbert space" to try to hide the plagiarism. What a legend.
- jmugan 3y agoIt's funny how memory works, "complicated doors" is an almost perfect combination of those two mistakes.
- kordlessagain 3y agoAdd taking our repo/project name to the list: https://GitHub.com/featurebasedb/doctorgpt https://GitHub.com/featurebasedb/doctorgpt
- rossjudson 3y ago"Please I need to bring my daughter in to see a doctor." "I did not understand what you said; please repeat." "I need to bring my daughter in to see the doctor!" "I am sorry but the medical insurance registered to your phone does not include in-person contact. Do you wish to upgrade?" "No! I really need to bring her in." "I am sorry that our service is not meeting your expectations. Is there anything more I can do for you?" "Please let me speak to a doctor!" "I am sorry that our service is not meeting your expectations. Is there anything more I can do for you?"
- EdwardDiego 3y agoI've implemented my own medical AI, it's called DrHouse and the entire implementation follows: def diagnosis(): return "It's not lupus."
- m3kw9 3y agoAl I know is that just passing comp sci exam won’t get you a senior programming job.
- dbmikus 3y agoI was reading through the notebook, and I saw that the `extract_pipe_output` function is filtering out elements without a "POSITIVE" label. There's a snippet like this: logits = extract_pipe_output(sentiment_pipe(texts, **sentiment_pipe_kwargs)) rewards = pos_logit_to_reward(logits, task_list) `pos_logit_to_reward` expects `logits` and `task_list` to be the same length, but `logits` is the result of some filtering. I must be missing something about the code and libraries (I'm a PyTorch newb), because I'd otherwise expect this to be a bug.
- ForestCritter 3y agoDoesn't say much for the medical licensing exam...