7 ms·
PMC-LLaMA: Further Finetuning LLaMA on Medical Papers
- opisthenar84 3y agoWith 75B tokens, would it be better to train from scratch, then apply instruction fine-tuning? Training from llama seems like it could introduce a lot of unexpected behavior.
- Taek 3y ago75B tokens is not really enough data to make an intelligent model. Llama was trained on over 1 trillion tokens. And yes, training on top of LLaMA could introduce a lot of unexpected behavior, but that's just where the State-of-the-Art is today
- panabee 3y agoBrief tweetalysis: https://twitter.com/omarsar0/status/1652840470878232576?s=20 https://twitter.com/omarsar0/status/1652840470878232576?s=20
- randomcatuser 3y agocan someone explain the PEFT vs Zero-shot results to me? (in the table) How can Llama go from 1.8 to around 68?
- deleted 3y ago[deleted]
- kristjansson 3y agoAIUI they (1) finetuned LLaMA on 75B tokens of medical papers, and called the resulting model PMC-LLaMA, (2) finetuned that model on task-specifc data, then (3) separately finetuned the base model they started with in (1) on the same task specific data for comparison. So, they're reporting 6 separate conditions: {LLaMA, PMC-LLaMA} x {Zero-shot, Finetune, PEFT}. Could definitely use some more exposition around that though.
- khimaros 3y agoweights available here: https://huggingface.co/chaoyi-wu/PMC_LLAMA_7B/ https://huggingface.co/chaoyi-wu/PMC_LLAMA_7B/
- deleted 3y ago[deleted]
- nologic01 3y agoWould you trust a LLM giving medical advice in the wild?
- esquire_900 3y agoAt some point it becomes a serious tradeoff. It will definitely excel in finding niche solutions, in bringing up that one mechanism that's rarely used but very relevant in some case. It cannot say "I don't know" and will start babbeling instead, but if you know the downsides the upsides can be significant
- tudorw 3y agoIf I was given the choice of no medical help at all versus take the risk and ask AI, well, I guess I would take the chance?
- yieldcrv 3y agowomen and minorites receive notoriously dismissive diagnosis from human physicians which means over 75% of the population would chose this immediately and of the people that are content with their medical attention, a good portion of them would also choose this tl;dr yes
- nologic01 3y agoLet me make it more plain. Somebody you love is in an life threatening condition. You need advice what to do or you lose them. You have a one shot option to get advice from a human expert or a LLM. Which one you pick.
- newswasboring 3y agoWhat exactly are you trying to reason about with this hypothetical? Do you only buy processors which can be sent to space?
- yieldcrv 3y agoLLM on medical forum posts was more empathic and precise than the physicians. Thats the only data we have so far. I’ll pick the LLM. The humans are too random. I’ll take my chances with “hallucination” from something that’s also read every research paper in every language. Its just like Flowers for Algernon but even better.
- zwaps 3y agoIt said to be trained on 75 Billion tokens with 8 A100 in 7 days. However, in the paper they say that for each of the 5 epochs, they sample 512 tokens from each paper randomly. This would come down to an effective training size of 12 billion tokens.
- zwaps 3y agoAlso, if I am not reading this wrong, the paper shows that when Fine-tuning on a task, this additional Medical training (7 days on A100) essentially does nothing (benefit of maybe 1% over standard LLaMA). Zero shot comparisons are not shown. This looks like a failed experiment to me?
- thisancog 3y agoIf you have a hammer, everything looks like a nail. Medicine isn't a good nail to hammer with a LLM, though. I struggle to see how a stochastic token completion system, prone to hallucination, let alone all the attack vectors described over the last months and years, would do any good in the medical domain. A medical paper isn't just a digestible blog post or conversational op-ed, it's full of technical jargon and statistics that even require expert readers to do some unpacking in order to evaluate how the presented findings fit into a larger picture. Indeed, it's a skill and often a form of art to be able to read and understand medical papers, even from within one's field of expertise. Aside from that, medical knowledge is never truly established, it is always in flux, both in respect to ongoing research and in the skill that derives from decade long practice of negotiating this with often more complex and nuanced real-world situations, i.e. with the patients that come through the doctor's office doors. Doctors already face(d) the toll of diagnosis within a Google search's reach: misinformed, anxious and often outright overconfident patients often stand in their own way. That is not to say, that medical literacy and knowledge for the masses would be a detrimental thing, quite the contrary is true. However, with the high stakes faced in medicine, any misinformation, be it by Google or LLM can lead to grave harms.
- andy_ppp 3y agoWhy not? People said the same thing about programming and LLMs are usually quite helpful for a lot of tasks.
- ianpurton 3y agoIt's too late, it's out there now. It's just a matter of time before we see a HN Launch for a startup that's going to disrupt the medical industry with LLM.
- raffraffraff 3y agoI wonder would it be better off using sources like the Cochrane library? Smaller data set, meta-analysis. While doctors face google, patients face poor doctors. My father is 96, so lots of things have inevitably started to go wrong. Knees, heart, bladder, prostate, blood pressure. His doctor is absolutely shit. He started feeling light-headed a few months back and the doctor confidently pronounced that he's just old. My mother intervened. Turns out the handfuls of pills he's been prescribing my father included blood pressure pills, and if he ha bothered taking routine blood pressure readings the doctor would have realized that my father's blood pressure was dangerously low. He quit those pills and bounced back. His doctor has been useless, lazy and shamelessly apathetic for 10 years. Terrible doctors are everywhere. My wife's doctor told her that her extremely heavy periods just happen to some women. Oh, and that her chronic anemia and B12 deficiency were normal in some women, so he'd just keep giving her those shots. She demanded a scan, which revealed a fibroid but the doctor told her: "oh, you can just go on the pill!" It took another doctor to schedule a procedure to remove the fibroid. She's been fine for a decade since. But has met many women online who went through hell with untreated fibroids. One woman said that it ruined her life because her anemia was disabling, her heavy periods left her housebound, and when the fibroid finally reached the size of a grapefruit she had to have a hysterectomy. Up to that point her doctor called her a hypochondriac. Oh, and when my sister told her doctor that she had "new" pains 2 months after giving birth, he told her it was normal, take pain killers and go home. It took several attempts before she convinced him to schedule a scan. Turns out it was terminal cancer! I don't even know that catching it a few weeks earlier would have changed anything, but still. Err on the side of caution. The thing about a decent LLM doctor, if one were to exist, is that it won't get bored, apathetic, depressed, tired, hateful etc.