6 ms·
BioGPT: A language model pre-trained on large-scale biomedical literature
- amelius 4y agoIf I ask GPT3 to name a few good books on some expert topic, it makes up book titles. What prevents it from doing the same when asked about research papers?
- ilostmyshoes 4y agoLiterally nothing, these models are not good for scientific literature We saw similar stuff with the model that Meta released and quickly took down
- KMnO4 4y agoWhy are you so cynical? You’ve posted nothing but negative comments in this thread.
- ceejayoz 4y agoThere's every reason to be cynical about AI models of this nature being used for healthcare purposes at this point in time.
- scotty79 4y agoI don't really like the word cynical. It's usually used against people who are just accurate only not optimistic enough for somebody's taste.
- wittycardio 4y ago[dead]
- ilostmyshoes 4y agoBecause I'm a scientist and I've used several of these models. I see how they perform and it isn't even close to being useful, in fact in some cases it could be harmful if someone who doesn't know the material already believes the nonsense it spits out
- lostoldsession 4y agoThese models are infant technology. They are the chess playing computers of 1960. It's completely missing the forest for the trees to call it out as useless. It's not meant to be useful right now, it's exploratory tech meant to feel out the space. Wait till it gets developed to hold it to some standard. I should note though that progress has been far faster than computer chess. We went from "An AI wont beat a Go champion for at least 30 years" to "AI beats Go champion" in two years.
- ilostmyshoes 4y agoHow long after the chess computers of the 60s did it take to beat a human?
- lostoldsession 4y agoCertainly you're not implying that technological progress is linear?
- ilostmyshoes 4y agoNot linear but comparable Language and understanding abstract ideas is a bit more complicated than chess
- eatsyourtacos 4y ago>in fact in some cases it could be harmful if someone who doesn't know the material already believes the nonsense it spits out I get what you mean, but that's also kind of silly. First, almost no one who is not in the field is going to be even asking it the kind of questions you are. Second, even if they do and try to use that as reasoning etc.. well, then that moron would basically find any type of source (i.e., random youtube video) to prove their "point". So I honestly think the "harmful" argument is completely out the window. We already opened that pandoras box years ago with basically every piece of human knowledge online in so many places and people making videos on almost everything with all sorts of varying views. People say this about the coding aspect as well "oh this is going to hurt the world because people will be just asking for code and it will be wrong!" Ok.. but it pretty much sounds like they were just going to do that anyway. Let's focus on the positives of this infant technology that is still insanely impressive. We are just seeing the beginning, right?
- samuell 4y agoCheck out https://elicit.org https://elicit.org which is optimized for scientific papers (using the Semantic Scholar API somehow). Perhaps it is possible to get it to make some stuff up, but it has been pretty solid in my experience so far, at least to the point where it has been seriously useful.
- ilostmyshoes 4y agoWhile it does give some accurate summaries I noticed that it uses terms incorrectly. I can see this being a decent tool for finding papers but definitely not for creating accurate summaries, especially for use without in depth review. It's basically a 50/50 shot
- highwayman47 4y agoAs a non-dev, how do I use this?
- cinntaile 4y agoYou can't (yet). But there are pretty detailed instructions on how to get started with the model here https://github.com/microsoft/BioGPT https://github.com/microsoft/BioGPT
- ilostmyshoes 4y agoWhat could you even use it for?
- dsr_ 4y agoMaking up technobabble for a fictional story.
- RobotToaster 4y agoHow long until AI can write literature reviews and meta analysis on it's own?
- ilostmyshoes 4y agoAt least 40 years, these models just string words together without understanding their meaning or relationship
- FeepingCreature 4y agoSeems like an overreach. It clearly has some understanding, just not a very deep one.
- ilostmyshoes 4y agoUnderstanding basic grammar isn't even close to understanding scientific concepts and knowing how to connect them
- tiborsaas 4y agoIt took an astrophysics exam and did quite well in my opinion. It's worth watching this video: https://www.youtube.com/watch?v=K0cmmKPklp4 https://www.youtube.com/watch?v=K0cmmKPklp4 For human students, the median score was 75.6%, ChatGPT performed slightly worse than a Columbia non-science major undergraduate, 73.9%. Some of the errors was just messing up the arithmetics but it got the reasoning right.
- beepbooptheory 4y agoYeah but that test is for humans, its like saying a tiger is a pretty good wrestler because it did well in the wrestling ring, instead of just understanding that a tiger is big and brutal, which just happens to coincide with the sport. The tiger needs its own sport/test.
- ddalex 4y agoAnd now we start relying on machine hallucinations for people's health. I don't see any way this could go wrong.
- passion__desire 4y agotrust but verify as we do in real life.
- dsr_ 4y agoWhen a technology repeatedly fails to be trustworthy, stop trusting at all.
- ElectricalUnion 4y agoWhen humans repeatedly fails to be trustworthy, stop trusting at all.
- dsr_ 4y agoWell, yes. That's a major reason for why we stop being friends, or change jobs, or fire employees, or find someone else to vote for.
- passion__desire 4y agoLet me give you an example of what I meant by the parent. I asked ChatGPT, if there is any research area where Clifford Algebras and Expander graphs are related in any manner. It mentioned something about error-correcting codes. I verified it and sure enough there was a mathoveflow post about it.
- deleted 4y ago[deleted]
- lm28469 4y agoWhen we'll all be in "almost fully autonomous" cars and have to "trust but verify" it'll be really fun. You'll be at the wheel 0.1% of the time, lose most of your skills because you'll almost never need the, and only have to be in charge in the worst edge cases. Now apply that to other topics...
- distantsounds 4y agohey Microsoft, IBM called. They have this tech called "Watson" you might have heard of. Trained on tons of medical literature, even showed up on Jeopardy! a couple times. I can't imagine this turning out any different, though.
- kderbyma 4y agoHaving tried to parse textbooks to text and then TTS....I highly doubt they did a good job of imparting anything beyond regurgitation. Most complex subjects require images and diagrams, and complex relationship between structure and form.....
- isaacfrond 4y agoThe paper (https://arxiv.org/pdf/2210.10341.pdf https://arxiv.org/pdf/2210.10341.pdf) has some examples of its output. For example this one: The drug that can treat COVID-19 is The drug that can treat COVID-19 is hydroxychloroquine (HCQ), which has been shown to inhibit the replication of the SARS-CoV-2 virus in cell culture and in animal models of COVID-19, and has been approved by the US Food and Drug Administration (FDA) for the treatment of patients with COVID-19 in March 2020, and by the European Medicines Agency (EMA) for the treatment of patients with COVID-19 in April 2020, and by the European Medicines Agency (EMA) for the treatment of patients with COVID-19 in May 2020. But... The FDA has not approved the use of hydroxychloroquine for treating or preventing COVID-19 in humans. source: https://scdhec.gov/covid19/dangers-using-hydroxychloroquine-ivermectin-preventing-or-treating-covid-19 https://scdhec.gov/covid19/dangers-using-hydroxychloroquine-...
- fasteo 4y ago1 point by fasteo 0 minutes ago | next | edit | delete [–] >>> But... The FDA has not approved the use of hydroxychloroquine for treating or preventing COVID-19 in humans. source: It seems that it was approved[1] and later revoked[2] [1] https://www.accessdata.fda.gov/drugsatfda_docs/nda/2020/EUA% https://www.accessdata.fda.gov/drugsatfda_docs/nda/2020/EUA%... [2] https://www.fda.gov/media/138945/download https://www.fda.gov/media/138945/download
- binarymax 4y agoThis model looks much too small to be effective. They announced an open model of 1.5B parameters [0] coming soon. Contrast this with davinci-003 or BigScience BLOOM, both with ~176 params. I don't see how this will generate text that is comprehensive or useful. The open GPT models of that size aren't useable for text generation for anything serious. Adding the sensitivity of health/clinical I'd steer clear of this. Even BLOOM's model card specifically calls out biomedical as a misuse of the model [1]. [0] https://github.com/microsoft/BioGPT#news https://github.com/microsoft/BioGPT#news [1] See "Out-of-scope" in https://huggingface.co/bigscience/bloom https://huggingface.co/bigscience/bloom
- deleted 4y ago[deleted]
- samuell 4y agoI'm surprised at how little attention has been given to elicit.org that is already trained on the scientific literature, and not limited to just the biomedical ditto: https://elicit.org https://elicit.org Oh, and it provides real references! (also, summaries both of the top papers answering your question, as well as each of the top papers individually, etc etc)
- zora_goron 4y agoI've been working on integrating GPT-3 into a clinical reference tool by "grounding" its responses using data from the NCBI StatPearls clinical handbook -- doing that seems to lead to accurate and relevant responses much of the time, but still occasionally returning partially-correct or irrelevant information. [0] demo - https://labs.cactiml.com/medqa https://labs.cactiml.com/medqa [1] some screenshots and details - https://twitter.com/samarthrawal/status/1620545601296564228 https://twitter.com/samarthrawal/status/1620545601296564228
- bertman 4y agoI find it a bit curious that their "large" (ie. based on GPT2 XL) model only outperforms their standard model by less than 3% on PubMedQA.
- testbiogpt 4y agohow design mRNA sequence for vaccine
- voidcrab 4y agoI went through the trouble of installing it, and it's super bad for some tasks. The model is apparently too small. Examples of what I tried: > The SNP rs9651229 predicts multiple traits for humans. The predictions depend largely on the genotype. The TT genotype predicts the following traits: This gives the same result as when I typed CC genotype instead. > The following list contains all SNPs that predict depression: This didn't list any SNP at all, and instead listed a bunch of things such as bipolar disorder