7 ms·
LLMs approach expert-level clinical knowledge and reasoning in ophthalmology
- dedosk 3y agobehind paywall
- mtmail 3y agoBut please don't post complaints about paywalls. https://news.ycombinator.com/newsfaq.html https://news.ycombinator.com/newsfaq.html
- danesparza 3y agoHow do I read the article?
- batch12 3y agohttps://news.ycombinator.com/item?id=40074620 https://news.ycombinator.com/item?id=40074620 And https://news.ycombinator.com/item?id=40074640 https://news.ycombinator.com/item?id=40074640
- mewpmewp2 3y agoSo it doesn't match?
- lionkor 3y agoIf smart people made language,yea, youd be right. But most people speaking english dont have the capacity, so no, "all but" means the "completely", for no logical reason.
- SiempreViernes 3y agoNo, in this case "all but" means "as good as, or worse, than the bottom 2/5 of doctors tested on a written exam", see table 1.
- ethagknight 3y agoI know you are being funny, but I would just say this title is an incorrect use of the idiom. I’m assuming the intent is to have the words “model matches doctors” as clickbait. “All but” idiom means “very nearly”, not completely.
- mewpmewp2 3y agoIt means very nearly, but I also hate how it means that, because as a foreigner it just sounds if somebody "they did all, but the work they were assigned", it means they went in completely different direction or slacked off to me.
- mdrzn 3y agohttps://archive.is/w3Sk0 https://archive.is/w3Sk0
- monkeydust 3y agoLink to paper: https://journals.plos.org/digitalhealth/article?id=10.1371/journal.pdig.0000341 https://journals.plos.org/digitalhealth/article?id=10.1371/j...
- gwern 3y ago> FRCOphth Part 2 questions were sourced from a textbook for doctors preparing to take the examination [17]. This textbook is not freely available on the internet, making the possibility of its content being included in LLMs’ training datasets unlikely [1]. I can't believe they're serious. They didn't even write any new questions?
- hyperorca 3y agoFunny thing is that this textbook can be easily found on LibGen. I do not know a lot about LLM datasets, but they probably include books from these shadow libraries, right ?
- rdedev 3y agoNot just that but wouldn't the references for the textbook be mostly research papers that would be freely available?
- cameldrv 3y agoNice catch. GPT-3 at least was trained on “Books2” which is very widely suspected of containing all of Libgen and Zlibrary. If the questions are in Libgen, this whole paper is invalid.
- pants2 3y agoConsidering that textbooks are probably the single highest quality source of training data for LLMs, I would be very surprised if OpenAI wasn't buying and scanning textbooks for their own training data (including books that aren't even in Books2).
- RandomLensman 3y agoOn questions, not actually examining eyes, though.
- SiempreViernes 3y agoIt also scored 0% on ethics :D (Was just one question though, 8/10 humans got that one right)
- JaimeThompson 3y agoMaybe it enabled modern MBA mode for that question ;)
- bidandanswer 3y agoOn that note, legacy computer vision models have exceeded humans at identifying pathologies from images of retinas, etc., for probably decades now.
- greybox 3y agoSo. . . It doesn't work
- hestefisk 3y ago“What this work shows is that the knowledge and reasoning ability of these large language models in an eye health context is now almost indistinguishable from experts,” said Arun Thirunavukarasu, the lead author of a paper on the findings published in PLOS Digital Health journal.” FTFA.
- boyka 3y agoFTFA?
- alex_suzuki 3y agoFrom The F**ing Article, presumably.
- Gorath 3y agoIt’s ok, you can swear here.
- bheadmaster 3y agoIt doesn't hurt anybody! Fuck, fuckity, fuck fuck fuck.
- woodrowbarlow 3y agoi've been assuming, on HN, it stands for "the featured article".
- nicklecompte 3y agoWhat this work actually shows is that a bunch of scientists are ignorant about how LLMs work, but are rushing to publish papers about them anyway. It is ridiculous for Thirunavukarasu to draw this conclusion from GPT-4's performance on a written ophthalmology exam. From the good folks at AI Snake Oil[1] > Memorization is a spectrum. Even if a language model hasn’t seen an exact problem on a training set, it has inevitably seen examples that are pretty close, simply because of the size of the training corpus. That means it can get away with a much shallower level of reasoning....In some real-world tasks, shallow reasoning may be sufficient, but not always. The world is constantly changing, so if a bot is asked to analyze the legal consequences of a new technology or a new judicial decision, it doesn’t have much to draw upon. In short, as Emily Bender points out, tests designed for humans lack construct validity when applied to bots. > On top of this, professional exams, especially the bar exam, notoriously overemphasize subject-matter knowledge and underemphasize real-world skills, which are far harder to measure in a standardized, computer-administered way. In other words, not only do these exams emphasize the wrong thing, they overemphasize precisely the thing that language models are good at. Also[2]: > Undoubtedly, AI and LLMs will transform every facet of what we do, from research and writing to graphic design and medical diagnosis. However, its current success in passing standardized test after standardized test is an indictment of what and how we train our doctors, our lawyers, and our students in general. ChatGPT passed an examination that rewards memorizing the components of a system rather than analyzing how it works, how it fails, how it was created, how it is maintained. Its success demonstrates some of the shortcomings in how we train and evaluate medical students. Critical thinking requires appreciation that ground truths in medicine continually shift, and more importantly, an understanding how and why they shift. Perhaps the most important lesson from the success of LLMs in passing examinations such as the USMLE is that now is the time to rethink how we train and evaluate our students. [1] https://www.aisnakeoil.com/p/gpt-4-and-professional-benchmarks https://www.aisnakeoil.com/p/gpt-4-and-professional-benchmar... [2] https://journals.plos.org/digitalhealth/article?id=10.1371/journal.pdig.0000205 https://journals.plos.org/digitalhealth/article?id=10.1371/j...
- thiago_fm 3y agoLots of people publish papers to get cited and be first in the field, without any rigorous research behind it, to try to net them a good career. Mostly recently the best to do is to ignore this, and soon enough the things that are true will emerge, without academia. The media is the worst, that tends to look for whatever research that would be click-baity. It's a real waste of time to follow FT and others.
- aantix 3y agoThis doesn’t surprise me. I find most doctors are terrible at debugging root cause. They’re simply working through a protocol of most common symptoms/cause and if your ailment doesn’t fit these narrow requirements, you’re screwed. Your 10 minute consult has ended and they’re not thinking about the problem anymore.
- mewpmewp2 3y agoThis is exactly what a doctor told me during my health check up as well. He casually hit up a monologue on how dumb doctors are. I didn't really ask, but I wonder if he does this monologue to every patient he has. It seemed like for years he had developed this theatrical and interactive comedy piece for all the patients he was doing the check up on.
- RandomLensman 3y agoCommon things are common, so going through common things first doesn't strike me as wrong. If someone suffers from something really exotic then I'd argue that is what specialists are for. The 10 minute window is more an external/financial constraint, I'd say.
- wredue 3y agoOne of the main issues doctors face today is that everyone is really fat, and being fat leads to every diagnosis under the sun. People don’t like being told that being fat is causing their problem (when it almost definitely is), and this is creating a lot of distrust in doctors. I mean, you see these examples of it being more than just overweight, but that’s the minority, and everyone being fat unfortunately contributes to increasing the rate of these misdiagnosis.
- aantix 3y agoI saw multiple cardiologists in Lincoln, NE, and San Francisco, CA. Multiple nephrologists throughout the 25 years. These are the "specialists" I was referred to from my general physician. I had to work side-by-side with them to manage my condition. I had lost significant weight in the past, but it never reduced my blood pressure medication intake. But there was nothing special about these doctors. They simply had no answers beyond the typical. The equivalent for software would be that a bug report comes in and the engineer responds with "have you tried rebooting the servers?" Just superficial, unhelpful recommendations.
- svaha1728 3y agoAsk the model to provide an image of its answers written on a chalkboard. Or, ask it to provide you an image of pathological eye conditions and look at the text labels.
- sethammons 3y agoMy wife had gone through half a dozen doctors or more over a decade, giving them the same description of her issues. They never correctly diagnosed the issue. We typed her descriptions into chatgpt and it suggested multiple sclerosis. It was right. It must be what other folks feel when they talk to tech support, but I've never, not once, been impressed by a doctor and have found them lacking in nearly every interaction.
- xanderlewis 3y agoThe best doctors are the ones who listen carefully to the patient before dispensing their expert opinion. Very few do. Perhaps that’s just because they’re jaded from having to deal with the public all day long…
- joelfried 3y agoI agree with you - listening is very important. I worked in an ophthalmology practice when I was quite young many years ago. The head of the practice often had over 50 patients a day (one of my responsibilities was pulling the folders for all daily patients from the wall-o'-folders; this was in the age everything was on paper). 50 patients over 8 hours gives you less than 10 minutes per patient. I suspect a nontrivial part of the problem is being rushed.
- azinman2 3y agoRegulation could help solve that. You shouldn’t see so many patients in a day unless you also double your working hours. I’m surprised this was many years ago. Now this pressure exists because medical school is so expensive and people need to pay off their loans. Want to see improvements in care? Make medical school free / cheap.
- joelfried 3y agoIt's also extremely long. Assuming a 3 year residency the average Doctor doesn't even start practicing until 31.[1] So you've not just got a house's worth of debt without a house, you've also lost a decade to training. I'd love improvements in care, but nobody is going to fix this problem. It's too lucrative for everyone else involved. [1] https://averagedoctor.com/the-average-age-of-doctors-graduating-medical-school/ https://averagedoctor.com/the-average-age-of-doctors-graduat...
- henriquenunez 3y agoI do not trust doctors in general (unless they are scientists); and I also think that clinical knowledge is not that hard in ophthalmology…
- troq13 3y agoWell it is not going anywhere near my eyes.
- victorbstan 3y ago[flagged]
- rafaelero 3y agoI am very bullish for GPT-5. It's going to surpass even the best diagnosticians we have right now. And we will all have access for $20 a month. Crazy and exciting times.
- squeegmeister 3y agoThere are reasons to believe we’ve reached a plateau with gpt 4. Check out Gary Marcus’s Substack
- rafaelero 3y agoUnless he has access to GPT-5 his opinion is worthless.
- staticman2 3y agoPlease practice what you preach and delete your comment about how great GPT-5 will be.
- rafaelero 3y agoI am just extrapolating on a trend that started in 2018. I think I am well justified in my opinion, but feel free to ignore the trend and listen to substack fools.
- daveguy 3y agoDo you think hallucinations will be solved with GPT-5? If so, that would be an amazing breakthrough. If not, it still won't be suitable for medical advice.
- rafaelero 3y agoIt will certainly decrease. Also, there are multiple ways to deal with hallucinations. You can sample GPT-4 not once, but 10, 100, 1000 times. The chances of it hallucinating the same things asymptotically reaches 0. It all depends on how much money you are willing to invest in getting the right opinion, which in the field of medicine, can be quite a lot.
- Euphorbium 3y agoDoctors cannot be automated soon enough, and nothing of value will be lost.
- VeejayRampay 3y agodoctors are by and large terrible trained to condescend and ignore / infantilize patients, generally thinking highly of themselves, never providing explanations for anything, thinking they're better than everyone, I can't wait for them to get humbled
- poliphilo 3y ago[dead]