8 ms·
GPT-5o-mini hallucinates medical residency applicant grades
- medicalthrow 1y agoHi HN, submitting from a burner since I'm an applicant this current medical residency admissions cycle. I thought it was interesting to show the real world implications of using LLMs to extract information from PDFs. For context, thalamus is a company that handles the "backend" for residency programs and all the applications they receive (including handling who to invite for interviews, etc). One of the more important factors in deciding applicant competitiveness is their medical school performance (their grades), but that information is buried in PDFs sent by schools (often not standardized). So this year, they decided to pilot a tool that would extract that info (using "GPT-5o-mini": https://www.thalamusgme.com/blogs/methodology-for-creation-and-processing-of-a-novel-transcript-normalization-tool-in-cortex-application-screening-and-review-platform https://www.thalamusgme.com/blogs/methodology-for-creation-a...). Some programs have noticed there is a discrepancy between extracted vs reported grades (often in the direction of hallucinating "fails") and brought it to the attention of thalamus. Unfortunately, it doesn't look like the main company is discontinuing usage of the tool. Regardless, given that there have been a number of posts looking into usage of LLMs for numerical extraction, I thought this story useful would be a cautionary tale. EDIT: I put "GPT-5o-mini" in quotes since that was in their methodology...yes, I know the model doesn't exist
- philipallstar 1y agoThank you for sharing this. It's astonishing that places like this will do almost anything rather than create a simple API to ingest data that could easily be pushed automatically.
- daemonologist 1y agoThe trouble is getting people to use your API - in this case med schools, but it can be much, much worse (more and smaller organizations sending you data, and in some industries you have a legal obligation to accept it in any format they care to send).
- simonw 1y agoI imagine they would love to create a simple API for this, but the problem is convincing thousands of schools to use that API. If all you can get are PDFs, attempting to automatically extract information from those PDFs is a reasonably decision to make. The challenge is doing it well enough to avoid these kind of show-stopper problems.
- a-dub 1y agothey're essentially an ATS SAAS for medical school, if they have enough schools or enough prestigious schools, they can ask for whatever they want and the applicant schools would oblige. cheeky way to make it happen overnight: give a slight advantage to transcripts that are submitted digitally- the conversion would be complete within months.
- avarun 1y agoIf you want to get sued, sure.
- deleted 1y ago[deleted]
- aprilthird2021 1y ago> this year, they decided to pilot a tool that would extract that info (using "GPT-5o-mini": https://www.thalamusgme.com/blogs/methodology-for-creation-a https://www.thalamusgme.com/blogs/methodology-for-creation-a...). Mind-boggling idea to do this because OCR and pulling info out of PDFs has been done better and for longer by so many more mature methods than having an LLM do it
- b112 1y agoWelcome to the world of greybeards, baffled by everyone using AWS at 100s to 100000s of times the cost of your own servers.
- lazystar 1y agospectre/meltdown, finding out your 6 month order of ssd's was stolen after opening empty boxes in the datacenter, and having to write RCA's for customers after your racks go over the PSU's limit are things ya'll greybeards seem to gloss over in your calculations, heh
- mattnewton 1y agoNit, I’d say as someone who spend a fair amount of time doing it in the life insurance space, actually parsing arbitrary pdfs is very much not a solved problem without LLMs. Parsing a particular pdf is, at least until they change their table format or w/e. I don’t think this idea is totally cursed, I think the implementation is. Instead of using it to shortcut filling in grades that the applicant could spot check, like a resume scraper, they are just taking the first pass from the LLM as gospel.
- simonw 1y agoRight - the problem with PDF extraction is always the enormous variety of shapes that data might take in those PDFs. If all the PDFs are the same format you can use plenty of existing techniques. If you have no control at all over that format you're in for a much harder time, and vLLMs look perilously close to being a great solution. Just not the GPT-5 series! My experiments so far put Gemini 2.5 at the top of the pack, to the point where I'd almost trust it for some tasks - but definitely not for something as critical as extracting medical grades that influence people's ongoing careers!
- alexpotato 1y agoIt's amazing how much of "inter organization information flow" still happens over PDFs and/or just FTP'ing files around. A couple jobs ago at a hedged fund, I owned the system that would take financial data from counterparties, process it and send it to internal teams for reconciliation etc. The spectrum went from "receive updates via SWIFT (as in financial) protocol" to "small oil trading shop sending us PDFs that are different every month". As you can imagine, the time and effort to process those PDFs occasionally exceeded the revenue from the entire transaction with that counterparty. As others have pointed out: yes, the overall thrust of the industry is to get to something standardized but 100% adoption will probably never happen. I write more about the FTP side of things in the Twitter thread below: https://x.com/alexpotato/status/1809579426687983657 https://x.com/alexpotato/status/1809579426687983657
- jaggederest 1y ago> the time and effort to process those PDFs occasionally exceeded the revenue from the entire transaction with that counterparty. I'm interested in what the conditions were that didn't let you reject those kind of transactions, or blacklist them for the future. We hear about companies firing/banning unprofitable customers sometimes, surprised it doesn't happen more often honestly.
- animalmother 1y agoHi wondering if you could message me at shane.shifflett@dowjones.com or via signal at 929 638 0009? https://www.wsj.com/news/author/shane-shifflett https://www.wsj.com/news/author/shane-shifflett
- apt-apt-apt-apt 1y agoYou are so brave. I get like 8 spammers calling me daily about loans like I owe them money, and that's without blasting my phone number out to the internet.
- gambiting 1y agoIf you're not using it yet, then I recommend enabling the call screening feature on your phone, it has basically reduced my number of spam calls down to zero. It's available on iPhones and pixels and Samsung phones(and probably others?)
- mandeepj 1y agoI have it, but you still have junk in missed calls and voicemails.
- motbus3 1y agoI started receiving spam calls lately again but I was just denying the calls. It was with a predictable frequency few times a week. weeks ago I decided to use screening call, and maybe it was a coincidence, after the first usage I've not no more calls from them
- lozenge 1y agoWhy don't they just email a form after/when you apply and you fill in all the grades in a structured data way? How many grades are we talking about here. Then the PDF would just be the proof that your grades were real.
- simonw 1y agoBecause you'd have to get thousands of schools to agree to using the same format.
- lozenge 1y agoDoes the student not have access to the grades? As they are applying to medical school, a few hours of drudgery form filling will still be the easiest part of the process.
- medicalthrow 1y agoIt's a bit complicated. Each school has their own grading system (some pass fail, others four tiered, others full letter grades). Additionally, there are reported distributions for each grade. Lastly, there's sometimes a summary statement at the end that usually says "X student was 'superlative'" and then a table at the end that says 'superlative' means top X% of class. On top of that, students may not get their full dean's letter that says all of this stuff. Basically, self reporting is very difficult to do given the amount of variability in grade reporting.
- zeven7 1y agoMaybe have the students verify their own grades extracted by the LLM?
- doctorpangloss 1y agoHow should medical residency work? Like how should admissions work, is the match doing what you would want it to do, is there a radical alternative, etc? You have our attention! Don’t tell me the grades should be gathered accurately. Obviously. Tell me something bigger.
- temp_correct_02 1y agoThere is no such thing as GPT-5o-mini, or GPT-5o. Concerning that the methodology seems to repeat the same error, not just the submitted title.
- notfromhere 1y agothey probably mean gpt-5-low. but the small models are bad for parsing data where the data has strong implications
- busssard 1y agohttps://www.thalamusgme.com/blogs/methodology-for-creation-and-processing-of-a-novel-transcript-normalization-tool-in-cortex-application-screening-and-review-platform https://www.thalamusgme.com/blogs/methodology-for-creation-a... they actually write it: > For this cycle, we have refined our model architecture, expanded the catalog of medical schools and grading schemas, and upgraded to include the GPT-5o-mini model for increased accuracy and efficiency. Real-time validation has also been strengthened to provide programs with more reliable percentile and grade distribution data. Together, these enhancements make transcript normalization an even more powerful tool to support fair, consistent, and data-driven review in the transition to residency.
- beernet 1y agoNothing new to see here. If you are still surprised by model hallucinations in 2025, it might be time for you to catch up or jump on the next hype bandwagon. Also, they reacted well: > Once confirmed, we corrected the extracted grade immediately. > Where the extracted grade was accurate, we provided feedback and guidance to the reporting program or school about its interpretation and the extraction methodology. I still dislike the term "hallucinations". It comes across like the model did something wrong. It did not, as factually wrong outputs happen per design.
- softwaredoug 1y agoIt's true, but I think people have a misunderstanding that if you add search / RAG to ground the LLM, the LLM won't hallucinate. When in reality the LLM can still hallucinate, just convincingly in the language of whatever PDF it retrieved.
- bigzyg33k 1y agoRAG certainly doesn't reduce hallucinations to 0, but using RAG correctly in this instance would have solved the hallucinations they describe. The purpose of the system described in this post is OCR inaccuracies - it's convenient to use LLMs for OCR of PDFs because PDFs do not have standard layouts - just using the text strings extracted from the PDFs code results in incorrect paragraph/sentence sequencing. The way they *should* have used RAG is to ensure that subsentence strings extracted via LLM appear in the PDF at all, but it appears they were just trusting the output without automated validation of the OCR.
- eoinbmorg 1y agoIs RAG the right tool for this? My understanding was that RAG uses vector similarity to compare queries (the extracted string) versus the search corpus (the PDF file) using vector similarities. The use case you describe is verification, which sounds like it would be better done with an exhaustive search via string comparison isntead of vector similarities. I could be totally wrong here.
- rarisma 1y ago5o mini?
- johnfn 1y agoGuess it hallucinated the model name as well.
- tripplyons 1y agoI'm assuming they mean gpt-5-mini. I'm honestly surprised how many people I've heard say "5o".
- ikeashark 1y ago>AI hallucinates >Look inside >GPT-4o-mini
- Aurornis 1y agoFrustrating that their official recommendation is to verify the grades manually. If a tool is designed to extract the grades for easy access, do we really believe that the end users will then verify the grades manually to confirm the output? If they’re doing that, why use the tool at all? Maybe the tool can extract what it believes is the grades section and show a screenshot for a human to interpret.
- landl0rd 1y agoBecause the contract has already been signed, they can't guarantee it works right, and they don't want to be open to lawsuits. "You, mister wrongly-denied applicant, cannot sue us; we specifically told them to check all grades manually!"
- SketchySeaBeast 1y agoThis is why this particular emperor has no clothes. They keep trying to jam AI into stuff to make it "easier", but the LLMs, by their very nature do the tasks in lossy or incorrect ways. Imagine if Microsoft had sold Excel with a "be sure to verify all the calculations" caveat.
- GoatInGrey 1y ago> If they’re doing that, why use the tool at all? Because the people purchasing the tool aren't the ones who will actually use it. The former get a "Deployed AI tooling to X to increase productivity by X%" on their resume. The latter get left to deal with the mess.
- softwaredoug 1y agoI see _even with search/RAG_ LLMs hallucinate. They just hallucinate more convincingly in the language of the documents you retrieved. So you really have to double check when researching information that really matters.
- owenthejumper 1y agoThis sucks. Residency match is stressful as it is, and adding systems like these just make the experience even worse for the applicants. Source: spouse matched in 2018. It was one of the most stressful periods of our lives.
- OldGreenYodaGPT 1y ago[flagged]
- Narciss 1y agoNot only did the AI hallucinate the applicant grade, but also the model name! GPT-5o-whatever ain’t a thing. The irony is sweeeeet
- lukeschlather 1y agoUsing a mini model for this seems grossly irresponsible. I've been doing some work testing models for similar extraction tasks (nothing where a failure affects someone's grade or anything) and gpt mini / Gemini flash simply can't do this sort of thing. Using anything less than the highest model with reasoning, you're guaranteed to get this sort of thing happening. It is very tempting to do it, obviously, with the cost difference, but it's not worth it. But on the other hand, people talk about LLMs with a broad brush and I don't know, there's still testing but I would be surprised to hear that GPT-5-pro with thinking had an issue like this.
- bilekas 1y agoAm I crazy or has text parsing been mastered long before AI. Why is GPT being used in this scenario in the first place ?
- hansonkd 1y agoIt seems like a default mode for AI should be to generate repeatable Regex for text extraction.
- tdeck 1y agoUnfortunately many PDFs don't even internally represent text in a contiguous way.
- tdeck 1y agoBecause it's less effort to get an MVP set up. Instead of having to test on a bunch of different PDFs and figure out how to address the right location in the text, just write a paragraph asking the LLM to do it. Of course, there are certain drawbacks...
- mattnewton 1y agoBecause it’s easier than asking for a consistently formatted data from all the sources who just output random PDFs. Basically this is a coordination / people problem we’re papering over with a fancy engineering solution. Many such cases.
- deleted 1y ago[deleted]
- hluska 1y agoNot in PDF.
- bilekas 1y agoNo, not in the PDF spec, but are we allowed process images of every page, text adaption, etc. Where does GPT come in ?
- dabei 1y agoNothing new to see here. Human also hallucinates, as you can tell from the model name.
- Yizahi 1y agoLLM can't hallucinate. Correct phrase would be "GPT-5o-mini generates medical residency applicant grades". Everywhere you see word hallucinate in regards of a program output, it should be replaced with generate for clarity.
- noboostforyou 1y agoIf you're being 100% literal, sure. But language evolves and it's the accepted term for the concept. OpenAI themselves uses the phrase - https://openai.com/index/why-language-models-hallucinate/ https://openai.com/index/why-language-models-hallucinate/
- Yizahi 1y agoOpenAI are the last people who I would take as a reference, because they are financially motivated to keep the charade of a "thinking" LLM or so called "AI". That's why they are widely using anthropomorphic terms like "hallucination" or "reasoning" or "thinking", while their computer programs can't do neither of those things. LLM companies sometimes even expose their hypocrisy. My favorite example for now is when Antropic showed in their own paper that asking LLM how it "reasoned" through calculating a sum of numbers doesn't match reality at all, it's all generated slop. This is why it is important that users (us) don't fall into the anthropomorphism trap and call programs what they are are and what they really do. Especially important since general populace seems to be deluded by the OpenAI and Anthropic aggressive lies and believe that LLMs can think.
- simonw 1y agoWhere did the "GPT-5o-mini" in this headline come from? That's not a real model name: there's GPT-5-mini and GPT-4o-mini but no GPT-5o-mini. UPDATE: Here's where the GPT-5o-mini came from: https://www.thalamusgme.com/blogs/methodology-for-creation-and-processing-of-a-novel-transcript-normalization-tool-in-cortex-application-screening-and-review-platform https://www.thalamusgme.com/blogs/methodology-for-creation-a... - via this comment: https://news.ycombinator.com/item?id=45581030 https://news.ycombinator.com/item?id=45581030 That said, I've been disappointed by OCR performance from the GPT-5 series. I caught it hallucinating some of the content for a pretty straight-forward newspaper scan a few weeks ago: https://simonwillison.net/2025/Aug/29/the-perils-of-vibe-coding/ https://simonwillison.net/2025/Aug/29/the-perils-of-vibe-cod... Gemini 2.5 is much more reliable for extracting text from images in my experience.
- deleted 1y ago[deleted]
- kbyatnal 1y agoSchool transcripts are surprisingly one of the hardest documents to parse. The thing that makes them tricky is (1) the multi-column tabular layouts and (2) the data ambiguity. Transcript data is usually found in some sort of table, but they're some of the hardest tables for OCR or LLMs to interpret. There's all kinds of edge cases with tables split across pages, nested cells, side-by-side columns, etc. The tabular layout breaks every off-the-shelf OCR engine we've run across (and we've benchmarked all of them). To make it worse, there's no consistency at all (every school in the country basically has their own format). What we've seen help in these cases are: 1. VLM based review and correction of OCR errors for tables. OCR is still critical for determinism, but VLMs really excel at visually interpreting the long tail. 2. Using both HTML and Markdown as an LLM input format. For some of the edge cases, markdown cannot represent certain structures (e.g. a table cell nested within a table cell). HTML is a much better representation for this, and models are trained on a lot of HTML data. The data ambiguity is a whole set of problems on its own (e.g. how do you normalize what a "semester" is across all the different ways it can be written). Eval sets + automated prompt engineering can get you pretty far though. Disclaimer: I started a LLM doc processing company to help companies solve problems in this space (https://extend.ai/ https://extend.ai/).
- gallerdude 1y agoI wonder if they're using reasoning? It usually eliminates these types of errors
- nisten 1y agoWhile I don't want to discount the work of any physician-founded org knowing the pain they go through from working with them after they've seen 18 patients in a days work, this still just just looks like bad software. With no testing, no internal bench. Did you do some kind of zod schema, or compare the error rate of how different models perform for this task? Did you bother setting up any kind of json output at all? Did you add a second validation step with a different model and then compared their numbers are the same? It looks like no, they just deferred to authority the whole thing. Technically theres no difference between them saying that gpt5-mini or llama2-7b did this. Literally every single llm will make errors and hallucinate. It's your job to put all the scaffolding around to make sure it doesn't or that it does a lot less than a skilled human would. So then have you measured the error rate or maybe tried to put some kind of error catching mechanism just like any professional software would do?
- simonw 1y agoLots of comments in here that seem to have missed that this is about using vision-LLMs for OCR. This makes it a slightly different issue from "hallucination" as seen in text based models. The model (which I think we can assume is GPT-5-mini in this case) is being fed scanned images of PDFs and is incorrectly reading the data from them. Is this still a hallucination? I've been unable to identify a robust definition of that term, so it's not clearly wrong to call a model misinterpreting a document a "hallucination" even though it feels to me like a different category of mistake to an LLM inventing the title of a non-existent paper or lawsuit.
- lysecret 1y agoThese kinds of errors have always existed and will always exist there is no perfect way to extract info from documents like this.
- simonw 1y agoThe models really are getting better though. Compare Gemini 1.5 and Gemini 2.5 on the same PDF document (I've done this a bunch) and you can see the difference. The open question is how much better they need to get before they can be deployed for situations like this that require a VERY high level of reliability.
- lysecret 1y agoI fully agree. My point was more a lot of commenters seem or implicitly compare the llm based approach with some “better” or “simpler” approach which really doesn’t exist from my estimation LLMs are sota for this kind of extractions (though they still have issues).
- hoosieree 1y agoPeople don't respect the chasm between "obviously no mistakes" and "no obvious mistakes".
- fxwin 1y ago> this is about using vision-LLMs for OCR Is it? To me it sounds like they do OCR first, then extract from the result with LLM: "Cortex uses automated extraction (optical character recognition (OCR) and natural language processing (NLP)) to parse clerkship grades from medical school transcripts."
- powersurge360 1y agoI keep circling this with AI and I'm not really sure what to do with it. They mention that the AI is meant to be used as reference only in the linked article but what does that actually mean? Who is checking who? Is the AI filling out the data from what it sees in the PDF and the user is expected to check it or is the user filling out the data and the AI is expected to check it? Is the cost of AI useful if all you're doing is something like 'linting' the extraction? How do you guarantee that people really, truly, are doing the same work as before and not just blindly clicking 'looks good'. What is the value of the AI telling you something when you cannot tell if it is lying?
- omnicognate 1y agoYeah, I've seen this "for reference only" wording in many places, often used as a sort of disclaimer on stuff that could be wrong, but I have absolutely no idea what it means in that context. To me "reference" implies comprehensive, high quality information that I can refer to when I need to know some obscure detail of something. Is there some legal context in which this phrase has a specific meaning, perhaps?
- bobbyprograms 1y agoSeems like hallucination will always be an issue for predict the next word training. Maybe we need to rethink pretraining .
- lysecret 1y agoI see your point here but please take a look at the “standard” unstructured pdf extraction algos they have a lot of problems as well. Llm based extraction is still (on avergad) a big improvement.
- moomoo11 1y agoSemi-related but Sonnet 4.5 drives me absolutely insane. I tell it a date, like March 2024 as the start, and October 2025 as the current month. It still thinks that is 7 months somehow... and this is Anthropic's latest model..
- socrateswasone 1y agoIt's predicting the next token by statistical approximation. Hallucination vs fact is an ad-hoc distinction we impose on the result to suit our purpose.
- deleted 1y ago[deleted]
- lawlessone 1y agoI thought this was supposed to be AGI?
- anotherpaulg 1y agoI regularly use LLM-as-OCR and find it really helpful to: 1. Minimize the number of PDF pages per context/call. Don't dump a giant document set into one request. Break them into the smallest coherent chunks. 2. In a clean context, re-send the page and the extracted target content and ask the model to proofread/double-check the extracted data. 3. Repeat the extraction and/or the proofreading steps with a different model and compare the results. 4. Iterate until the proofreadings pass without altering the data, or flag proofreading failures for stronger models or human intervention.
- SketchySeaBeast 1y agoWhat's the typical run for you cost?
- vishdipsheet 1y agoWow. Never ceases to amaze me how some people in these comment sections remain blind to the power of Artificial Intelligence (AI). Have you not tried prompting the model correctly? My startup gets 0 hallucinations on the latest iteration of Claude Sonnet using a custom proprietary reflecting RAG framework inspired by ontology.
- hluska 1y agoIt never ceases to amaze me when startup founders claim that every problem is the same. Some use cases (like parsing text out of PDF) can’t be distilled down to a prompt.
- constantcrying 1y ago>Reviewers are strongly encouraged to verify all information against the applicant’s official PDF transcript. This reminder is also displayed directly within the product. This is not how this works. You know people will not do this. In fact the whole value proposition hinges on people not doing this. If the information needs to be verified by a human, then it takes more time than just going through the document. If your product can not be trusted, then it can not be used to make important decisions. Pushing the responsibility to not use your product on the user is absurd and does not make your actions any less negligent.
- deleted 1y ago[deleted]
- deleted 1y ago[deleted]