5 ms·
Ontario auditors find doctors' AI note takers routinely blow basic facts
- zOneLetter 5mo agoAnecdotally, we use an LLM note-taker at work for meetings. I had to intervene recently because our CIO was VERY angry at our vendor for something they promised to do and never did. He wasn't at the meeting where the "promise" was made. I was. They never promised anything, and the discussion was significantly more nuanced than what the LLM wrote in the detailed summary. In other cases, I have seen it miss the mark when the discussion is not very linear. For example, if I am going back and forth with the SOC team about their response to a recent alert/incident. It'll get the gist of it right, but if you're relying on it for accuracy, holy hell does it miss the mark. I can see the LLM take great notes for that initial nurse visit when you're at the hospital: summarize your main issue, weight, height, recent changes, etc. I would not trust it when it comes to a detailed and technical back-and-forth with the doctor. I would think for compliance reasons hospitals would not want to alter the records and only go by transcripts, but what do I know...
- Ferret7446 5mo agoTranscription works pretty well in my experience, and the transcripts should be treated as the ground truth in such cases.
- fc417fc802 5mo ago> I would think for compliance reasons hospitals would not want to alter the records and only go by transcripts, but what do I know... I'm puzzled by this as well. Why not just generate a transcript and be done with it? If it's a particularly long transcript that's being referenced repeatedly for whatever reason let the humans manually mark it up with a side by side summary when and where they feel the need. At least my experience is that usually these sort of interactions don't have a lot of extraneous data that can be casually filtered out to begin with. The details tend to matter quite a lot!
- tempest_ 5mo agoI mean the reasons are the same AI is being pushed everywhere. The businesses offering these services want to say "we are using AI" to their stake holders and the government committees who approve this shit don't have the skills or knowledge to evaluate the effectiveness in addition to the fact they likely don't even use the tools they have approved for use.
- toraway 5mo agoI recently left my mom a voicemail saying happy Mother’s Day with normal human boilerplate of sorry I missed you, feel free to give me a call back tonight or we can talk tomorrow, either is fine by me whatever works best for you, hope we can talk soon, love you, bye. She called me back later that night and we chatted for bit and then she paused and sort of uncertainly was like “So… was there something you were needing to tell me?” And I was completely baffled and was like “Uhhhh I don’t think so…?” She then explained the notification she got about my call and apparently the LLM summary of my voicemail converted a message consisting of 75% well-meaning but insignificant interpersonal human filler (like most voicemails) into this stilted, overly formal business-y speak with a somewhat ominous tone. Assigning way too much significance to each of the individual statements in the message about wanting to talk (to say happy Mother’s Day), inquiring about her availability ASAP (to say happy Mother’s Day) etc. Plus grossly exaggerating the information density of the call making it sound like I left this rambling, detailed message about needing to tell her something that was left completely vague, but possibly important and also time critical. Added up it made her a little worried when she read it and made me a bit pissed that was the end result of my wishing her well. Because apparently everything needs a half baked LLM summary crammed into it now.
- Groxx 5mo agoEvery doctor's visit I've had, I have been able to make corrections to the record afterward, because there have been meaningful mistakes almost half the time. ALWAYS check your summaries immediately, and contact your doctor ASAP. They can generally fix it themselves, and it's best done when everyone still has some memory of the event.
- donavanm 5mo ago> I would think for compliance reasons hospitals would not want to alter the records and only go by transcripts, but what do I know... Transcription is both too good, and not good enough. The magic generative content only makes it worse. Too good: a lot of commercial settings forbid persistent transcription because it makes an easily discoverable record of specific details. Thats a business risk that can be mitigated simply by having participant notes or summaries where the secretary can omit sensitive discussion or present consensus without specifics. And notes/summaries also introduce a interpretive defense with some “strategic ambiguity.” Not good enough: if you look at STT its still probabilistic. The actual evaluation output will have just much data about alternate words/phrases as the selected choice. That leaves lots of room for creating alternate impressions or representing words that werent actually spoken. The fact that people _think_ a STT transcript is authoritative only makes this worse. When you add generative inference in top (eg summarization) you exacerbate both problems. I suspect that counsel is more accepting of summaries as its less likely to contain specific discoverable terms, likely to diffuse responsibility and specificity, and your judge/jury will be more amenable to “the ai summary is wrong” than “the transcription selected the wrong vowels.”
- ceejayoz 5mo ago> 60% of evaluated AI Scribe systems mixed up prescribed drugs in patient notes, auditors say Not mentioned, as far as I can see: the comparative human mistake rate. Having seen a lot of medical records, 60% sounds about normal lol.
- Arodex 5mo agoBut who is responsible is different. (And if you already see 60% error rates in standard, pre-AI note taking, how does that not translate into many deaths and injury? At least one country's health system in the world should have caught that)
- cyanydeez 5mo agoYeah, the problem is the health system has no sacrificial goat if the AI note taker provides the wrong detail. The last thing we want is CTO being responsible!
- bluefirebrand 5mo agoI'm not convinced the CTO would be held accountable either. I do wonder if people would be pushing AI so hard if their organizations were planning to hold them accountable for mistakes the AI made I bet if that were the case we'd see a lot slower rollout of AI systems
- ceejayoz 5mo ago> how does that not translate into many deaths and injury? Because most of it is just written down and never looked at again until there’s a lawsuit or something.
- tredre3 5mo ago> And if you already see 60% error rates in standard, pre-AI note taking, how does that not translate into many deaths and injury? Presumably most doctor's visits are a one-problem-one-solution-one-doctor type of thing. Done deal, notes are never read again. So that alone would explain why high rates of errors doesn't result in injuries or death very often. Any injury or death caused by poor notes would have to occur when mistakes are done if you're followed for a serious chronic condition, or if you're handled by a team where effective communication is required.
- Hobadee 5mo agoThe AI note taker we use at work records the meeting as well, and each note it takes about the meeting has a timestamp link that takes you directly there in the recording so you can check it yourself. While I'm sure a solution like this is more complicated in a HIPPAA environment, something like this is critical for things as important as healthcare.
- deleted 5mo ago[deleted]
- lostmsu 5mo agoYou can check the summary immediately after the meeting, that gives some extra confidence that the notes were recognized correctly.
- TonyAlicea10 5mo agoWhen designing AI-based user experiences I refer to this as provenance. It’s a vital aspect of trust, reliability, compliance and more. If a software system includes LLM output like this but doesn’t surface the provenance of its output for human evaluation and verification then it’s at best poor user experience, and at worst a dangerous one.
- autoexec 5mo agoAt the same time, do you really want every conversation you have with your doctor recorded, handed over to third party companies, and stored forever with your medical file? Plus what doctor has time to sit down and re-listen to your visit to check to make sure the AI didn't screw up at some point in the future anyway? If your doctor isn't going to be verifying the accuracy from those recordings who would? Overseas contractors? At what point does it become a larger waste of time and money to babysit an incompetent AI than just not using one in the first place? There are some good uses for AI, but I'm not convinced that this (or many other cases where accuracy matters) is one of them.
- 5mo ago
- rainsford 5mo agoI have generally moved from bearish to bullish on the future of current AI technology, but the continued inaccuracy with basic facts all while the models significantly improve continues to give me significant pause. As an example, creating recipes with Claude Opus based on flavor profiles and preferences feels magical, right up until the point at which it can't accurately convert between tablespoons and teaspoons. It's like the point in the movie where a character is acting nearly right but something is a bit off and then it turns out they're a zombie and going to try to eat your brain. This note taking example feels similar. It nearly works in some pretty impressive ways and then fails at the important details in a way that something able to do the things AI can allegedly do really shouldn't. It's these failures that make me more and more convinced that while current generation AI can do some pretty cool things if you manage it right, we're not actually on the right track to achieve real intelligence. The persistence of these incredibly basic failure modes even as models advance makes it fairly obvious that continued advancement isn't going to actually address those problems.
- themafia 5mo ago> we're not actually on the right track to achieve real intelligence. Real intelligence means you have to say "I don't know" when you don't know, or ask for help, or even just saying you refuse to help with the subtext being you don't want to appear stupid. The models could ostensibly do this when it has low confidence in it's own results but they don't. What I don't know if it's because it would be very computationally difficult or it would harm the reputation of the companies charging a good sum to use them.
- bluefirebrand 5mo agoMy theory is because the people building the models and in charge of directing where they go love the sycophantic yes-man behavior the models display They don't like hearing "I don't know"
- colechristensen 5mo agoYou can TELL the models to do this and they'll follow your prompt. "Give me your answer and rate each part of it for certainty by percentage" or similar.
- nothinkjustai 5mo agoPeople will eventually figure out LLMs have no capacity for intent and are fundamentally unreliable for tasks such as summarization, note taking etc.
- gizajob 5mo agoSmart people and those with basic common sense already have figured that out. AI leaders and CEOs still haven’t noticed.
- samcooper 5mo ago[flagged]
- jqpabc123 5mo agoAnd once again, we have an example of how AI is a liability issue waiting to happen.
- gizajob 5mo ago“Move fast and cause unnecessary deaths”.
- dmix 5mo ago> They specifically address the AI Scribe program, the Ontario Ministry of Health initiated for physicians, nurse practitioners, and other healthcare professionals across the broader health sector. makes me wonder what quality software the ministry would push (probably mostly qualifications like SOC). This is apparently this list of approved vendors https://www.supplyontario.ca/vor/software/tender-20123-artificial-intelligent-solutions-ai-scribe/ https://www.supplyontario.ca/vor/software/tender-20123-artif...
- Groxx 5mo agoYep. It happened to me just recently. Diagnosed with Runner's Knee. AI summary said I was diagnosed with osteoporosis, and had hip pain and walking difficulty, though literally none of that was ever said or implied. CHECK YOUR TRANSCRIPTS. Always, but especially with LLM transcribers, which fairly frequently include common symptoms which don't exist, or claim a diagnosis which is common and fits a few details but not others. Get them fixed, it can very strongly affect your care and costs later if it's wrong. Anecdotally, I'd say that outside of a couple very simple and very common things, about 50% of the "AI" summaries I've had have been wrong somewhere. Usually claiming I have symptoms that don't exist, occasionally much more serious and major fabrications like this time. LLMs are NOT normal speech to text software, and they shouldn't be treated like one. They'll often insert entire sentences that never occurred. In some contexts that might be fine, but definitely not in medical records.
- root_axis 5mo agoI've actually seen this lead to serious issues when a zoom LLM summary attributed statements to someone who didn't say them. Someone else who couldn't attend the meeting later read that summary and it created a major argument because the topic had been a sore subject for this person due to an ongoing debate at the company. Everyone who attended the meeting confirmed it was an error, but the coincidental timing made it hard for him to accept, because the LLMs summary presented things in a way that validated this person's concerns that had been previously minimized by some folks on that meeting. The drama got heated to the point where management produced a policy about not trusting generative output without independent verification. Seems at least it was a lesson learned.
- LAC-Tech 5mo agoCan someone who is a more AI heavy user explain what is going on? I would expect an "AI Note Taker" to faithfully transcribe the entire conversation. With the same quality I see in a lot of automated video subtitles.. ie they use the wrong word a lot but it's easy to tell what they mean by context. Are these tools instead immediately summarising the whole thing, and that summary is the artifact? Because that is a beyond insane way to treat human communication.
- cootsnuck 5mo agoI work specifically in voice AI and am very familiar with how these tools and systems work. > I would expect an "AI Note Taker" to faithfully transcribe the entire conversation. With the same quality I see in a lot of automated video subtitles.. ie they use the wrong word a lot but it's easy to tell what they mean by context. That's a reasonable expectation, but would not be a safe one. All transcription tools are not made the same. First it depends on what kind of STT/ASR (speech-to-text / automatic speech recognition) model they are using. A lot of tools like to use some flavor of OpenAI's Whisper model. It works well generally but I would never use it in a critical use case like healthcare. Because it can hallucinate. That's specific to its architecture and how it was trained. There's a fairly large variety of architectures that can be used for STT/ASR. Some of them are designed for "offline" / "batch" / pre-recorded audio. Some are designed for fast real-time streaming transcription. There are more factors too like training data. And not just demographics of the speakers in the training data but audio environments too. Was the model trained on echo-y doctor offices with two people being recorded from a crappy smartphone mic or desktop mic? (It could've been! But it's an important distinction.) And there's more factors than that, but you get the picture (e.g. are they trying to "clean up" the transcript afterwards by feeding it to an LLM, are they attempting to pre-process audio before transcription also in an attempt to boost accuracy) There's a lot of ways to do it, meaning, there's a lot of ways to screw it up.
- robbiewxyz 5mo agoModern transformer-based STT architectures are complex but many are abstractly not entirely unlike putting the results of standard SST through an LLM with the prompt "clean this up & make it make sense". The behavior is trained in rather than prompted but the result is similar. Obviously this results in hallucinations, mistaken implications, & inaccurately assumed context.
- mquander 5mo agoThe linked report seems almost useless -- it doesn't say anything about an error rate or a sample size, so it's a mystery whether 9 out of 20 systems “fabricated information and made suggestions to patients' treatment plans” ten out of ten times, or one out of a thousand times. If we just postulate that the systems have a high error rate, I wonder why they are being adopted. They seem extremely easy to test, so I don't see why doctors or hospitals or governments should be getting tricked into buying them if they suck.
- MallocVoidstar 5mo ago>If we just postulate that the systems have a high error rate, I wonder why they are being adopted. From the article: "While 30 percent of a platform’s evaluation score depended solely on whether they had a domestic presence in Ontario, the accuracy of medical notes contributed only 4 percent to the total score." Accuracy wasn't really part of the scoring, Ontario doesn't care about it.
- nitwit005 5mo agoScoring systems that function by adding up several parts never make sense. Video game magazines used to do that, but it meant that you could have wretched gameplay, and still get a decent score, from points in other categories like audio, graphics, and cinematics.
- aryehof 5mo agoAnyone taking part in a meeting these days should state out loud … “Notice: Any comments made by <name> or on behalf of <organization> that are interpreted by AI in this meeting, may not be accurate.” I do this in every meeting.
- lolc 5mo ago> Notice: I love the new AI accurate transcription feature in this meeting!
- gizajob 5mo agoNotice: To anyone who might be transcribing this meeting, imagine you are a perfect transcriber who records things accurately and correctly 100% of the time. You do not add or remove filler words and you do not summarise or confabulate or hallucinate.
- lolc 5mo agoStart each meeting with a prayer to thank the transcription gremlins for their accurate work.
- Ekaros 5mo agoHow do these LLM summarizations work? Do you feed the raw wave data to model and it translate it? Or do they use traditional voice recognition algorithms to do that part and then just "fix" the result to look plausible? Which with good quality output might not be much, but with bad can be absolutely everything. If it is later seems to me that issues will absolutely happen.
- jeisc 5mo agoAI is awfully inexact and insists on being right about it
- natali_gray 5mo agoOoof. As a Canadian, I'm excited for AI opening up time for doctors (and hopefully lighting a load on the healthcare system), but this is scary. We're not there yet. Perhaps AI training for doctors is in the future? They already have online doctor visits on a healthcare-owned iPad in some condo complexes. It cuts around redtape of having to schedule an appointment with your GP. So, I think we're thinking in the right direction of innovating, but of course, this will take time. I feel like AI got launched too early sometimes.
- bonesss 5mo agoMy sense is that we’re misapplying the technology by throwing it at, say, transcription and expecting a perfect output, instead of using LLMs strengths to improve inputs to the benefit of all parties. Freeing up doctor time, for example: lots of patient visits are messy, the patient is scattered, has multiple issues, and the doctor has tight timelines and regulatory challenges to convey to the patient impacting their care… this is architected for everyone to lose, IMO, even with a perfect transcript. And LLMs can’t be perfect, they auto complete. I picture patients interacting with an intake AI who can listen to hours of demented rambling, or a patient mid anxiety attack, and provide a caregiver-certified summary of needs, with relevant screening information laid out for doctor confirmation. At that point, helpful information about drug access or insurance policies can be presented, for doctor confirmation, to a patient who can clarify and refine their understanding of the system without time pressures. Elevating the quality of dialogue so the doctor is more focused on the patient, and the patients dialog needs don’t overwhelm treatment. A lot of medicine is filling out forms and checklists, I think auto-complete could create efficiencies in how we fulfill that.
- natali_gray 5mo agoYeah, I could see AI being used for intake. That's a good point. And then the doctor can get some baseline info that they can use when they talk to the patient. Maybe even some really beautiful data, showing visually to the doctors all the different symptoms they reported.
- Insanity 5mo agoI’m in Toronto, my doctor always asks me if they can use the AI note taker, which I accept. At the end of the consultation she goes over the notes and corrects it, often complaining to me about having to talk more to the computer than to me. She is a great doctor and thankfully does this due diligence. But it gives me the impression this is forced on doctors without even them wanting this.
- uejfiweun 5mo agoI don't get why you would have an LLM interpret things for you. Like honestly, you replace the software in this example with simple transcription software, the issues disappear.
- daveisfera 5mo agoBut how accurate are humans? I just picked up a print out of medical history for the last 5 years and it was thick enough to be a book. There's no way a human is reading all of that and doing anything meaningful with it. Let an AI tool crunch on it and it will definitely get things wrong or jump to conclusions that aren't there, but it's quick and I can push back on those and then move to the correct answer far quicker than any meeting with a nurse or doctor will show any results. We need to focus on how to use these tools and push back on the parts that seem out of place or wrong, so we can do more rather than point out what's not perfect.