22 ms·

Show HN: Using GPT-3 and Whisper to save doctors’ time

117 points by ar7hur 4y ago

Hey HN, We're Alex, Martin and Laurent. We previously founded Wit.ai (W14), which we sold to Facebook in 2015. Since 2019, we've been working on Nabla (https://nabla.com https://nabla.com), an intelligent assistant for health practitioners. When GPT-3 was released in 2020, we investigated it's usage in a medical context[0], to mixed results. Since then we’ve kept exploring opportunities at the intersection of healthcare and AI, and noticed that doctors spend am awful lot of time on medical documentation (writing clinical notes, updating their EHR, etc.). Today, we're releasing Nabla Copilot, a Chrome extension generating clinical notes from video consultations, to address this problem. You can try it out, without installation nor sign up, on our demo page: https://nabla.com/copilot-demo/ https://nabla.com/copilot-demo/ Here’s how it works under the hood: - When a doctor starts a video consultation, our Chrome extension auto-starts itself and listens to the active tab as well as the doctor’s microphone. - We then transcribe the consultation using a fine-tuned version of Whisper. We've trained Whisper with tens of thousands of hours of medical consultation and medical terms recordings, and we have now reached an error rate which is 3× lower than Google's Speech-To-Text. - Once we have the transcript, we feed it to a heavily trained GPT-3, which generates a clinical note. - We finally return the clinical note to the doctor through our Chrome extension, the doctor can copy it to their EHR, and send a version to the patient. This allows doctors to be fully focused on their consultation, and saves them a lot time. Next, we want to make this work for in-person consultation. We also want to extract structured data (in the FHIR standard) from the clinical note, and feed it to the doctor’s EHR so that it is automatically added to the patient's record. Happy to further discuss technical details in comments! --- [0]: https://nabla.com/blog/gpt-3/ https://nabla.com/blog/gpt-3/

tomasyany 4y ago
It looks pretty neat! Just wondering, what other languages are supported? English seems pretty natural as it is where GPT-3 thrives, but have you been able to support other languages as well?
- l5t 4y ago
  We currently support English and French and we will be supporting more soon
stopachka 4y ago
This is really cool! What surprised you guys about fine-tuning whisper + gpt-3?
- mraison 4y ago
  Domain specificity is a killer advantage when fine-tuning! For Whisper in particular, fine-tuning with actual medical speech gave us an immediate strong advantage over competing APIs.
v4dok 4y ago
I like the idea, but I would like it even more if it was on-prem. The doctors (at least in EU) will be very wary of having their client meeting essentially recorded by a third party. With this as a cloud SaaS, patient confidentiality is essentially broken since the raw data is available to you while you transcribe it. I understand that you compete with "google speech-to-text" but this is not a feature meant to be used by doctors (even if they "illegally" do). Obviously the business model is harder with on-prem, but cloud-first for doctor notes is in the long run much harder.
- agilob 4y ago
  There are so many regulations and certifications to do that there are no chances anyone will have money and time to do all paperwork to host it on-perm. Because of these regulatory reasons most hospitals use so old systems that it surprises no one to see Windows XP with IE6 there. I had the pleasure myself of fixing a Material Angular bug that wasn't displaying form fields validation correctly in IE6 only 2 years ago, and that was for network of hospitals in Canada, US and Germany. Effort to allow them to use anything newer is too big and there are too many documents to review. It's simply cheaper to keep WinXP and IE6 running for as long as possible.
  - v4dok 4y ago
    I've read about yesterday someone running LLAMA in a single GPU. Maybe if you optimise the model enough, you can give it to them as a box.
  - actionfromafar 4y ago
    Off prem is also a rats nest of regulations.
    - agilob 4y ago
      Yes, but it's done once, on demand for each customer and service-wide certificates don't expire for a few years or releases, so the cost is lower.
908B64B197 4y ago
> This allows doctors to be fully focused on their consultation Here's the thing: rigorous medical documentation is not a burden on the practitioner, but rather an integral part of treating someone. You need detailed historical data to diagnose some tricky conditions and treat them correctly. What's the tool doing to prevent a practitioner from simply blindly copy-pasting the output without verifying it? > and saves them a lot time. Which means more billable. Have you considered charging per medical act performed?
- khyryk 4y ago
  Unfortunately I've had interactions with the medical system where the doctors wrote their after meeting notes with severe errors and pieces of fiction, as well as specialists who almost ran out of the room after 5 minutes of an appointment they billed 45 minutes for to the insurance. I doubt such cases are rare and I suspect this sort of product, which I'm a priori neutral about, would change the situation much for the worse.
orcajerk 4y ago
Last time I went to the doctor, he charged me $300 to google symptoms on a computer. Now you're saying I get to pay $300 for him to ask ChatGPT? No thanks. We all know that is where this is going - doctor decision support system, i.e. the AI has access to information locked behind expensive journals that the patient does not.
- Shared404 4y ago
  > Last time I went to the doctor, he charged me $300 to google symptoms on a computer. Would you be upset if someone were to ask you to help with their code and you needed to Google for documentation? It's not exactly the same thing, but it's a similar situation. I do tend to agree with you on the rest of your concerns, especially info being used that the patient has no way to verify - but that seems tangential at worst to the product in it's current state.
  - JohnFen 4y ago
    > It's not exactly the same thing, but it's a similar situation. It's pretty close to the same thing. Programmers are paid for knowing how to program, not for memorizing APIs, system calls, etc. Being able to look up details is essential because it lightens the cognitive load, allowing deeper thought to be put into the real meat of the problem. No programmer can realistically memorize every technical detail they'll need to know. It's the same with doctors. They're paid because they know the process of diagnosing illnesses, not for memorizing Gray's Anatomy and every pharmacopoeia. Being able to look stuff up is essential for the same reason it is for programmers. Not to mention that medicine changes, and you want your doctor to have the latest information about whatever it is that ails you.
- clemailacct1 4y ago
  The tone of this reply is excessively snarky. You clearly have a personal bad experience with your doctor - but that doesn’t mean that this potential product is a terrible idea.
- matsemann 4y ago
  Last time I asked a developer to fix a bug, he charged me $300 to google the api docs..! But this isn't even about the doctor asking GPT, this is transcribing the video call so the journaling becomes easier and the doctor can spend more time with their patients. A good thing, no?
ouid 4y ago
it sounds like you're removing a checklist from doctors.
avgDev 4y ago
I feel uncomfortable knowing doctor would use anything like this. How are 2 party consent states handled? Is this HIPPA compliant?
- jimnotgym 4y ago
  Imagine a doctor instead mumbled his notes in a non native accent into a dictation machine, and had someone on minimum wage type them up?
  - avgDev 4y ago
    Um there is dictation tools which are HIPPA compliant?
    - sp332 4y ago
      Of course, medical dictation is a whole industry. https://www.bls.gov/ooh/Healthcare/Medical-transcriptionists.htm https://www.bls.gov/ooh/Healthcare/Medical-transcriptionists...
  - shagie 4y ago
    That job is known as a medical scribe. https://www.scribeamerica.com/what-is-a-medical-scribe/ https://www.scribeamerica.com/what-is-a-medical-scribe/ > A Medical Scribe is a revolutionary concept in modern medicine. Traditionally, a physician's job has been focusing solely on direct patient contact and care. However, the advent of the Electronic Health Record (EHR) created an overload of documentation and clerical responsibilities that slows physicians down and pulls them away from actual patient care. To relieve the documentation overload, physicians across the country are turning to Medical Scribe services. > A Medical Scribe is essentially a personal assistant to the physician; performing documentation in the EHR, gathering information for the patient's visit, and partnering with the physician to deliver the pinnacle of efficient patient care.
- moomoo11 4y ago
  HIPAA only applies to healthcare providers I think. Private companies like the extension maker can do whatever they want. At least that’s what someone on the internet told me and maybe it’s wrong.
BCM43 4y ago
Will you sign a BAA for HIPAA covered entities?
- doctoring 4y ago
  They list it under their "legal documents" page: https://www.nabla.com/docs/business-associate-agreement https://www.nabla.com/docs/business-associate-agreement
  - bitshiftfaced 4y ago
    They're saying that they will keep the information private, but I'm wondering about whether GPT will uphold this standard as well.
  - ceejayoz 4y ago
    If I were a provider, I'd probably want to see their BSA with OpenAI.
- ar7hur 4y ago
  Yes absolutely.
phren0logy 4y ago
Side-note: your email field to sign up for your newsletter is very wonky with both Safari and Firefox (at least for me). It only lets me type one letter at a time, for some reason.
- liautaud 4y ago
  Thanks for the report, it should be fixed now.
priyanmuthu 4y ago
My worry is verification of facts. What if the model summarized incorrect facts, and it gets added to the medical history? How can doctors easily verify and fix things?
- jjoonathan 4y ago
  Have you had a medical billing experience recently? They don't care about facts, they care about maximizing the bill subject to plausibility constraints. I fear that LLMs will kick this to a whole new level :(
- ar7hur 4y ago
  Thanks for your comment. We clearly show a diff to the doctor before updating patient records. We are very explicit that they should check it (and edit if necessary).
- sp332 4y ago
  It’s easier to revise than to write a first draft. It’s possible to miss mistakes on review, but relatively, I think it’s more likely that a distracted human will mess something up on the first draft.
  - D13Fd 4y ago
    That's exactly the thought process that leads to us trusting LLMs in places where they shouldn't be trusted.
- qgin 4y ago
  A good benchmark could be to compare the accuracy of generated notes to the accuracy of other currently accepted practices like human medical scribes and assistants.
urduntupu 4y ago
Technically interesting but solving the wrong problem. Doctor's already today spent too little time with their patients to understand diseases holistically enough. Adding technically between these 2 will make treatments in most of the cases worse, not better.
- hartleybrody 4y ago
  Wow, lots of haters in this thread. Having doctors spend less time with clerical/administrative work needed to record conversations will free up more time to focus on patients. I assume the generated note text can be easily modified by the doc if necessary, but saves them the bulk of having to type things up and remember everything that was discussed.
- vsareto 4y ago
  Nothing's going to fix that but having more doctors around. Going from a recorded transcript to a summary note, extracted structured data, and diagnostic codes is a big time saver.
- stopachka 4y ago
  Right now, drs type notes as you speak to them. If this works, they'd have more time to spend actually listening and talking to you.
- nharada 4y ago
  > solving the wrong problem You seem quite confident about this, but based on the doctors I know writing notes is a real pain and mostly seen as (important) scutwork. They're only given a set amount of time to see a patient (including notes), and if you can reduce notes they'd actually get more time talking and seeing the patient.
  - thfuran 4y ago
    What percent of that scutwork is actually note taking rather than clicking a bunch of checkboxes or combo boxes or otherwise navigating through EHR UI?
  - 4y ago
ysavir 4y ago
I'd prefer to have 0 AI interpreting my session with a doctor. Will there be any way for me to identify which physicians use this service so that I can avoid using them?
- narwally 4y ago
  Hell, it's hard enough these days to identify which physicians are actually physicians. I recently had an NP introduce herself to me as "Dr. So-and-so."
lukah 4y ago
This is really interesting work, congratulations! A lot of negativity in the comments seems to stem from cynicism around doctors and their billing, but in a country like the UK I know removing essentially "admin tasks" from the GP would be very welcome and give time back to the clinician, allowing them to spend more time focussing on the patient and outcomes.
NorthOf33rd 4y ago
This is the scariest real life attempt to use the bullshit machine I’ve seen so far. Nothing like “subtly wrong” clinical notes to affect positive medical outcomes. And time pressed medical professionals are certainly well suited to vigilance tasks like correcting machine generated errors. /s. I sincerely hope this never sees the light of day.
- manv1 4y ago
  If you think this is scary you should see how it's done today without AI. Doctor's handwriting is notoriously bad. Now try putting that into an EMR. Good luck with that! You think you're going to get someone who gets paid $2000/hr to type shit up? Yeah, guess again.
  - Pigalowda 4y ago
    We don’t get paid even close to $2k/hr. Where is that number from? Many physicians type notes, others use dragon dictation, or medical assistants/scribes. I used to type, now I dictate. Quality of note is provider dependent though. If an LLM can improve quality of notes across the board through simple summary that’s interesting. But the input matters - is the provider actually asking the right questions? Did they look at past relevant history and other notes and put it in their current note? The LLM should be in a HIPAA compliant environment and have access to the EMR. Then it provides a tidy summary for that. Second it should have relevant questions for the provider to ask during the interview but also a real time/dynamic component which generates relevant follow up questions dependent on patient responses. Lastly it should put together the old and the new and generate a new note which can then be edited by the provider. Editing notes is much easier than generation of a new high quality note.
ar9av 4y ago
Maybe the future of the first line of healthcare will start to use AI with the assistance of a specialized nurse to follow your health overtime and only fall back to a real doctor when a real problem occurs. This could elevate a lot of problems we have with our public healthcare(Canada).
mjdowney 4y ago
That's super cool!
swatcoder 4y ago
I don't spend much time worrying about AI ethics, but bringing AI close to patient interactions and record-keeping in healthcare seems grossly premature and irresponsible. There are countless brilliant applications of AI and the last one we need is a mediocre automated transcriptionist that distances a doctor from their responsibility to engage with the patient and can bear no accountability for error. This is a task that perhaps can be supported with AI some day, but there are fields that deserve the application of a mature technology, not the gold rush rush game of integrating today's hottest thing.
- D13Fd 4y ago
  The transcription part doesn't bother me. That's just advanced speech-to-text. The summarization part, though, is dangerous. It's a very quick path to us losing all faith in our own medical records. Any way you slice it, no matter how much you train it, it's still going to be vulnerable to hallucination errors that slip by the reviewing doctor and become part of the patient's medical history.
- ar7hur 4y ago
  I understand your concerns. Our objectives are, on the contrary, to reduce the doctor/patient distance that was created by EHRs and all the required administrative processes. We already measure that when AI takes care of this stuff, doctors do better engage with patients.
- ttpphd 4y ago
  "I don't spend much time worrying about AI ethics" Maybe you should re-evaluate whether that's the right choice based on your own comment, especially when ethics and safety experts are being fired by big tech co.
hartleybrody 4y ago
I used to work in a hospital and one thing that was already tough was when doctors would copy massive (multiple page) notes that were mostly boilerplate, probably stored as some template in the EHR. After seeing the same note a few times, you started to learn which page to scroll to in order to view the relevant section for the patient. I would hope this system only generates clinically-relevant text for that patient, usually only 2-3 paragraphs for a standard consultation.
mitchellpkt 4y ago
Just read the blog post, very interesting. I thought sections about changing the temperature and repetition penalty were particularly neat, thanks for including examples.
cinntaile 4y ago
> We then transcribe the consultation using a fine-tuned version of Whisper. We've trained Whisper with tens of thousands of hours of medical consultation and medical terms recordings, and we have now reached an error rate which is 3× lower than Google's Speech-To-Text. I think the general idea is interesting but this is the wrong benchmark to convince people imo. What is the actual error rate of Google's Speech-to-Text on this your data? How convinced are you that it's properly labeled? That it's 3x lower isn't necessarily impressive. Then a follow up question... What I as a patient would want to know is something else, how does it compare against medical secretaries that usually transcribe documents like these?
tzm 4y ago
Love this and have had discussions w/ doctors and practitioners about this exact problem. I think extracting to FHIR is key, along with making it searchable as a longitudinal data store for a more robust personal record. Clear need. Would love to help.
D13Fd 4y ago
The "feed it to a heavily trained GPT-3" part is backwards. The transcribed notes should be stored and retained. The LLM should be applied later, in real time, when someone is referring back to the notes. That way you still get the benefit of summaries, without polluting the medical records with hallucinated AI nonsense.
rubenstern 4y ago
Looks very interesting! Are you recruiting ?
- ar7hur 4y ago
  Yes we are! https://www.nabla.com/careers/ https://www.nabla.com/careers/
norgie 4y ago
Is Whisper really up to this? When I've used it, it's...fine. But it also randomly dreams up weird transcriptions that have nothing to do with what's actually being said, and with perfect audio too. Like the audio will be a presentation at a tech conference, and Whisper will predict "check out our site at randomlink.com", when no one said anything close to that. I'd be pretty concerned if my doctor relied on this to be honest.
- l5t 4y ago
  Good point, this is why we finetune Whisper with our own medical dataset to improve the performance
JohnFen 4y ago
As a patient, this is an application that I would be extremely wary of, personally. I don't want the details of my conversations with my doctor to be sent to a third party in the first place, and I wouldn't trust the results of the transcription to be correct. My doctor could vet it for accuracy, I suppose, but why? He's already putting his notes in my records anyway.
- l5t 4y ago
  It takes 40% of your doctor time to put his notes in your medical records. We have been testing with doctors to reduce this time drastically so that they can spend more quality time with you. We also interviewed a lot of patients who take notes to be able to remember what’s been discussed during a consultation. So it could also benefit the patient
  - Veen 4y ago
    Your response completely ignores the concerns raised in the comment you are replying to. "It saves your doctor 40% of their time" is not a reasonable response to "I don't want my private medical details divulged to a third party".
    - jeroenhd 4y ago
      Perhaps avoiding commenting on patient privacy concerns may be a better answer than any evasive answer may ever be. Their website's privacy policy doesn't say anything about their product. It's also incorrect (they seem to have switched tracking providers without updating their privacy policy). I would quote the sections that are incorrect, but their terms and conditions forbid copying any content from their site. Their T&C also forbids me from linking to their page or terms. I very much doubt that using this product as advertised is even legal under the GDPR. The company is French and health data, even pseudonymized, is strictly regulated. The "demo" doesn't feature any explicit consent at the very least. It looks like they're making all privacy risks and concerns the doctors' problems. After all, they're the ones violating the law when they use this product.
kgiddens1 4y ago
Congrats on the launch Nabla!
JUNGLEISMASSIVE 4y ago
[dead]
gigel82 4y ago
What is OpenAI's privacy policy? How do you ensure the transcription doesn't get stored by OpenAI or used in training their future LLM versions? I'd recommend looking at a fully self-hosted LLM alternative or ensuring the GPT-3 endpoint you're using is compliant (I've heard Azure has a GPT-3 API that is supposedly more "compliant").
avipars 4y ago
how can this be compliant with HIPAA?
- dragonwriter 4y ago
  It can be HIPAA compliant by every company in the chain of use of the data signing a BAA with every one of the companies they deal directly with that handles PHI, and then following the terms of HIPAA in dealing with the data.
- deleted 4y ago
  [deleted]
ismosoft 4y ago
amazing. you're ROCKKKKKKKKKK
ismosoft 4y ago
this is amazing. great great
rubatuga 4y ago
I did some work at a startup that tried to solve a similar problem and currently finishing medical school. Nobody here is discussing the paperwork and burnout problem that exists in medicine that will make these tools valuable. Fortunately, language models used for summarization based on a transcript are much less likely to hallucinate compared to upscaling/denoising image algorithms. Recent LMs are scoring top marks on LSAT, so they are maintaining accuracy as they get larger. If doctors can save time, this will be useful for the patient in the long-run. On-prem stuff is more tricky, but useful for privacy. Nvidia is supporting the space with their health AGX series.
- JohnFen 4y ago
  > On-prem stuff is more tricky, but useful for privacy. An on-prem installation, assuming that it doesn't phone home to the mothership, is something I would entertain. But one that requires disclosure of medical information to a service provider would be a lot harder for me to accept.
futureisvintage 4y ago
Amazing idea!!
retox 4y ago
How tied are you to the name, it's almost Nambla.
- whitethunder 4y ago
  That's all I see when I see this company's name. Unfortunate choice.
SkyPuncher 4y ago
Pretext: My wife is a psychiatrist. Via her, I'm am close friends with many doctors. Also, I know it's super easy to be critical, but I'm going to save you an insane amount of effort and heartache. This is actually a solved problem in medicine, already. Maybe not solved, but absolutely "good enough". M-Modal, Dragon, and Telephone services (yes, call someone and have them translate) all exist. They all translate notes well enough. For most doctors, it's a huge improvement over typing. The problem is primarily access. Most institutions simply do not want to pay for this. They don't view it as necessary. Further complicating the issue. Unless a physician is in private practice, they have little to no control over their ability to use a tool like this. HIPAA is far too high of a barrier. You're trying to implement a technical solution to what's ultimately a politics/human problem. ---- I've spent a bunch of time in the space, both as a MedSpouse and the CTO of a HealthTech company. I wish you the best of luck as I'd love to see life be easier for my wife, but this is _extremely_ far from being MVP viable. * HIPAA - You absolutely, 100%, without a doubt need to be HIPAA compliant. You likely want to be HITRUST compliant as well. You're very, very clearly a Business Associate (HIPAA term) and processing PHI. Without that, you're dead in the water. * Chrome extension - This is not going to cut it. At baseline, it requires copy and pasting between Chrome and the EHR system. Bad UX. * More tactically, most EHR access occurs via remote access, like Citrix. Often from a thin-terminal on the hospital floor or at a med station. These are locked down environments that have a high likely hood of not having Chrome installed _and_ not being allow to install _any_ software (including Chrome extensions). * Doctors need real-time feedback and the ability to correct. Real-time feedback is critical because carrying a patient takes mental load. Part of the purpose of notes is to create a written record that "closes" a patient for the time being. An async service isn't really helpful since it generates an interrupt after the physician has already moved on. * Accuracy is important. Accuracy is even more important with _extremely_ similar and _extremely_ uncommon words. For example, can this system differential between prednisone and prednilisone? In fact, can it even identify those words? Can it do it with an accent? * FHIR is only as much of a standard as tables are to relational databases. The hard part is the thousands of custom fields that are different between health systems. * Structured data really isn't used or useful in medicine. You might have a medication list that can be structured as data, but actual notes include a lot of prose. I believe part of this is necessary to capture the nuance/minutiae of some healthcare. Part of it stems from the lack of any standard. * Again, this is all possible already. Tele-notes have been done since the introduction of EHR systems. * Nobody is going to use a Chrome extension.
Terretta 4y ago
Ignore the nay-sayers in this thread. I doubt the ones commenting have spent many days personally observing or personally doing the problem scenario you are solving for. This is spectacular for telemedicine in the post-covid regulation environment. The new model is an onsite medical assistant taking patient vitals and setting up the Zoom, the patient then talks to an APRN (Advanced Practice Registered Nurse) in the patient's field (e.g. subacute care, gerontology, etc.). The APRNs spend 30 minutes reading the file, tele-consult with a patient for a half hour. Medicare pays for the half hour. Then the APRN has to spend ~60 minutes turning that session into the EHR notes. Many just don't, they make an ultra short note, and hope Medicare doesn't challenge it. Patient loses. If too strapped for time, they also may not pre-read, because they only get paid for the video conf time. Patient loses. Downside I see with this is the APRNs doing telemedicine are making about 1/4 of the rate they should be, and can't afford an expensive tool. The telehealth provider is not going to pay for it, they get paid for the 1/2 hour, the wasted time is not out of their pocket, so they won't spend to make it up. This is going to be on the nurse, who likely can't afford it. On the plus side: time saved with this tool would go directly to ability to pre-read, or ability to see a patient an hour instead of one per two or three hours. It would also permit a higher caliber of expertise nurse to do tele-medicine visits in off days, because the time saved could bring comp back more in line with the expertise. Go for it!
Banjor 4y ago
One thing that a lot of folks still don't get is that some problems are bad, while others are not-so bad.
CiceroCiceronis 4y ago
I'm a medical student and one of my main takeaways from spending time in clinical environments is that a lot of doctors and other staff seem to either not realise or not care about how much time is wasted due to slow, inefficient processes and technology. I was wondering to myself about what it might take to train a model to recognise medical jargon to help doctors rapidly transcribe audio recordings of their consults/spoken recounts of their notes. Not surprised that somebody is looking to make a commercial product of it, there's definitely space for it.

117 points by ar7hur 4y ago

tomasyany 4y ago

It looks pretty neat! Just wondering, what other languages are supported? English seems pretty natural as it is where GPT-3 thrives, but have you been able to support other languages as well?

l5t 4y ago
We currently support English and French and we will be supporting more soon

stopachka 4y ago

This is really cool! What surprised you guys about fine-tuning whisper + gpt-3?

mraison 4y ago
Domain specificity is a killer advantage when fine-tuning! For Whisper in particular, fine-tuning with actual medical speech gave us an immediate strong advantage over competing APIs.

v4dok 4y ago

I like the idea, but I would like it even more if it was on-prem. The doctors (at least in EU) will be very wary of having their client meeting essentially recorded by a third party. With this as a cloud SaaS, patient confidentiality is essentially broken since the raw data is available to you while you transcribe it. I understand that you compete with "google speech-to-text" but this is not a feature meant to be used by doctors (even if they "illegally" do). Obviously the business model is harder with on-prem, but cloud-first for doctor notes is in the long run much harder.

agilob 4y ago
There are so many regulations and certifications to do that there are no chances anyone will have money and time to do all paperwork to host it on-perm. Because of these regulatory reasons most hospitals use so old systems that it surprises no one to see Windows XP with IE6 there. I had the pleasure myself of fixing a Material Angular bug that wasn't displaying form fields validation correctly in IE6 only 2 years ago, and that was for network of hospitals in Canada, US and Germany. Effort to allow them to use anything newer is too big and there are too many documents to review. It's simply cheaper to keep WinXP and IE6 running for as long as possible.
- v4dok 4y ago
  I've read about yesterday someone running LLAMA in a single GPU. Maybe if you optimise the model enough, you can give it to them as a box.
- actionfromafar 4y ago
  Off prem is also a rats nest of regulations.
  - agilob 4y ago
    Yes, but it's done once, on demand for each customer and service-wide certificates don't expire for a few years or releases, so the cost is lower.

908B64B197 4y ago

> This allows doctors to be fully focused on their consultation Here's the thing: rigorous medical documentation is not a burden on the practitioner, but rather an integral part of treating someone. You need detailed historical data to diagnose some tricky conditions and treat them correctly. What's the tool doing to prevent a practitioner from simply blindly copy-pasting the output without verifying it? > and saves them a lot time. Which means more billable. Have you considered charging per medical act performed?

khyryk 4y ago
Unfortunately I've had interactions with the medical system where the doctors wrote their after meeting notes with severe errors and pieces of fiction, as well as specialists who almost ran out of the room after 5 minutes of an appointment they billed 45 minutes for to the insurance. I doubt such cases are rare and I suspect this sort of product, which I'm a priori neutral about, would change the situation much for the worse.

orcajerk 4y ago

Last time I went to the doctor, he charged me $300 to google symptoms on a computer. Now you're saying I get to pay $300 for him to ask ChatGPT? No thanks. We all know that is where this is going - doctor decision support system, i.e. the AI has access to information locked behind expensive journals that the patient does not.

Shared404 4y ago
> Last time I went to the doctor, he charged me $300 to google symptoms on a computer. Would you be upset if someone were to ask you to help with their code and you needed to Google for documentation? It's not exactly the same thing, but it's a similar situation. I do tend to agree with you on the rest of your concerns, especially info being used that the patient has no way to verify - but that seems tangential at worst to the product in it's current state.
- JohnFen 4y ago
  > It's not exactly the same thing, but it's a similar situation. It's pretty close to the same thing. Programmers are paid for knowing how to program, not for memorizing APIs, system calls, etc. Being able to look up details is essential because it lightens the cognitive load, allowing deeper thought to be put into the real meat of the problem. No programmer can realistically memorize every technical detail they'll need to know. It's the same with doctors. They're paid because they know the process of diagnosing illnesses, not for memorizing Gray's Anatomy and every pharmacopoeia. Being able to look stuff up is essential for the same reason it is for programmers. Not to mention that medicine changes, and you want your doctor to have the latest information about whatever it is that ails you.
clemailacct1 4y ago
The tone of this reply is excessively snarky. You clearly have a personal bad experience with your doctor - but that doesn’t mean that this potential product is a terrible idea.
matsemann 4y ago
Last time I asked a developer to fix a bug, he charged me $300 to google the api docs..! But this isn't even about the doctor asking GPT, this is transcribing the video call so the journaling becomes easier and the doctor can spend more time with their patients. A good thing, no?

ouid 4y ago

it sounds like you're removing a checklist from doctors.

avgDev 4y ago

I feel uncomfortable knowing doctor would use anything like this. How are 2 party consent states handled? Is this HIPPA compliant?

jimnotgym 4y ago
Imagine a doctor instead mumbled his notes in a non native accent into a dictation machine, and had someone on minimum wage type them up?
- avgDev 4y ago
  Um there is dictation tools which are HIPPA compliant?
  - sp332 4y ago
    Of course, medical dictation is a whole industry. https://www.bls.gov/ooh/Healthcare/Medical-transcriptionists.htm https://www.bls.gov/ooh/Healthcare/Medical-transcriptionists...
- shagie 4y ago
  That job is known as a medical scribe. https://www.scribeamerica.com/what-is-a-medical-scribe/ https://www.scribeamerica.com/what-is-a-medical-scribe/ > A Medical Scribe is a revolutionary concept in modern medicine. Traditionally, a physician's job has been focusing solely on direct patient contact and care. However, the advent of the Electronic Health Record (EHR) created an overload of documentation and clerical responsibilities that slows physicians down and pulls them away from actual patient care. To relieve the documentation overload, physicians across the country are turning to Medical Scribe services. > A Medical Scribe is essentially a personal assistant to the physician; performing documentation in the EHR, gathering information for the patient's visit, and partnering with the physician to deliver the pinnacle of efficient patient care.
moomoo11 4y ago
HIPAA only applies to healthcare providers I think. Private companies like the extension maker can do whatever they want. At least that’s what someone on the internet told me and maybe it’s wrong.

BCM43 4y ago

Will you sign a BAA for HIPAA covered entities?

doctoring 4y ago
They list it under their "legal documents" page: https://www.nabla.com/docs/business-associate-agreement https://www.nabla.com/docs/business-associate-agreement
- bitshiftfaced 4y ago
  They're saying that they will keep the information private, but I'm wondering about whether GPT will uphold this standard as well.
- ceejayoz 4y ago
  If I were a provider, I'd probably want to see their BSA with OpenAI.
ar7hur 4y ago
Yes absolutely.

phren0logy 4y ago

Side-note: your email field to sign up for your newsletter is very wonky with both Safari and Firefox (at least for me). It only lets me type one letter at a time, for some reason.

liautaud 4y ago
Thanks for the report, it should be fixed now.

priyanmuthu 4y ago

My worry is verification of facts. What if the model summarized incorrect facts, and it gets added to the medical history? How can doctors easily verify and fix things?

jjoonathan 4y ago
Have you had a medical billing experience recently? They don't care about facts, they care about maximizing the bill subject to plausibility constraints. I fear that LLMs will kick this to a whole new level :(
ar7hur 4y ago
Thanks for your comment. We clearly show a diff to the doctor before updating patient records. We are very explicit that they should check it (and edit if necessary).
sp332 4y ago
It’s easier to revise than to write a first draft. It’s possible to miss mistakes on review, but relatively, I think it’s more likely that a distracted human will mess something up on the first draft.
- D13Fd 4y ago
  That's exactly the thought process that leads to us trusting LLMs in places where they shouldn't be trusted.
qgin 4y ago
A good benchmark could be to compare the accuracy of generated notes to the accuracy of other currently accepted practices like human medical scribes and assistants.

urduntupu 4y ago

Technically interesting but solving the wrong problem. Doctor's already today spent too little time with their patients to understand diseases holistically enough. Adding technically between these 2 will make treatments in most of the cases worse, not better.

hartleybrody 4y ago
Wow, lots of haters in this thread. Having doctors spend less time with clerical/administrative work needed to record conversations will free up more time to focus on patients. I assume the generated note text can be easily modified by the doc if necessary, but saves them the bulk of having to type things up and remember everything that was discussed.
vsareto 4y ago
Nothing's going to fix that but having more doctors around. Going from a recorded transcript to a summary note, extracted structured data, and diagnostic codes is a big time saver.
stopachka 4y ago
Right now, drs type notes as you speak to them. If this works, they'd have more time to spend actually listening and talking to you.
nharada 4y ago
> solving the wrong problem You seem quite confident about this, but based on the doctors I know writing notes is a real pain and mostly seen as (important) scutwork. They're only given a set amount of time to see a patient (including notes), and if you can reduce notes they'd actually get more time talking and seeing the patient.
- thfuran 4y ago
  What percent of that scutwork is actually note taking rather than clicking a bunch of checkboxes or combo boxes or otherwise navigating through EHR UI?
- 4y ago

ysavir 4y ago

I'd prefer to have 0 AI interpreting my session with a doctor. Will there be any way for me to identify which physicians use this service so that I can avoid using them?

narwally 4y ago
Hell, it's hard enough these days to identify which physicians are actually physicians. I recently had an NP introduce herself to me as "Dr. So-and-so."

lukah 4y ago

This is really interesting work, congratulations! A lot of negativity in the comments seems to stem from cynicism around doctors and their billing, but in a country like the UK I know removing essentially "admin tasks" from the GP would be very welcome and give time back to the clinician, allowing them to spend more time focussing on the patient and outcomes.

NorthOf33rd 4y ago

This is the scariest real life attempt to use the bullshit machine I’ve seen so far. Nothing like “subtly wrong” clinical notes to affect positive medical outcomes. And time pressed medical professionals are certainly well suited to vigilance tasks like correcting machine generated errors. /s. I sincerely hope this never sees the light of day.

manv1 4y ago
If you think this is scary you should see how it's done today without AI. Doctor's handwriting is notoriously bad. Now try putting that into an EMR. Good luck with that! You think you're going to get someone who gets paid $2000/hr to type shit up? Yeah, guess again.
- Pigalowda 4y ago
  We don’t get paid even close to $2k/hr. Where is that number from? Many physicians type notes, others use dragon dictation, or medical assistants/scribes. I used to type, now I dictate. Quality of note is provider dependent though. If an LLM can improve quality of notes across the board through simple summary that’s interesting. But the input matters - is the provider actually asking the right questions? Did they look at past relevant history and other notes and put it in their current note? The LLM should be in a HIPAA compliant environment and have access to the EMR. Then it provides a tidy summary for that. Second it should have relevant questions for the provider to ask during the interview but also a real time/dynamic component which generates relevant follow up questions dependent on patient responses. Lastly it should put together the old and the new and generate a new note which can then be edited by the provider. Editing notes is much easier than generation of a new high quality note.

ar9av 4y ago

Maybe the future of the first line of healthcare will start to use AI with the assistance of a specialized nurse to follow your health overtime and only fall back to a real doctor when a real problem occurs. This could elevate a lot of problems we have with our public healthcare(Canada).

mjdowney 4y ago

That's super cool!

swatcoder 4y ago

I don't spend much time worrying about AI ethics, but bringing AI close to patient interactions and record-keeping in healthcare seems grossly premature and irresponsible. There are countless brilliant applications of AI and the last one we need is a mediocre automated transcriptionist that distances a doctor from their responsibility to engage with the patient and can bear no accountability for error. This is a task that perhaps can be supported with AI some day, but there are fields that deserve the application of a mature technology, not the gold rush rush game of integrating today's hottest thing.

D13Fd 4y ago
The transcription part doesn't bother me. That's just advanced speech-to-text. The summarization part, though, is dangerous. It's a very quick path to us losing all faith in our own medical records. Any way you slice it, no matter how much you train it, it's still going to be vulnerable to hallucination errors that slip by the reviewing doctor and become part of the patient's medical history.
ar7hur 4y ago
I understand your concerns. Our objectives are, on the contrary, to reduce the doctor/patient distance that was created by EHRs and all the required administrative processes. We already measure that when AI takes care of this stuff, doctors do better engage with patients.
ttpphd 4y ago
"I don't spend much time worrying about AI ethics" Maybe you should re-evaluate whether that's the right choice based on your own comment, especially when ethics and safety experts are being fired by big tech co.

hartleybrody 4y ago

I used to work in a hospital and one thing that was already tough was when doctors would copy massive (multiple page) notes that were mostly boilerplate, probably stored as some template in the EHR. After seeing the same note a few times, you started to learn which page to scroll to in order to view the relevant section for the patient. I would hope this system only generates clinically-relevant text for that patient, usually only 2-3 paragraphs for a standard consultation.

mitchellpkt 4y ago

Just read the blog post, very interesting. I thought sections about changing the temperature and repetition penalty were particularly neat, thanks for including examples.

cinntaile 4y ago

> We then transcribe the consultation using a fine-tuned version of Whisper. We've trained Whisper with tens of thousands of hours of medical consultation and medical terms recordings, and we have now reached an error rate which is 3× lower than Google's Speech-To-Text. I think the general idea is interesting but this is the wrong benchmark to convince people imo. What is the actual error rate of Google's Speech-to-Text on this your data? How convinced are you that it's properly labeled? That it's 3x lower isn't necessarily impressive. Then a follow up question... What I as a patient would want to know is something else, how does it compare against medical secretaries that usually transcribe documents like these?

tzm 4y ago

Love this and have had discussions w/ doctors and practitioners about this exact problem. I think extracting to FHIR is key, along with making it searchable as a longitudinal data store for a more robust personal record. Clear need. Would love to help.

D13Fd 4y ago

The "feed it to a heavily trained GPT-3" part is backwards. The transcribed notes should be stored and retained. The LLM should be applied later, in real time, when someone is referring back to the notes. That way you still get the benefit of summaries, without polluting the medical records with hallucinated AI nonsense.

rubenstern 4y ago

Looks very interesting! Are you recruiting ?

ar7hur 4y ago
Yes we are! https://www.nabla.com/careers/ https://www.nabla.com/careers/

norgie 4y ago

Is Whisper really up to this? When I've used it, it's...fine. But it also randomly dreams up weird transcriptions that have nothing to do with what's actually being said, and with perfect audio too. Like the audio will be a presentation at a tech conference, and Whisper will predict "check out our site at randomlink.com", when no one said anything close to that. I'd be pretty concerned if my doctor relied on this to be honest.

l5t 4y ago
Good point, this is why we finetune Whisper with our own medical dataset to improve the performance

JohnFen 4y ago

As a patient, this is an application that I would be extremely wary of, personally. I don't want the details of my conversations with my doctor to be sent to a third party in the first place, and I wouldn't trust the results of the transcription to be correct. My doctor could vet it for accuracy, I suppose, but why? He's already putting his notes in my records anyway.

l5t 4y ago
It takes 40% of your doctor time to put his notes in your medical records. We have been testing with doctors to reduce this time drastically so that they can spend more quality time with you. We also interviewed a lot of patients who take notes to be able to remember what’s been discussed during a consultation. So it could also benefit the patient
- Veen 4y ago
  Your response completely ignores the concerns raised in the comment you are replying to. "It saves your doctor 40% of their time" is not a reasonable response to "I don't want my private medical details divulged to a third party".
  - jeroenhd 4y ago
    Perhaps avoiding commenting on patient privacy concerns may be a better answer than any evasive answer may ever be. Their website's privacy policy doesn't say anything about their product. It's also incorrect (they seem to have switched tracking providers without updating their privacy policy). I would quote the sections that are incorrect, but their terms and conditions forbid copying any content from their site. Their T&C also forbids me from linking to their page or terms. I very much doubt that using this product as advertised is even legal under the GDPR. The company is French and health data, even pseudonymized, is strictly regulated. The "demo" doesn't feature any explicit consent at the very least. It looks like they're making all privacy risks and concerns the doctors' problems. After all, they're the ones violating the law when they use this product.

kgiddens1 4y ago

Congrats on the launch Nabla!

JUNGLEISMASSIVE 4y ago

[dead]

gigel82 4y ago

What is OpenAI's privacy policy? How do you ensure the transcription doesn't get stored by OpenAI or used in training their future LLM versions? I'd recommend looking at a fully self-hosted LLM alternative or ensuring the GPT-3 endpoint you're using is compliant (I've heard Azure has a GPT-3 API that is supposedly more "compliant").

avipars 4y ago

how can this be compliant with HIPAA?

dragonwriter 4y ago
It can be HIPAA compliant by every company in the chain of use of the data signing a BAA with every one of the companies they deal directly with that handles PHI, and then following the terms of HIPAA in dealing with the data.
deleted 4y ago
[deleted]

ismosoft 4y ago

amazing. you're ROCKKKKKKKKKK

ismosoft 4y ago

this is amazing. great great

rubatuga 4y ago

I did some work at a startup that tried to solve a similar problem and currently finishing medical school. Nobody here is discussing the paperwork and burnout problem that exists in medicine that will make these tools valuable. Fortunately, language models used for summarization based on a transcript are much less likely to hallucinate compared to upscaling/denoising image algorithms. Recent LMs are scoring top marks on LSAT, so they are maintaining accuracy as they get larger. If doctors can save time, this will be useful for the patient in the long-run. On-prem stuff is more tricky, but useful for privacy. Nvidia is supporting the space with their health AGX series.

JohnFen 4y ago
> On-prem stuff is more tricky, but useful for privacy. An on-prem installation, assuming that it doesn't phone home to the mothership, is something I would entertain. But one that requires disclosure of medical information to a service provider would be a lot harder for me to accept.

futureisvintage 4y ago

Amazing idea!!

retox 4y ago

How tied are you to the name, it's almost Nambla.

whitethunder 4y ago
That's all I see when I see this company's name. Unfortunate choice.

SkyPuncher 4y ago

Pretext: My wife is a psychiatrist. Via her, I'm am close friends with many doctors. Also, I know it's super easy to be critical, but I'm going to save you an insane amount of effort and heartache. This is actually a solved problem in medicine, already. Maybe not solved, but absolutely "good enough". M-Modal, Dragon, and Telephone services (yes, call someone and have them translate) all exist. They all translate notes well enough. For most doctors, it's a huge improvement over typing. The problem is primarily access. Most institutions simply do not want to pay for this. They don't view it as necessary. Further complicating the issue. Unless a physician is in private practice, they have little to no control over their ability to use a tool like this. HIPAA is far too high of a barrier. You're trying to implement a technical solution to what's ultimately a politics/human problem. ---- I've spent a bunch of time in the space, both as a MedSpouse and the CTO of a HealthTech company. I wish you the best of luck as I'd love to see life be easier for my wife, but this is _extremely_ far from being MVP viable. * HIPAA - You absolutely, 100%, without a doubt need to be HIPAA compliant. You likely want to be HITRUST compliant as well. You're very, very clearly a Business Associate (HIPAA term) and processing PHI. Without that, you're dead in the water. * Chrome extension - This is not going to cut it. At baseline, it requires copy and pasting between Chrome and the EHR system. Bad UX. * More tactically, most EHR access occurs via remote access, like Citrix. Often from a thin-terminal on the hospital floor or at a med station. These are locked down environments that have a high likely hood of not having Chrome installed _and_ not being allow to install _any_ software (including Chrome extensions). * Doctors need real-time feedback and the ability to correct. Real-time feedback is critical because carrying a patient takes mental load. Part of the purpose of notes is to create a written record that "closes" a patient for the time being. An async service isn't really helpful since it generates an interrupt after the physician has already moved on. * Accuracy is important. Accuracy is even more important with _extremely_ similar and _extremely_ uncommon words. For example, can this system differential between prednisone and prednilisone? In fact, can it even identify those words? Can it do it with an accent? * FHIR is only as much of a standard as tables are to relational databases. The hard part is the thousands of custom fields that are different between health systems. * Structured data really isn't used or useful in medicine. You might have a medication list that can be structured as data, but actual notes include a lot of prose. I believe part of this is necessary to capture the nuance/minutiae of some healthcare. Part of it stems from the lack of any standard. * Again, this is all possible already. Tele-notes have been done since the introduction of EHR systems. * Nobody is going to use a Chrome extension.

Terretta 4y ago

Ignore the nay-sayers in this thread. I doubt the ones commenting have spent many days personally observing or personally doing the problem scenario you are solving for. This is spectacular for telemedicine in the post-covid regulation environment. The new model is an onsite medical assistant taking patient vitals and setting up the Zoom, the patient then talks to an APRN (Advanced Practice Registered Nurse) in the patient's field (e.g. subacute care, gerontology, etc.). The APRNs spend 30 minutes reading the file, tele-consult with a patient for a half hour. Medicare pays for the half hour. Then the APRN has to spend ~60 minutes turning that session into the EHR notes. Many just don't, they make an ultra short note, and hope Medicare doesn't challenge it. Patient loses. If too strapped for time, they also may not pre-read, because they only get paid for the video conf time. Patient loses. Downside I see with this is the APRNs doing telemedicine are making about 1/4 of the rate they should be, and can't afford an expensive tool. The telehealth provider is not going to pay for it, they get paid for the 1/2 hour, the wasted time is not out of their pocket, so they won't spend to make it up. This is going to be on the nurse, who likely can't afford it. On the plus side: time saved with this tool would go directly to ability to pre-read, or ability to see a patient an hour instead of one per two or three hours. It would also permit a higher caliber of expertise nurse to do tele-medicine visits in off days, because the time saved could bring comp back more in line with the expertise. Go for it!

Banjor 4y ago

One thing that a lot of folks still don't get is that some problems are bad, while others are not-so bad.

CiceroCiceronis 4y ago

I'm a medical student and one of my main takeaways from spending time in clinical environments is that a lot of doctors and other staff seem to either not realise or not care about how much time is wasted due to slow, inefficient processes and technology. I was wondering to myself about what it might take to train a model to recognise medical jargon to help doctors rapidly transcribe audio recordings of their consults/spoken recounts of their notes. Not surprised that somebody is looking to make a commercial product of it, there's definitely space for it.