4 ms·

Call me when an LLM can ingest a pts medical record from 3 different sources in multiple formats ranging from structured data to free form text stored (I shit y

Call me when an LLM can ingest a pts medical record from 3 different sources in multiple formats ranging from structured data to free form text stored (I shit you not) as TIFF files, extract the actual information, and present it for review prior to a visit and for a follow up visit, read the prior visit's note and orders, search/track down all the test results (may be at more than one institution with multiple EHR systems that don't interchange) and summarize or even just be able to take verbal orders like it used to be: "get a CBC, CMP, TSH, and CXR" The NEJM CPC's are after the hard work has already been done to get the data to present the case

arcticfox 3y ago
> Call me when an LLM can ingest a pts medical record from 3 different sources in multiple formats ranging from structured data to free form text stored (I shit you not) as TIFF files, extract the actual information, and present it for review prior to a visit LLMs were born to do this. > or even just be able to take verbal orders like it used to be: "get a CBC, CMP, TSH, and CXR" GPT-4 of course knew what that meant, although I didn't. I told it to roleplay a potential follow-up: > "Patient with fatigue, weight loss, mild fever. CBC shows anemia, elevated WBC. CMP reveals elevated liver enzymes. TSH normal. CXR clear. Consider ESR, CRP, ANA, and abdominal ultrasound. Possible infection or autoimmune condition. Referral to a hematologist may be warranted."
- dogmatism 3y ago
  I know an LLM can do this when given the data in an ingestible form it's the taking of data from disparate sources including literal paper faxes that's the hard part
  - Sai_ 3y ago
    You’re being dogmatic about your, what seems to me, objections to LLM skills. Making data ingestible is just laborious and unappetising to an engineer. So hand it over to a data entry operator who’ll do this happily for a wage. I’m not sure why the issue of making data available to an LLM is a failure of the LLM. Btw, OpenAI’s LLM can very well parse images, read handwriting, fix sloppy spellings, and extract objects from unstructured data.
  - vidarh 3y ago
    There's no need for paper faxes - fax is a digital format. But of course, if you intentionally and pointlessly output to formats that make input to an LLM more work, it will be hard. Other than that dealing with disparate data formats is something LLMs can already do very well, and that can be improved fairly significantly by putting a "straightjacket" on it (making it use APIs or selecting only tokens that fits a given grammar)
modeless 3y ago
I mean, have you used GPT-4? Free form text is its bread and butter, and yes, it can absolutely ingest images and read text in them, and combine that with other information from multiple sources in structured or unstructured text or image form, no problem! What you are describing sounded decades away last year. Today it's not something you can rely on quite yet if you need accuracy. But if you still believe it's decades away then you need to update your perception of what's going on in AI right now. Forget the free GPT-3.5 or Bard, you really need to try GPT-4 to understand what's about to happen.
- dogmatism 3y ago
  see my comment above Pt sees Dr in office A, has labs and scans ordered. Labs order is printed out and given to pt, who takes it to lab A. Scan #1 is ordered, and order faxed to facility B, who independently calls, schedules, performs, and interprets scan #1, same for scan #2, but at a different facility. Office A has no electronic link to any of these facilities How does LLM get the data? A human knows the patterns in the area, and calls around for it, then collates it and feeds in to office A's EHR, where if lucky, LLM can do it's thing LLM is only saving the bare minimum of the work In an industry that is still reliant on the fax, I'm not holding my breath for the advent of the LLM savior
  - modeless 3y ago
    In the situation you describe humans fail constantly. My experience with the medical system has decidedly not been that magically competent back office people call every office you've ever visited to assemble your complete medical history from scraps and effectively organize it for the doctor. It's a constant game of telephone where you have to do most of the work yourself, calling around and giving instructions to receptionists to give to other people to get this stuff done. They can't even communicate reliably between the nurses and doctors in the same damn office to avoid asking the same questions multiple times every time you visit. What you describe, if it worked reliably, would be an advance over current human-level average performance. And AI will absolutely be able to do it, too. Probably within ten years, though it will take longer to deploy after the capabilities are there.
vidarh 3y ago
So you describe a couple of things LLMs seem near ideal for, and one thing that is a horrific failure of information management that it's shocking if doctors are wasting their time on, and a case for other automation. At my GP at least, all of this is presented to doctors digitally, all notes are digital, and all requisition of follow up tests are digital, and my initial consultations are all phone or video and only follow up visits are in person and only if a physical examination or tests are needed.

arcticfox 3y ago

> Call me when an LLM can ingest a pts medical record from 3 different sources in multiple formats ranging from structured data to free form text stored (I shit you not) as TIFF files, extract the actual information, and present it for review prior to a visit LLMs were born to do this. > or even just be able to take verbal orders like it used to be: "get a CBC, CMP, TSH, and CXR" GPT-4 of course knew what that meant, although I didn't. I told it to roleplay a potential follow-up: > "Patient with fatigue, weight loss, mild fever. CBC shows anemia, elevated WBC. CMP reveals elevated liver enzymes. TSH normal. CXR clear. Consider ESR, CRP, ANA, and abdominal ultrasound. Possible infection or autoimmune condition. Referral to a hematologist may be warranted."

dogmatism 3y ago
I know an LLM can do this when given the data in an ingestible form it's the taking of data from disparate sources including literal paper faxes that's the hard part
- Sai_ 3y ago
  You’re being dogmatic about your, what seems to me, objections to LLM skills. Making data ingestible is just laborious and unappetising to an engineer. So hand it over to a data entry operator who’ll do this happily for a wage. I’m not sure why the issue of making data available to an LLM is a failure of the LLM. Btw, OpenAI’s LLM can very well parse images, read handwriting, fix sloppy spellings, and extract objects from unstructured data.
- vidarh 3y ago
  There's no need for paper faxes - fax is a digital format. But of course, if you intentionally and pointlessly output to formats that make input to an LLM more work, it will be hard. Other than that dealing with disparate data formats is something LLMs can already do very well, and that can be improved fairly significantly by putting a "straightjacket" on it (making it use APIs or selecting only tokens that fits a given grammar)

modeless 3y ago

I mean, have you used GPT-4? Free form text is its bread and butter, and yes, it can absolutely ingest images and read text in them, and combine that with other information from multiple sources in structured or unstructured text or image form, no problem! What you are describing sounded decades away last year. Today it's not something you can rely on quite yet if you need accuracy. But if you still believe it's decades away then you need to update your perception of what's going on in AI right now. Forget the free GPT-3.5 or Bard, you really need to try GPT-4 to understand what's about to happen.

dogmatism 3y ago
see my comment above Pt sees Dr in office A, has labs and scans ordered. Labs order is printed out and given to pt, who takes it to lab A. Scan #1 is ordered, and order faxed to facility B, who independently calls, schedules, performs, and interprets scan #1, same for scan #2, but at a different facility. Office A has no electronic link to any of these facilities How does LLM get the data? A human knows the patterns in the area, and calls around for it, then collates it and feeds in to office A's EHR, where if lucky, LLM can do it's thing LLM is only saving the bare minimum of the work In an industry that is still reliant on the fax, I'm not holding my breath for the advent of the LLM savior
- modeless 3y ago
  In the situation you describe humans fail constantly. My experience with the medical system has decidedly not been that magically competent back office people call every office you've ever visited to assemble your complete medical history from scraps and effectively organize it for the doctor. It's a constant game of telephone where you have to do most of the work yourself, calling around and giving instructions to receptionists to give to other people to get this stuff done. They can't even communicate reliably between the nurses and doctors in the same damn office to avoid asking the same questions multiple times every time you visit. What you describe, if it worked reliably, would be an advance over current human-level average performance. And AI will absolutely be able to do it, too. Probably within ten years, though it will take longer to deploy after the capabilities are there.

vidarh 3y ago

So you describe a couple of things LLMs seem near ideal for, and one thing that is a horrific failure of information management that it's shocking if doctors are wasting their time on, and a case for other automation. At my GP at least, all of this is presented to doctors digitally, all notes are digital, and all requisition of follow up tests are digital, and my initial consultations are all phone or video and only follow up visits are in person and only if a physical examination or tests are needed.