4 ms·

Doctors don’t cite claims on the spot

MauranKilom 3y ago
A fair comparison (on a technical level) to GPT-4 RAG would be a doctor in a relevant field who also has internet access. I think this would be indeed interesting to compare to assess the resulting quality of care, so to speak! (The other models being only partially able to source good references is unsurprising/"unfair" on a technical level, but that's not relevant for assessing their safety.)
esoleyman 3y ago
Give me a few minutes and I can pull up any number of medical studies or references to back up my claims. I don’t have them memorized to the actual URL but I have kept up to date with the latest studies and summaries that pertain to my field and my patients.
SkyPuncher 3y ago
Neither do most professionals- unless they’re doing a prepared segment. The difference is LLMs can’t back their claims.
numpad0 3y ago
Some engineers can cite RFCs and ISO standards on the spot sometimes, probably same for doctors
- Loggias 3y ago
  [dead]
larsiusprime 3y ago
They often give clinical recommendations and prescriptions, usually after a quick skim of the medical history and a rushed five minute conversation with the patient. It would be nice to know how many of these typical in-office recommendations wind up being actually backed by the current state of the research, whether a citation is given in the visit or not.
- salad-tycoon 3y ago
  It’s more like this, there is an algorithm / flow chart. You follow it down and do the next step in the process. You learn the reason why and then you just follow whatever the latest update to the algorithm is. Doctors are then graded by administrators as to how well they follow that flow chart. Have you done this or that screening, is BP well controlled, etc. Medicare sets many of the standards, you look up the USPTF guidelines for some more idea of what your doctor should be doing. So yeah. Kinda like factory work. “Pt male 70 wakes up urinate - does he have signs of infection > yes/ no> proceed to no > next step do this test > prescribe this med > re evaluate in 2-4 weeks. “
  - __loam 3y ago
    Why would we use proven processes involving experts with over a decade of education like this when we could use an LLM thrown together by a software developer who may not have even taken a biology course?
    - razodactyl 3y ago
      Spot on

A fair comparison (on a technical level) to GPT-4 RAG would be a doctor in a relevant field who also has internet access. I think this would be indeed interesting to compare to assess the resulting quality of care, so to speak! (The other models being only partially able to source good references is unsurprising/"unfair" on a technical level, but that's not relevant for assessing their safety.)

esoleyman 3y ago

Give me a few minutes and I can pull up any number of medical studies or references to back up my claims. I don’t have them memorized to the actual URL but I have kept up to date with the latest studies and summaries that pertain to my field and my patients.

SkyPuncher 3y ago

Neither do most professionals- unless they’re doing a prepared segment. The difference is LLMs can’t back their claims.

numpad0 3y ago

Some engineers can cite RFCs and ISO standards on the spot sometimes, probably same for doctors

Loggias 3y ago
[dead]

larsiusprime 3y ago

They often give clinical recommendations and prescriptions, usually after a quick skim of the medical history and a rushed five minute conversation with the patient. It would be nice to know how many of these typical in-office recommendations wind up being actually backed by the current state of the research, whether a citation is given in the visit or not.

salad-tycoon 3y ago
It’s more like this, there is an algorithm / flow chart. You follow it down and do the next step in the process. You learn the reason why and then you just follow whatever the latest update to the algorithm is. Doctors are then graded by administrators as to how well they follow that flow chart. Have you done this or that screening, is BP well controlled, etc. Medicare sets many of the standards, you look up the USPTF guidelines for some more idea of what your doctor should be doing. So yeah. Kinda like factory work. “Pt male 70 wakes up urinate - does he have signs of infection > yes/ no> proceed to no > next step do this test > prescribe this med > re evaluate in 2-4 weeks. “
- __loam 3y ago
  Why would we use proven processes involving experts with over a decade of education like this when we could use an LLM thrown together by a software developer who may not have even taken a biology course?
  - razodactyl 3y ago
    Spot on