3 ms·

And do you think that “caval hiatus,” the phrase that you used in your prompts, is the term most likely to elicit that from LLMs? I ask because using plain med

And do you think that “caval hiatus,” the phrase that you used in your prompts, is the term most likely to elicit that from LLMs? I ask because using plain medical language seems to elicit the correct answer from Claude and ChatGPT3.5 (I didn’t test llama from my phone.)

haldujai 3y ago
As a radiologist who looks at thoracoabdominal anatomy for a living I completely agree. That was also my initial reaction, I don't think I've ever read "caval hiatus" used in clinical text other than maybe during medical school. It's not even correct, anatomically it's most correctly described as the caval foramen. Hiatus refers to a slit/gap, foramen refers to a circumscribed orifice. Caval hiatus is either misremembering or an archaic text. Clinically people use aperture, foramen or opening. Other problems with this post: I assume the incredibly poor way of prompting is attempting to simulate a user using a chat interface? If so, are they even using the right LLaMA model (at the very least chat fine-tuned, ideally also on the open medical conversation dataset)? What GPT model are we comparing with? Is there system context provided for the prompt? If you were deploying this you wouldn't only pass the inputed user text in the prompt. ChatGPT's interface adds this so if you're going to make a fair comparison you should be doing the same. It's still going to favor OpenAI for this specific use case (vague and poorly phrased short prompts of esoteric trivia) as they've had much more training in this domain.
- wink 3y ago
  As a non-native speaker I wonder if this could still be of use because it's maybe better than using a dictionary. I think getting medical terms right is one of the hardest things to do in a hurry (like when visiting a doctor on vacation, not if you're living in an English-speaking country). You're only guessing anyway. Can't believe I'm somehow defending bad LLM output, but here I am, even with this huge caveat,
  - haldujai 3y ago
    Potentially, but no patient or physician is going to be using "caval hiatus". If this was the intention, the author should disclose that he's intentionally using very obscure antiquated wording not "basic human anatomy". If he is instead unaware that "caval hiatus" is not accepted terminology it is further evidence supporting how obscure this fact is.
    - johntiger1 3y ago
      It's literally being taught in classes at a major Canadian medical school, in 2023.
      - haldujai 3y ago
        Which means nothing, you have a pre-clerkship anatomy course director using archaic and obscure terminology that is extremely rarely used (and arguably incorrect, but we don't have to debate etymology). You will never see this used clinically or in any major anatomy textbook. As you are a medical student you should be familiar with accepted references for anatomy. In case you're not here are some: Netter's anatomy (most commonly used clinical anatomy text): caval foramen. Gray's anatomy: inferior vena cava opening. eAnatomy: caval foramen. Your other clue is in ChatGPT's response: >The term "caval hiatus" is typically used in reference to human anatomy, specifically the anatomical structure known as the "caval opening" or "caval foramen.".... If you ask Claude the same question using 'caval foramen': >The caval foramen, also known as the foramen of vena cava, is located at the level of T8 vertebra in the thoracic region of the vertebral column. Specifically... Some friendly advice as you seem to be early in your medical training: 1. When an attending (in this case two) corrects you, you're probably wrong. Your next step should be to confirm your knowledge not retort with "it's what I was taught". 2. Never say a "major medical school" in Canada (where I also graduated from) which implies there are tiers. Unlike the US, all Canadian medical schools and (almost all) residency programs are excellent.

haldujai 3y ago

As a radiologist who looks at thoracoabdominal anatomy for a living I completely agree. That was also my initial reaction, I don't think I've ever read "caval hiatus" used in clinical text other than maybe during medical school. It's not even correct, anatomically it's most correctly described as the caval foramen. Hiatus refers to a slit/gap, foramen refers to a circumscribed orifice. Caval hiatus is either misremembering or an archaic text. Clinically people use aperture, foramen or opening. Other problems with this post: I assume the incredibly poor way of prompting is attempting to simulate a user using a chat interface? If so, are they even using the right LLaMA model (at the very least chat fine-tuned, ideally also on the open medical conversation dataset)? What GPT model are we comparing with? Is there system context provided for the prompt? If you were deploying this you wouldn't only pass the inputed user text in the prompt. ChatGPT's interface adds this so if you're going to make a fair comparison you should be doing the same. It's still going to favor OpenAI for this specific use case (vague and poorly phrased short prompts of esoteric trivia) as they've had much more training in this domain.

wink 3y ago
As a non-native speaker I wonder if this could still be of use because it's maybe better than using a dictionary. I think getting medical terms right is one of the hardest things to do in a hurry (like when visiting a doctor on vacation, not if you're living in an English-speaking country). You're only guessing anyway. Can't believe I'm somehow defending bad LLM output, but here I am, even with this huge caveat,
- haldujai 3y ago
  Potentially, but no patient or physician is going to be using "caval hiatus". If this was the intention, the author should disclose that he's intentionally using very obscure antiquated wording not "basic human anatomy". If he is instead unaware that "caval hiatus" is not accepted terminology it is further evidence supporting how obscure this fact is.
  - johntiger1 3y ago
    It's literally being taught in classes at a major Canadian medical school, in 2023.
    - haldujai 3y ago
      Which means nothing, you have a pre-clerkship anatomy course director using archaic and obscure terminology that is extremely rarely used (and arguably incorrect, but we don't have to debate etymology). You will never see this used clinically or in any major anatomy textbook. As you are a medical student you should be familiar with accepted references for anatomy. In case you're not here are some: Netter's anatomy (most commonly used clinical anatomy text): caval foramen. Gray's anatomy: inferior vena cava opening. eAnatomy: caval foramen. Your other clue is in ChatGPT's response: >The term "caval hiatus" is typically used in reference to human anatomy, specifically the anatomical structure known as the "caval opening" or "caval foramen.".... If you ask Claude the same question using 'caval foramen': >The caval foramen, also known as the foramen of vena cava, is located at the level of T8 vertebra in the thoracic region of the vertebral column. Specifically... Some friendly advice as you seem to be early in your medical training: 1. When an attending (in this case two) corrects you, you're probably wrong. Your next step should be to confirm your knowledge not retort with "it's what I was taught". 2. Never say a "major medical school" in Canada (where I also graduated from) which implies there are tiers. Unlike the US, all Canadian medical schools and (almost all) residency programs are excellent.