4 ms·

> Based on these observations, I'd bet the house that no, large language models cannot under any circumstance reason about medical questions at present. Wtf? I

> Based on these observations, I'd bet the house that no, large language models cannot under any circumstance reason about medical questions at present. Wtf? It just did. Have you read the paper? Wake up.

dekhn 4y ago
It's unclear where in the paper they demonstrate, unequivocally, that reasoning occurred, rather than pure statistical model-fitting (we generally believe this is one of the key differences between AGI and modern LLMs)
- rafaelero 4y ago
  It gives me the correct answer (as judged by experts) to a complex medical problem. If you want to keep saying this isn't reasoning, be my guest. But IMO that's irrelevant.
  - dekhn 4y ago
    Spitting out the results of a bunch of experts is the opposite of reasoning. Or more correctly, it's one limited form of reasoning: an ensemble prediction made from a mixture of experts. That's not an interesting form of reasoning because literally all the decisions are encoded in the weights you apply to the experts.
    - rafaelero 4y ago
      You seem very confused and I invite you to think again about what you said.
- mahathu 4y ago
  > reasoning occurred, rather than pure statistical model-fitting What's the difference?
  - dekhn 4y ago
    reasoning takes into account much more contextual information. I think many would also say it's a step-by-step process in which the explicit assumptions and conclusions could be stated clearly, although I think that's probably just anthropomorphism and projection.
arcturus17 4y ago
Have you asked GPT-3.5 about something as simple as the classification of sailfish? Wake up.
- rafaelero 4y ago
  Are you going to keep repeating this forever? This paper is not even using GPT-3.5, it's using PaLM.
  - arcturus17 4y ago
    > This paper is not even using GPT-3.5, it's using PaLM. Where are you getting this? From the excerpt: "We set out to investigate whether GPT-3.5 (Codex and InstructGPT) can be applied to answer and reason about difficult real-world-based questions" I am going to be repeating that ChatGPT cannot classify popular animals correctly as many times as needed. I've been using the app daily and it's quite frequently factually wrong about many other topics, too (for example, programming, or history).
    - rafaelero 4y ago
      Oh, I mistook this paper for the one that was released more recently from DeepMind + Google. [1] [1] Can large language models reason about medical questions? https://arxiv.org/abs/2212.13138 https://arxiv.org/abs/2212.13138
      - dekhn 4y ago
        Elsewhere on this thread, you're rude to me whilst I made an intelligent comment in good faith. Based on your current comment (arguing with people over technical details), may I suggest you perhaps back off a bit and not be so aggressive about "being right"?
        rafaelero 4y ago
        Sorry, I did think you were not doing a good faith argument. But if you are, then let me ask you: if you believe that reaching expert's opinion is not evidence of reasoning then what are medical students doing for 6 years at college?

dekhn 4y ago

It's unclear where in the paper they demonstrate, unequivocally, that reasoning occurred, rather than pure statistical model-fitting (we generally believe this is one of the key differences between AGI and modern LLMs)

rafaelero 4y ago
It gives me the correct answer (as judged by experts) to a complex medical problem. If you want to keep saying this isn't reasoning, be my guest. But IMO that's irrelevant.
- dekhn 4y ago
  Spitting out the results of a bunch of experts is the opposite of reasoning. Or more correctly, it's one limited form of reasoning: an ensemble prediction made from a mixture of experts. That's not an interesting form of reasoning because literally all the decisions are encoded in the weights you apply to the experts.
  - rafaelero 4y ago
    You seem very confused and I invite you to think again about what you said.
mahathu 4y ago
> reasoning occurred, rather than pure statistical model-fitting What's the difference?
- dekhn 4y ago
  reasoning takes into account much more contextual information. I think many would also say it's a step-by-step process in which the explicit assumptions and conclusions could be stated clearly, although I think that's probably just anthropomorphism and projection.

arcturus17 4y ago

Have you asked GPT-3.5 about something as simple as the classification of sailfish? Wake up.

rafaelero 4y ago
Are you going to keep repeating this forever? This paper is not even using GPT-3.5, it's using PaLM.
- arcturus17 4y ago
  > This paper is not even using GPT-3.5, it's using PaLM. Where are you getting this? From the excerpt: "We set out to investigate whether GPT-3.5 (Codex and InstructGPT) can be applied to answer and reason about difficult real-world-based questions" I am going to be repeating that ChatGPT cannot classify popular animals correctly as many times as needed. I've been using the app daily and it's quite frequently factually wrong about many other topics, too (for example, programming, or history).
  - rafaelero 4y ago
    Oh, I mistook this paper for the one that was released more recently from DeepMind + Google. [1] [1] Can large language models reason about medical questions? https://arxiv.org/abs/2212.13138 https://arxiv.org/abs/2212.13138
    - dekhn 4y ago
      Elsewhere on this thread, you're rude to me whilst I made an intelligent comment in good faith. Based on your current comment (arguing with people over technical details), may I suggest you perhaps back off a bit and not be so aggressive about "being right"?
      - rafaelero 4y ago
        Sorry, I did think you were not doing a good faith argument. But if you are, then let me ask you: if you believe that reaching expert's opinion is not evidence of reasoning then what are medical students doing for 6 years at college?