5 ms·

I'm not really sure what he expected here, ChatGPT was not trained to be a doctor, it is far more general than that. Asking ChatGPT for medical advice is like

I'm not really sure what he expected here, ChatGPT was not trained to be a doctor, it is far more general than that. Asking ChatGPT for medical advice is like asking someone who is very well read but has no experience as a doctor, and in that context it's doing very well. He also brings up one of the most salient points without really visiting it enough: ChatGPT does not ask for clarification, because it is not a knowledge base trying to find an answer. All it does is figure out what character is statistically most likely to come next, it has no heuristic to know that there is a task it hasn't fully completed. This is the same reason ChatGPT cannot yet write programs by itself: in order to do so you'd need to specify the entire program up front (which is exactly what code is). As soon as we have agents that can do a proper feedback loop of querying a LLM consecutively until some heuristic is reached then the kind of AI doctors are looking for will emerge.

izzydata 4y ago
He probably didn't have any expectations. Merely experimentation and observation and maybe a bit of word of warning after seeing the results for anyone thinking of using it in ways it shouldn't be.
onlyrealcuzzo 4y ago
> All it does is figure out what character is statistically most likely to come next To be pedantic, it's a token - not just a character - right?
- gpm 4y ago
  Yes
JacobDotVI 4y ago
My hunch is that this is exactly what he was expecting. There is a lot of hype around ChatGPT passing the medical exam and this exercise is a counter point to that.
- famouswaffles 4y ago
  any example that doesn't use the current sota isn't a very good counter point to be honest. 3 barely passed. 4 aced it. For all we know, GPT-4 erases most of his concerns (not saying it would).
- viraptor 4y ago
  The medical exam has very specific questions which define what you're expected to include in the answer. The question asked in this case was nowhere near that detailed, do I don't think they're comparable. To really evaluate something beyond the "random generic user" level, you need to be familiar with the tech as well. The article really tells us more about the experience of someone with no chatgpt knowledge checking their own symptoms rather than its usability for emergency diagnosis.
- thrill 4y ago
  GPT-4 passed the medical exam, not ChatGPT running GPT-3. There's a rather significant difference.
  - nyokodo 4y ago
    > GPT-4 passed the medical exam, not ChatGPT running GPT-3 With a subscription you can use GPT-4 with ChatGPT. ChatGPT is just the wrapper to the model.
    - hansihe 4y ago
      That's not true, ChatGPT is a model. Quote from the ChatGPT announcement post: > ChatGPT is a sibling model to InstructGPT, which is trained to follow an instruction in a prompt and provide a detailed response.
      - heartbreak 4y ago
        I’m looking at the ChatGPT interface right now (paid account) and I have a “Model:” drop-down at the start of a new chat that says: - Legacy (GPT-3) - Default (GPT-3.5) - GPT-4
bentcorner 4y ago
> ChatGPT does not ask for clarification, because it is not a knowledge base trying to find an answer. All it does is figure out what character is statistically most likely to come next, it has no heuristic to know that there is a task it hasn't fully completed. It absolutely can, but it requires the right prompts. If you just say "give me a diagnosis" it won't, but if you prompt it with a "you are a doctor and I am a patient" kind of prompt, it will. I'll agree that it tends to "close off" conversations without a lot of open endedness - I role-played through a session using myself as a patient with a chronic condition I suffer from and it correctly identified the problem, but I had to push it a little to get it to try to narrow down the problem (instead of "here's a bunch of things it could be" and ending the convo there).
- colonCapitalDee 4y ago
  ChatGPT's heuristic may lead it to emit a sequence of words that suggest to English speakers that they should provide clarification, but ChatGPT is congenitally incapable of determining that answering a question requires more information that was provided in the prompt and that clarification is needed. Rephrased, ChatGPT can ask for clarification, but only as a side effect of satisfying it's heuristic.
  - paradite 4y ago
    > but ChatGPT is congenitally incapable of determining that answering a question requires more information that was provided in the prompt and that clarification is needed. What's stopping the ChatGPT heuristic / model from determining that? That's within the capabilities of a statistical model.
    - Mezzie 4y ago
      I would say that ChatGPT like systems could probably be built that would understand when they need more context, but they would fail at effectively eliciting that context from their human users.
      - 4y ago
viraptor 4y ago
> ChatGPT does not ask for clarification It depends on how you use it. Knowing how it works, you can explicitly check "What further questions for the patient would make the diagnosis more certain?" ChatGPT is not magic - if you ask for diagnosis based on some info, that's exactly what you're going to get.
dannyphantom 4y ago
I don't mean this in any sort of negative context - curating your online presence is a never-ending task - but I believe that the authors primary expectation was to drive traffic to their online accounts by publishing a post on the subject matter. A cursory peek at their accounts show that this was published across all of them with Twitter showing the farthest reach. > Dr... is vice president of innovation at {company_name}, an innovation incubator for health tech.
barking_biscuit 4y ago
> This is the same reason ChatGPT cannot yet write programs by itself: in order to do so you'd need to specify the entire program up front (which is exactly what code is). As soon as we have agents that can do a proper feedback loop of querying a LLM consecutively until some heuristic is reached then the kind of AI doctors are looking for will emerge. https://github.com/Torantulino/Auto-GPT https://github.com/Torantulino/Auto-GPT
rvz 4y ago
> I'm not really sure what he expected here, ChatGPT was not trained to be a doctor, it is far more general than that. Except that O̶p̶e̶n̶AI.com and many others on the AI hype brigade were showing ChatGPT and GPT-4 to be passing "Medical Licensing Exams" with flying colours and it turns out to be a confident bullshitter offering dangerous untrustworthy medical advice when tested by a medical professional as I said before [0]. So they (O̶p̶e̶n̶AI.com and AI bros) set those high expectations from the start. > As soon as we have agents that can do a proper feedback loop of querying a LLM consecutively until some heuristic is reached then the kind of AI doctors are looking for will emerge. No. A black-box neural network AI model that is unable to transparently explain and reason about its own decisions is fundamental to the trust element that medical professionals are looking for. It is more than just spitting out numbers from a computer. This doctor is also the one still reviewing the so-called 'diagnosis' from ChatGPT which without him, ChatGPT alone would have certainly killed the patient. The human doctor also gave a transparent explanation about why ChatGPT's 'diagnosis' was incredibly wrong. ChatGPT was confident that it was still right. [0] https://news.ycombinator.com/item?id=35311996 https://news.ycombinator.com/item?id=35311996
gerdesj 4y ago
"I'm not really sure what he expected here, ChatGPT ..." Doctoring is quite similar to ITing if you are careful with your similes. In medicals you look for indications and contraindications. In IT you look for magic beans and sometimes look at logfiles. I have no idea why we (IT) don't follow the medical way and have indications and contras (I do). They have been doing complicated diagnostics way longer than we have. BTW: It doesn't worry about a char but about which word will come next. ChatGPT has written several programs for me. One of them was decent after much negotiation. The rest needed some changes. At one point I told it to change language (BASH to Python) for a script and it did. I spend 20 mins fining stuff and cracked on.
- HillRat 4y ago
  There was an old USENIX symposium where one of the talks was, "If computers had blood, they'd call us doctors." My thought at the time was, "If computers had blood, they'd call us serial killers."
- marcosdumay 4y ago
  > I have no idea why we (IT) don't follow the medical way and have indications and contras (I do). Because we have logs. Ignoring them and focusing on the symptoms would be like doctors refusing to do examinations on their patients.
  - iudqnolq 4y ago
    Doctors separate their sources of information into signs (objectively observed) and symptoms (subjectively reported). Both signs and symptoms can be indication or contradictions. Tangentially I wonder if there's sources of diagnostic info in tech that are closer to symptoms than signs. You might separate out potentially unreliable info reported by the suspect system from more objective info other systems see.

izzydata 4y ago

He probably didn't have any expectations. Merely experimentation and observation and maybe a bit of word of warning after seeing the results for anyone thinking of using it in ways it shouldn't be.

onlyrealcuzzo 4y ago

> All it does is figure out what character is statistically most likely to come next To be pedantic, it's a token - not just a character - right?

gpm 4y ago
Yes

JacobDotVI 4y ago

My hunch is that this is exactly what he was expecting. There is a lot of hype around ChatGPT passing the medical exam and this exercise is a counter point to that.

famouswaffles 4y ago
any example that doesn't use the current sota isn't a very good counter point to be honest. 3 barely passed. 4 aced it. For all we know, GPT-4 erases most of his concerns (not saying it would).
viraptor 4y ago
The medical exam has very specific questions which define what you're expected to include in the answer. The question asked in this case was nowhere near that detailed, do I don't think they're comparable. To really evaluate something beyond the "random generic user" level, you need to be familiar with the tech as well. The article really tells us more about the experience of someone with no chatgpt knowledge checking their own symptoms rather than its usability for emergency diagnosis.
thrill 4y ago
GPT-4 passed the medical exam, not ChatGPT running GPT-3. There's a rather significant difference.
- nyokodo 4y ago
  > GPT-4 passed the medical exam, not ChatGPT running GPT-3 With a subscription you can use GPT-4 with ChatGPT. ChatGPT is just the wrapper to the model.
  - hansihe 4y ago
    That's not true, ChatGPT is a model. Quote from the ChatGPT announcement post: > ChatGPT is a sibling model to InstructGPT, which is trained to follow an instruction in a prompt and provide a detailed response.
    - heartbreak 4y ago
      I’m looking at the ChatGPT interface right now (paid account) and I have a “Model:” drop-down at the start of a new chat that says: - Legacy (GPT-3) - Default (GPT-3.5) - GPT-4

bentcorner 4y ago

> ChatGPT does not ask for clarification, because it is not a knowledge base trying to find an answer. All it does is figure out what character is statistically most likely to come next, it has no heuristic to know that there is a task it hasn't fully completed. It absolutely can, but it requires the right prompts. If you just say "give me a diagnosis" it won't, but if you prompt it with a "you are a doctor and I am a patient" kind of prompt, it will. I'll agree that it tends to "close off" conversations without a lot of open endedness - I role-played through a session using myself as a patient with a chronic condition I suffer from and it correctly identified the problem, but I had to push it a little to get it to try to narrow down the problem (instead of "here's a bunch of things it could be" and ending the convo there).

colonCapitalDee 4y ago
ChatGPT's heuristic may lead it to emit a sequence of words that suggest to English speakers that they should provide clarification, but ChatGPT is congenitally incapable of determining that answering a question requires more information that was provided in the prompt and that clarification is needed. Rephrased, ChatGPT can ask for clarification, but only as a side effect of satisfying it's heuristic.
- paradite 4y ago
  > but ChatGPT is congenitally incapable of determining that answering a question requires more information that was provided in the prompt and that clarification is needed. What's stopping the ChatGPT heuristic / model from determining that? That's within the capabilities of a statistical model.
  - Mezzie 4y ago
    I would say that ChatGPT like systems could probably be built that would understand when they need more context, but they would fail at effectively eliciting that context from their human users.
    - 4y ago

viraptor 4y ago

> ChatGPT does not ask for clarification It depends on how you use it. Knowing how it works, you can explicitly check "What further questions for the patient would make the diagnosis more certain?" ChatGPT is not magic - if you ask for diagnosis based on some info, that's exactly what you're going to get.

dannyphantom 4y ago

I don't mean this in any sort of negative context - curating your online presence is a never-ending task - but I believe that the authors primary expectation was to drive traffic to their online accounts by publishing a post on the subject matter. A cursory peek at their accounts show that this was published across all of them with Twitter showing the farthest reach. > Dr... is vice president of innovation at {company_name}, an innovation incubator for health tech.

barking_biscuit 4y ago

> This is the same reason ChatGPT cannot yet write programs by itself: in order to do so you'd need to specify the entire program up front (which is exactly what code is). As soon as we have agents that can do a proper feedback loop of querying a LLM consecutively until some heuristic is reached then the kind of AI doctors are looking for will emerge. https://github.com/Torantulino/Auto-GPT https://github.com/Torantulino/Auto-GPT

rvz 4y ago

> I'm not really sure what he expected here, ChatGPT was not trained to be a doctor, it is far more general than that. Except that O̶p̶e̶n̶AI.com and many others on the AI hype brigade were showing ChatGPT and GPT-4 to be passing "Medical Licensing Exams" with flying colours and it turns out to be a confident bullshitter offering dangerous untrustworthy medical advice when tested by a medical professional as I said before [0]. So they (O̶p̶e̶n̶AI.com and AI bros) set those high expectations from the start. > As soon as we have agents that can do a proper feedback loop of querying a LLM consecutively until some heuristic is reached then the kind of AI doctors are looking for will emerge. No. A black-box neural network AI model that is unable to transparently explain and reason about its own decisions is fundamental to the trust element that medical professionals are looking for. It is more than just spitting out numbers from a computer. This doctor is also the one still reviewing the so-called 'diagnosis' from ChatGPT which without him, ChatGPT alone would have certainly killed the patient. The human doctor also gave a transparent explanation about why ChatGPT's 'diagnosis' was incredibly wrong. ChatGPT was confident that it was still right. [0] https://news.ycombinator.com/item?id=35311996 https://news.ycombinator.com/item?id=35311996

gerdesj 4y ago

"I'm not really sure what he expected here, ChatGPT ..." Doctoring is quite similar to ITing if you are careful with your similes. In medicals you look for indications and contraindications. In IT you look for magic beans and sometimes look at logfiles. I have no idea why we (IT) don't follow the medical way and have indications and contras (I do). They have been doing complicated diagnostics way longer than we have. BTW: It doesn't worry about a char but about which word will come next. ChatGPT has written several programs for me. One of them was decent after much negotiation. The rest needed some changes. At one point I told it to change language (BASH to Python) for a script and it did. I spend 20 mins fining stuff and cracked on.

HillRat 4y ago
There was an old USENIX symposium where one of the talks was, "If computers had blood, they'd call us doctors." My thought at the time was, "If computers had blood, they'd call us serial killers."
marcosdumay 4y ago
> I have no idea why we (IT) don't follow the medical way and have indications and contras (I do). Because we have logs. Ignoring them and focusing on the symptoms would be like doctors refusing to do examinations on their patients.
- iudqnolq 4y ago
  Doctors separate their sources of information into signs (objectively observed) and symptoms (subjectively reported). Both signs and symptoms can be indication or contradictions. Tangentially I wonder if there's sources of diagnostic info in tech that are closer to symptoms than signs. You might separate out potentially unreliable info reported by the suspect system from more objective info other systems see.