5 ms·

Did you read the article you posted? -ChatGPT initially gives the same diagnosis the vets did, Babesiosis -It notes that Babesiosis may have either been a mis

by Zircom 3y ago

Did you read the article you posted? -ChatGPT initially gives the same diagnosis the vets did, Babesiosis -It notes that Babesiosis may have either been a misdiagnosis or there may be a secondary condition/infection causing the remaining symptoms after the Babesiosis treatment didn't resolve all of them -It suggests such a hypothetical secondary condition could be IMHA, which the article notes is an extremely common complication of Babesiosis with this specific dog breed -A quick Google search brings up a fair amount of literature about the association between Babesiosis and IMHA So in fact this is the opposite of a never before seen situation, ChatGPT was just regurgitating common comorbidities of Babesiosis and the vets in question are terrible at their job.

sillysaurusx 3y ago
Are you suggesting that it was commonplace for machine learning models to be able to extrapolate medical case reports into an actual diagnosis? Even being able to read and understand a case report is a minor miracle in extrapolation, and it’s interesting how far the goal posts move.
- lossolo 3y ago
  I have another example of ChatGPT not generalizing but just being a really good statistical model. I needed a solution to a problem that you can't find on Google, and a variation of a problem that also can't be found on Google. I attempted to obtain around 15 lines of code from ChatGPT that would solve the problem, but it consistently failed to produce the correct solution. I spent a few hours trying different prompts, indicating its errors and receiving apologies, only for it to generate another incorrect solution while acknowledging its mistake. Solving out of distribution problems correctly seems almost impossible for it.
  - sillysaurusx 3y ago
    GPT 3.5 or 4? Surprisingly it makes a huge difference. I think a lot of peoples’ impressions are with 3.5, but many startups couldn’t have been built on it, whereas with 4 they can. If it was 4 I’d be curious about the specific problem if you’d be willing to link to the chat.
    - avereveard 3y ago
      I had similar issue with gpt4, was looking for a library that given a grammar and a string, would produce a list of next valid symbols. Gpt4 only ever suggested grammar validator to the point I had given up and was going to write a grammar generator, and so I started looking for the equivalent of antlr in python, and in three searches I find out nltk.grammar that actually solves the original problem It's not a new library either, so I'm dumbfounded by why gpt4 couldn't make sense of my request.
      - disgruntledphd2 3y ago
        If it hasn't been used much/talked about much for this purpose then you wouldn't expect it to, right?
        avereveard 3y ago
        true, but then it's neither interpolating between the public documentation nor extrapolating from the possible similarities between concepts
      - IanCal 3y ago
        I tried this to see. It does suggest using validators, though in a way that mostly solves the problem. The failure mode is when the string is valid complete or extended. It does have a bit of a workaround for that. https://chat.openai.com/share/54cb8876-96ff-4434-a479-4d2ddebf14cb https://chat.openai.com/share/54cb8876-96ff-4434-a479-4d2dde...
        avereveard 3y ago
        using the exception only works at whole symbol level, i.e. with "cat do" as imput No terminal matches 'd' in the current parser context, at line 1 col 5 cat do ^ Expected one of: * CAT * SPACE * FISH * DOG Next valid symbols: ['* CAT\n\t* SPACE\n\t* FISH\n\t* DOG']
        3y ago

by Zircom 3y ago

sillysaurusx 3y ago

Are you suggesting that it was commonplace for machine learning models to be able to extrapolate medical case reports into an actual diagnosis? Even being able to read and understand a case report is a minor miracle in extrapolation, and it’s interesting how far the goal posts move.

lossolo 3y ago
I have another example of ChatGPT not generalizing but just being a really good statistical model. I needed a solution to a problem that you can't find on Google, and a variation of a problem that also can't be found on Google. I attempted to obtain around 15 lines of code from ChatGPT that would solve the problem, but it consistently failed to produce the correct solution. I spent a few hours trying different prompts, indicating its errors and receiving apologies, only for it to generate another incorrect solution while acknowledging its mistake. Solving out of distribution problems correctly seems almost impossible for it.
- sillysaurusx 3y ago
  GPT 3.5 or 4? Surprisingly it makes a huge difference. I think a lot of peoples’ impressions are with 3.5, but many startups couldn’t have been built on it, whereas with 4 they can. If it was 4 I’d be curious about the specific problem if you’d be willing to link to the chat.
  - avereveard 3y ago
    I had similar issue with gpt4, was looking for a library that given a grammar and a string, would produce a list of next valid symbols. Gpt4 only ever suggested grammar validator to the point I had given up and was going to write a grammar generator, and so I started looking for the equivalent of antlr in python, and in three searches I find out nltk.grammar that actually solves the original problem It's not a new library either, so I'm dumbfounded by why gpt4 couldn't make sense of my request.
    - disgruntledphd2 3y ago
      If it hasn't been used much/talked about much for this purpose then you wouldn't expect it to, right?
      - avereveard 3y ago
        true, but then it's neither interpolating between the public documentation nor extrapolating from the possible similarities between concepts
    - IanCal 3y ago
      I tried this to see. It does suggest using validators, though in a way that mostly solves the problem. The failure mode is when the string is valid complete or extended. It does have a bit of a workaround for that. https://chat.openai.com/share/54cb8876-96ff-4434-a479-4d2ddebf14cb https://chat.openai.com/share/54cb8876-96ff-4434-a479-4d2dde...
      - avereveard 3y ago
        using the exception only works at whole symbol level, i.e. with "cat do" as imput No terminal matches 'd' in the current parser context, at line 1 col 5 cat do ^ Expected one of: * CAT * SPACE * FISH * DOG Next valid symbols: ['* CAT\n\t* SPACE\n\t* FISH\n\t* DOG']
        3y ago