5 ms·

Be careful and don't trust all it says. Sometimes it invents API functions which are not there, or doesn't see existing. And always very confident till you poin

Be careful and don't trust all it says. Sometimes it invents API functions which are not there, or doesn't see existing. And always very confident till you point it.

1f60c 3y ago
And then it’s like, “thank you for telling me about this, I’ll remember that for next time,” which is how a human ought to respond but not how ChatGPT actually learns.
- simonw 3y ago
  Hah, yeah that's so frustrating. It's memory is reset every time you start a new conversation, but it doesn't make that at all clear to people.
  - gverrilla 3y ago
    It also resets inside a single conversation if you go beyond a certain number of tokens, and as far as I know there is no warning whatsoever to the user when it happens.
- taneq 3y ago
  The obvious next thing to try would be to continuously fine tune the model on these conversations, so it actually fulfills the promises most of these models make about "learning continuously". I haven't yet seen any actual implementations of this, though. I'm sure someone's tried it, I wonder how it went.
  - simonw 3y ago
    Which models make promises about "learning continuously"?
    - taneq 3y ago
      I’ve had statements along that line from Bing, dolphin2.1 and I think maybe Wizard Vicuna.
simonw 3y ago
I'm finding that code is the area where hallucinations matter the least... because if it hallucinates an API function that doesn't exist, the mistake becomes apparent the moment you actually try to run it. It's like having an automated fact checker! I wish I had the same thing for the other kinds of output it produces.
- bugglebeetle 3y ago
  It will, however from time to time insert lines and variables that do nothing, but could result in bugs or confusion if not removed. I’ve encountered these hallucinations a few times. Overall, I agree with your sentiment, but I think it’s important to note that running isn’t always the indicator or correctness we think it to be.
  - BbzzbB 3y ago
    You're still supposed to read it, like you hopefully wouldn't blindly paste a big code block from SO. A useless/unused line or variable doesn't seem that hard to spot?
    - bugglebeetle 3y ago
      Of course, but ease of detection can vary relative to the complexity of the code being returned. GPT-4, correctly prompted, can produce some pretty complicated stuff. But it also hallucinates in ways that are more subtle than one might think. The example I’m thinking of, it created an unused variable in a set of fairly complex ML training set up scripts that I mostly caught because I was familiar with all the proper inputs. But the unused variable was quite plausible if you were not familiar, new to the domain etc.
      - gowld 3y ago
        Compilers automatically detect unused variables. Unused variables are the last of your problems. You should be far, far more worried about all the misused variables.
  - 3y ago
smsm42 3y ago
I see it all the time. Even more annoying is when the API exists, but in a different class or with different parameters and outputs than you need, and the model would claim it does exactly what you need right until the time you try to use it and discover it can't work in your case.
PeterStuer 3y ago
Often I find that the hallucinations is how the API or lib should have been if it was more sane. Maybe someone could turn this into a virtual API critique.

Be careful and don't trust all it says. Sometimes it invents API functions which are not there, or doesn't see existing. And always very confident till you point it.

1f60c 3y ago

And then it’s like, “thank you for telling me about this, I’ll remember that for next time,” which is how a human ought to respond but not how ChatGPT actually learns.

simonw 3y ago
Hah, yeah that's so frustrating. It's memory is reset every time you start a new conversation, but it doesn't make that at all clear to people.
- gverrilla 3y ago
  It also resets inside a single conversation if you go beyond a certain number of tokens, and as far as I know there is no warning whatsoever to the user when it happens.
taneq 3y ago
The obvious next thing to try would be to continuously fine tune the model on these conversations, so it actually fulfills the promises most of these models make about "learning continuously". I haven't yet seen any actual implementations of this, though. I'm sure someone's tried it, I wonder how it went.
- simonw 3y ago
  Which models make promises about "learning continuously"?
  - taneq 3y ago
    I’ve had statements along that line from Bing, dolphin2.1 and I think maybe Wizard Vicuna.

simonw 3y ago

I'm finding that code is the area where hallucinations matter the least... because if it hallucinates an API function that doesn't exist, the mistake becomes apparent the moment you actually try to run it. It's like having an automated fact checker! I wish I had the same thing for the other kinds of output it produces.

bugglebeetle 3y ago
It will, however from time to time insert lines and variables that do nothing, but could result in bugs or confusion if not removed. I’ve encountered these hallucinations a few times. Overall, I agree with your sentiment, but I think it’s important to note that running isn’t always the indicator or correctness we think it to be.
- BbzzbB 3y ago
  You're still supposed to read it, like you hopefully wouldn't blindly paste a big code block from SO. A useless/unused line or variable doesn't seem that hard to spot?
  - bugglebeetle 3y ago
    Of course, but ease of detection can vary relative to the complexity of the code being returned. GPT-4, correctly prompted, can produce some pretty complicated stuff. But it also hallucinates in ways that are more subtle than one might think. The example I’m thinking of, it created an unused variable in a set of fairly complex ML training set up scripts that I mostly caught because I was familiar with all the proper inputs. But the unused variable was quite plausible if you were not familiar, new to the domain etc.
    - gowld 3y ago
      Compilers automatically detect unused variables. Unused variables are the last of your problems. You should be far, far more worried about all the misused variables.
- 3y ago

smsm42 3y ago

I see it all the time. Even more annoying is when the API exists, but in a different class or with different parameters and outputs than you need, and the model would claim it does exactly what you need right until the time you try to use it and discover it can't work in your case.

PeterStuer 3y ago

Often I find that the hallucinations is how the API or lib should have been if it was more sane. Maybe someone could turn this into a virtual API critique.