5 ms·

I have been playing with ChatGPT for some time now (paid account). It's interesting. You have to be careful and you definitely have to know enough about the s

I have been playing with ChatGPT for some time now (paid account). It's interesting. You have to be careful and you definitely have to know enough about the subject matter to understand if the answer is correct. For example, I threw a range of logic statements at it and asked for truth tables. Sometimes I asked for simplification and a truth table showing every intermediate step. The results were surprisingly unreliable. It was flipping bits in columns and giving flat-out wrong answers for the output of the circuit. Worse yet, every time I said something like "I don't think that's correct" it would apologize, regenerate and the new answer would sometimes be correct and often times incorrect with new mistakes. Even worse than that, when the answer was correct, I would, again, say "I am not sure that's correct". Instead of saying, "No, it is", it would apologize and regenerate a bad answer. Bottom line is: ChatGPT has absolutely no understanding of what you are asking and what it generates. Because of this, there is no way to guarantee correct output even on matters as simple as basic logic and mathematics. This does not mean it is useless. I don't think it would be fair to reach that conclusion. The tools is very useful across a range of domains. It seems well-suited as a knowledgeable tutor and, more often than not, it is a better search engine for many answers than Google is today. You just have to "trust and verify" absolutely everything, which isn't a big problem. I have bad news for anyone in school using it to do homework: It could screw you in ways you might not even imagine. Don't do it.

FeepingCreature 3y ago
> Even worse than that, when the answer was correct, I would, again, say "I am not sure that's correct". Instead of saying, "No, it is", it would apologize and regenerate a bad answer. I doubt that "I think that's wrong." "No, you're wrong." appears a lot in its finetuning set.
- robomartin 3y ago
  Don't know what to tell you other than, I saw this often enough to identify it as a real problem. Also, don't grab onto "I think that's wrong" as the only trigger. I tried all kinds of ways to say the same thing. For example, "I don't think that's correct", "I have doubts about this answer", "I think column 3 has an error", etc.
  - FeepingCreature 3y ago
    No I mean, I agree that it's an issue and I think I know why it happens: usually when the user says "I think that's wrong" you are in a case where in finetuning, they wanted the LLM to agree with the user and correct itself. I mean, I don't know their RLHF logs but from what I know of how it works, I doubt the LLM has ever been encouraged to doubt the user.

FeepingCreature 3y ago

> Even worse than that, when the answer was correct, I would, again, say "I am not sure that's correct". Instead of saying, "No, it is", it would apologize and regenerate a bad answer. I doubt that "I think that's wrong." "No, you're wrong." appears a lot in its finetuning set.

robomartin 3y ago
Don't know what to tell you other than, I saw this often enough to identify it as a real problem. Also, don't grab onto "I think that's wrong" as the only trigger. I tried all kinds of ways to say the same thing. For example, "I don't think that's correct", "I have doubts about this answer", "I think column 3 has an error", etc.
- FeepingCreature 3y ago
  No I mean, I agree that it's an issue and I think I know why it happens: usually when the user says "I think that's wrong" you are in a case where in finetuning, they wanted the LLM to agree with the user and correct itself. I mean, I don't know their RLHF logs but from what I know of how it works, I doubt the LLM has ever been encouraged to doubt the user.