4 ms·
I wonder to what extent this is because ChatGPT is trained to take any feedback as a sign that its answer is wrong and needs to be corrected. Such a training wo
by FeepingCreature 3y ago
I wonder to what extent this is because ChatGPT is trained to take any feedback as a sign that its answer is wrong and needs to be corrected. Such a training would, I suspect, explain the effect where the model first generates the correct answer, then introspects and self-doubts. In training, when it was told to introspect, that would usually be a hint that its answer was wrong. This results in an agent whose confidence reliably falters under introspection.
I've observed that ChatGPT quite eagerly "corrects" itself at the first sign of negative feedback, even if its answer was already correct. It's actually quite annoying.
If this is correct, a more balanced training that contains cases where the human response is wrong would solve this issue.
- kromem 3y agoThis is brought up in the paper and is effectively their reasoning. That because it is instructed to "find problems" with the answer it leads to correct answers being biased towards changing to incorrect more often than it can successfully bias incorrect answers towards correct ones. Here, I think the research team might have had more success using neutral language like "grade this answer" where both verification and challenge would fit the prompt as opposed to "find problems with this answer" where only challenges would - but I'm skeptical it would be enough of an improvement that it would yield a net improvement on self-critique over the initial answer. There are other more specialized approaches I'd expect will always work better than feeding an incorrect answer back into what generated it in the first place (i.e. a backwards pass where the model has to match the answer provided to multiple possible questions with one being the question asked and the others generated on the fly).