4 ms·
> There will be changes in direction sometimes, but that’s because they also appear sometimes in the training text. Yeah. My point is essentially that I think
by twiss 4y ago
> There will be changes in direction sometimes, but that’s because they also appear sometimes in the training text.
Yeah. My point is essentially that I think they don't appear in the training text often enough, since it's trained on written, edited text, which don't often contain changes of opinion mid-answer, unlike spoken text, for example.
Perhaps if it could somehow be trained on a dataset of humans writing responses, including editing (e.g. removal of previously-written words), that might lead to better responses, since it can then edit the answer after reasoning it through (in OP's example).
Or, to simplify things, perhaps it should be allowed to emit a token meaning "actually, scrap all that, let me start over" when it detects a contradiction in its own response so far. (The model would still continue to see its own response including that token, but the UI could remove everything until the last instance of the token, if any.)
Or, again alternatively, if we want a response closer to spoken text, it could be trained on dialogues from humans that change their mind occasionally, and then verbalize that explicitly (e.g. actually emit "actually, scrap all that, let me start over").
- skybrian 4y agoExcept, why would it start over? A language model doesn’t actually have an opinion independent of whatever text it’s outputting. The prompt is the opinion. Start over with a new chat session and it will emit different words, causing it to have a different opinion. If you could edit the chat history directly, you would change its opinion and it would continue from there. It’s true that if the training data contained a lot of text where the authors changed their minds a lot then the language model would learn to “change its mind” a lot in a similar way, but neither the old opinion nor the new opinion is its “real” opinion. It’s just the opinion it’s pretending to have now. It would be tricky to come up with training data where, when an author changes its mind, the new text is always better than what they had before. Sometimes edits make things worse. It might be fun to try, though.
- twiss 4y agoIn the example, it "gained" additional information by reasoning through the steps. At that point, it should be more likely to give the correct answer. In general, I think also when humans are able to reason about their opinion and change their mind if necessary, it leads to better results on average :)
- fourthark 4y agoThat would be great. I don’t think it can detect the contradiction, though. It doesn’t have any logical model behind it.
- twiss 4y agoIt essentially did detect a contradiction, and "resolved" it by trying to square its initial (wrong) response with the reasoning afterwards (which contained the correct response). Perhaps if it would've been able to delete the initial paragraph of its own response it might have.