4 ms·

I think the second response may partially be because it gave the answer first and the reasoning second. If it would've removed the first paragraph (with the ini

by twiss 4y ago

I think the second response may partially be because it gave the answer first and the reasoning second. If it would've removed the first paragraph (with the initial answer) and last 3 paragraphs (with the tortuous attempt to square the initial answer with the correct answer), the response would actually be correct. ChatGPT seems unable to change its mind mid-answer - likely because few texts on the internet do so, as the author would just edit it. Perhaps if ChatGPT would be able to edit its own response somehow, it might end up with a better response?

skybrian 4y ago
Large language models work by choosing words [1] one at a time, much like autocomplete. It’s choosing the most consistent next word (with some randomness). It doesn’t actually care who wrote the previous words; whatever came before is the prompt. When writing multi-word answers, it’s effectively prompting itself for another word. It’s going to try to choose a word that’s consistent with what it previously wrote. If it’s in the middle of writing a wrong answer, the best autocompletion is usually to continue to be wrong. There will be changes in direction sometimes, but that’s because they also appear sometimes in the training text. This explains why writing out the steps before the answer often works. It’s going to try to choose the words in the answer so they are consistent with the previous steps. [1] tokens actually, but I’ll stick with “words” to keep things simple.
- twiss 4y ago
  > There will be changes in direction sometimes, but that’s because they also appear sometimes in the training text. Yeah. My point is essentially that I think they don't appear in the training text often enough, since it's trained on written, edited text, which don't often contain changes of opinion mid-answer, unlike spoken text, for example. Perhaps if it could somehow be trained on a dataset of humans writing responses, including editing (e.g. removal of previously-written words), that might lead to better responses, since it can then edit the answer after reasoning it through (in OP's example). Or, to simplify things, perhaps it should be allowed to emit a token meaning "actually, scrap all that, let me start over" when it detects a contradiction in its own response so far. (The model would still continue to see its own response including that token, but the UI could remove everything until the last instance of the token, if any.) Or, again alternatively, if we want a response closer to spoken text, it could be trained on dialogues from humans that change their mind occasionally, and then verbalize that explicitly (e.g. actually emit "actually, scrap all that, let me start over").
  - skybrian 4y ago
    Except, why would it start over? A language model doesn’t actually have an opinion independent of whatever text it’s outputting. The prompt is the opinion. Start over with a new chat session and it will emit different words, causing it to have a different opinion. If you could edit the chat history directly, you would change its opinion and it would continue from there. It’s true that if the training data contained a lot of text where the authors changed their minds a lot then the language model would learn to “change its mind” a lot in a similar way, but neither the old opinion nor the new opinion is its “real” opinion. It’s just the opinion it’s pretending to have now. It would be tricky to come up with training data where, when an author changes its mind, the new text is always better than what they had before. Sometimes edits make things worse. It might be fun to try, though.
    - twiss 4y ago
      In the example, it "gained" additional information by reasoning through the steps. At that point, it should be more likely to give the correct answer. In general, I think also when humans are able to reason about their opinion and change their mind if necessary, it leads to better results on average :)
  - fourthark 4y ago
    That would be great. I don’t think it can detect the contradiction, though. It doesn’t have any logical model behind it.
    - twiss 4y ago
      It essentially did detect a contradiction, and "resolved" it by trying to square its initial (wrong) response with the reasoning afterwards (which contained the correct response). Perhaps if it would've been able to delete the initial paragraph of its own response it might have.
blueblimp 4y ago
That example reminded me of dreaming. Whatever random elements come up in a dream, my mind finds a way to rationalize them as making sense. (At least until I awake and realize that it was nonsense.)
- JustAndy 4y ago
  That's mainly because activity in the prefrontal cortex is very low - that's the part of the brain that handles problem solving, comprehension etc. and reasoning.
mannykannot 4y ago
Picking part of a response and discarding the rest is problematical, and this shows why: The whole response, with its self-contradictions, makes it clear that ChatGPT does not in any sense understand the issue, despite the fact that a cherry-picked part of it is a correct argument. Here, it continued its correct argument with a passage that directly contradicted it, so there seems no reason to suppose that, if given an opportunity to edit its own response, it would pick out the good parts significantly better than chance. In fact, if this passage had been spoken by a person, we might well think he was trying to edit his initial response, and doing so pretty badly!

by twiss 4y ago

skybrian 4y ago

Large language models work by choosing words [1] one at a time, much like autocomplete. It’s choosing the most consistent next word (with some randomness). It doesn’t actually care who wrote the previous words; whatever came before is the prompt. When writing multi-word answers, it’s effectively prompting itself for another word. It’s going to try to choose a word that’s consistent with what it previously wrote. If it’s in the middle of writing a wrong answer, the best autocompletion is usually to continue to be wrong. There will be changes in direction sometimes, but that’s because they also appear sometimes in the training text. This explains why writing out the steps before the answer often works. It’s going to try to choose the words in the answer so they are consistent with the previous steps. [1] tokens actually, but I’ll stick with “words” to keep things simple.

twiss 4y ago
> There will be changes in direction sometimes, but that’s because they also appear sometimes in the training text. Yeah. My point is essentially that I think they don't appear in the training text often enough, since it's trained on written, edited text, which don't often contain changes of opinion mid-answer, unlike spoken text, for example. Perhaps if it could somehow be trained on a dataset of humans writing responses, including editing (e.g. removal of previously-written words), that might lead to better responses, since it can then edit the answer after reasoning it through (in OP's example). Or, to simplify things, perhaps it should be allowed to emit a token meaning "actually, scrap all that, let me start over" when it detects a contradiction in its own response so far. (The model would still continue to see its own response including that token, but the UI could remove everything until the last instance of the token, if any.) Or, again alternatively, if we want a response closer to spoken text, it could be trained on dialogues from humans that change their mind occasionally, and then verbalize that explicitly (e.g. actually emit "actually, scrap all that, let me start over").
- skybrian 4y ago
  Except, why would it start over? A language model doesn’t actually have an opinion independent of whatever text it’s outputting. The prompt is the opinion. Start over with a new chat session and it will emit different words, causing it to have a different opinion. If you could edit the chat history directly, you would change its opinion and it would continue from there. It’s true that if the training data contained a lot of text where the authors changed their minds a lot then the language model would learn to “change its mind” a lot in a similar way, but neither the old opinion nor the new opinion is its “real” opinion. It’s just the opinion it’s pretending to have now. It would be tricky to come up with training data where, when an author changes its mind, the new text is always better than what they had before. Sometimes edits make things worse. It might be fun to try, though.
  - twiss 4y ago
    In the example, it "gained" additional information by reasoning through the steps. At that point, it should be more likely to give the correct answer. In general, I think also when humans are able to reason about their opinion and change their mind if necessary, it leads to better results on average :)
- fourthark 4y ago
  That would be great. I don’t think it can detect the contradiction, though. It doesn’t have any logical model behind it.
  - twiss 4y ago
    It essentially did detect a contradiction, and "resolved" it by trying to square its initial (wrong) response with the reasoning afterwards (which contained the correct response). Perhaps if it would've been able to delete the initial paragraph of its own response it might have.

blueblimp 4y ago

That example reminded me of dreaming. Whatever random elements come up in a dream, my mind finds a way to rationalize them as making sense. (At least until I awake and realize that it was nonsense.)

JustAndy 4y ago
That's mainly because activity in the prefrontal cortex is very low - that's the part of the brain that handles problem solving, comprehension etc. and reasoning.

mannykannot 4y ago

Picking part of a response and discarding the rest is problematical, and this shows why: The whole response, with its self-contradictions, makes it clear that ChatGPT does not in any sense understand the issue, despite the fact that a cherry-picked part of it is a correct argument. Here, it continued its correct argument with a passage that directly contradicted it, so there seems no reason to suppose that, if given an opportunity to edit its own response, it would pick out the good parts significantly better than chance. In fact, if this passage had been spoken by a person, we might well think he was trying to edit his initial response, and doing so pretty badly!