2 ms·
Yes, “Chain of Thought” is a trick to make a model that predicts just a next word to come up with a conclusion that matches intermediate steps. Still, model do
by coolspot 4y ago
Yes, “Chain of Thought” is a trick to make a model that predicts just a next word to come up with a conclusion that matches intermediate steps.
Still, model doesn’t reason, but rather provides step-by-step “reasoning” using the same “predict the next word” mechanism.
- nl 4y agoThis is an incomplete understanding of what very large LMs are doing. At a very large scale new behaviors emerge[1][2]. It's true that the fluency of language models is easily explained by "predict next token given context" but that doesn't preclude the fact the LLMs are functionally doing reasoning up to some limits. To quote: > However, it is unclear how these models obtain the answers and whether they rely on simple heuristics rather than the generated chain-of-thought. To enable systematic exploration of the reasoning ability of LLMs, we present a new synthetic question-answering dataset called PrOntoQA, where each example is generated from a synthetic world model represented in first-order logic. This allows us to parse the generated chain-of-thought into symbolic proofs for formal analysis. Our analysis on InstructGPT and GPT-3 shows that LLMs are quite capable of making correct individual deduction steps, and so are generally capable of reasoning, even in fictional contexts. However, they have difficulty with proof planning: When multiple valid deduction steps are available, they are not able to systematically explore the different options. from "Language Models Can (kind of) Reason: A Systematic Formal Analysis of Chain-of-Thought"[3] To summarise that paper, they create imaginary scenarios and get the LLM to answer questions. For example: > Q: Every vumpus is a numpus. Each vumpus is dull. Dumpuses are vumpuses. Every dumpus is not hot. Every impus is a dumpus. Impuses are brown. Wumpuses are impuses. Wumpuses are opaque. Tumpuses are wumpuses. Tumpuses are small. Every yumpus is hot. Zumpuses are tumpuses. Zumpuses are feisty. Rompuses are zumpuses. Every rompus is not kind. Each jompus is a rompus. Jompuses are sour. Alex is a zumpus. True or false: Alex is hot. A: (Appendix A.3) This is a "5-hop" reasoning question, which the LLM fails at. But at 3 hop InstructGTP (davinci-002) scores 75% correct for completely fictional ontologies. [1] https://arxiv.org/pdf/2206.07682.pdf https://arxiv.org/pdf/2206.07682.pdf [2] https://hai.stanford.edu/news/examining-emergent-abilities-large-language-models https://hai.stanford.edu/news/examining-emergent-abilities-l... [3] https://openreview.net/forum?id=qFVVBzXxR2V https://openreview.net/forum?id=qFVVBzXxR2V
- coolspot 4y agoYes, LLMs are fascinating and there are many characteristics that are not completely understood. I am not _much_ surprised it can follow connections and do some math, given that it is Turing-complete. However my point in the initial comment was that whatever contradiction you have in logic of the prompt, won’t break the model, because it would just reply whatever is a statistically reasonable response to such a prompt.