3 ms·
The model can’t break down, neither it can reason about contradictions. All it can do is to predict most probable next word for a given input.
by coolspot 4y ago
The model can’t break down, neither it can reason about contradictions. All it can do is to predict most probable next word for a given input.
- nl 4y ago> In “Chain of Thought Prompting Elicits Reasoning in Large Language Models,” we explore a prompting method for improving the reasoning abilities of language models. Called chain of thought prompting, this method enables models to decompose multi-step problems into intermediate steps. With chain of thought prompting, language models of sufficient scale (~100B parameters) can solve complex reasoning problems that are not solvable with standard prompting methods. https://ai.googleblog.com/2022/05/language-models-perform-reasoning-via.html https://ai.googleblog.com/2022/05/language-models-perform-re...
- coolspot 4y agoYes, “Chain of Thought” is a trick to make a model that predicts just a next word to come up with a conclusion that matches intermediate steps. Still, model doesn’t reason, but rather provides step-by-step “reasoning” using the same “predict the next word” mechanism.
- nl 4y agoThis is an incomplete understanding of what very large LMs are doing. At a very large scale new behaviors emerge[1][2]. It's true that the fluency of language models is easily explained by "predict next token given context" but that doesn't preclude the fact the LLMs are functionally doing reasoning up to some limits. To quote: > However, it is unclear how these models obtain the answers and whether they rely on simple heuristics rather than the generated chain-of-thought. To enable systematic exploration of the reasoning ability of LLMs, we present a new synthetic question-answering dataset called PrOntoQA, where each example is generated from a synthetic world model represented in first-order logic. This allows us to parse the generated chain-of-thought into symbolic proofs for formal analysis. Our analysis on InstructGPT and GPT-3 shows that LLMs are quite capable of making correct individual deduction steps, and so are generally capable of reasoning, even in fictional contexts. However, they have difficulty with proof planning: When multiple valid deduction steps are available, they are not able to systematically explore the different options. from "Language Models Can (kind of) Reason: A Systematic Formal Analysis of Chain-of-Thought"[3] To summarise that paper, they create imaginary scenarios and get the LLM to answer questions. For example: > Q: Every vumpus is a numpus. Each vumpus is dull. Dumpuses are vumpuses. Every dumpus is not hot. Every impus is a dumpus. Impuses are brown. Wumpuses are impuses. Wumpuses are opaque. Tumpuses are wumpuses. Tumpuses are small. Every yumpus is hot. Zumpuses are tumpuses. Zumpuses are feisty. Rompuses are zumpuses. Every rompus is not kind. Each jompus is a rompus. Jompuses are sour. Alex is a zumpus. True or false: Alex is hot. A: (Appendix A.3) This is a "5-hop" reasoning question, which the LLM fails at. But at 3 hop InstructGTP (davinci-002) scores 75% correct for completely fictional ontologies. [1] https://arxiv.org/pdf/2206.07682.pdf https://arxiv.org/pdf/2206.07682.pdf [2] https://hai.stanford.edu/news/examining-emergent-abilities-large-language-models https://hai.stanford.edu/news/examining-emergent-abilities-l... [3] https://openreview.net/forum?id=qFVVBzXxR2V https://openreview.net/forum?id=qFVVBzXxR2V
- coolspot 4y agoYes, LLMs are fascinating and there are many characteristics that are not completely understood. I am not _much_ surprised it can follow connections and do some math, given that it is Turing-complete. However my point in the initial comment was that whatever contradiction you have in logic of the prompt, won’t break the model, because it would just reply whatever is a statistically reasonable response to such a prompt.