5 ms·
Interesting comment. I think it is clear that GPT-4 contains a LOT of information. You can ask it explicit factual questions and it often gets the right answer
by resters 3y ago
Interesting comment.
I think it is clear that GPT-4 contains a LOT of information. You can ask it explicit factual questions and it often gets the right answer. When it gets the answer wrong, its wording is typically syntactically correct English, or syntactically correct code.
I'd argue that just because some of the errors it commits are "semantic errors" such as calling a method by an incorrect (but often similar) name in a segment of code or printing false statement in well-crafted prose, it nonetheless gets a lot of the semantics right.
What is reasoning besides a set of language patterns that we define as valid reasoning? Imagine evaluating statements in various formal logics. Nonsense in one can be valid in another based on semantic rules alone.
One could derive the model (model as in model-theoretic semantics) of a formal logical system by sampling a list of valid and invalid statements.
LLMs are doing that kind of thing, it seems. There are gaps, but they are not necessarily gaps that the LLM itself cannot notice.
For example, I will often ask GPT-4 to formulate a plan for something or to create a list of priorities/considerations for an undertaking. I will then ask it to draft an initial plan. After that I will ask it to review/critique its draft based on the initial goals. It typically points out exactly the kinds of gaps that a human would point to as deficiencies that indicate the LLM is not reasoning.
In my view, this indicates that the "knowledge" of how to do the task was always aviailable to the LLM, but the interface (or some aspect of the internal implementation) did not allow the knowledge to be applied all at once. This is not necessarily dissimilar from human intellectual work, in which drafts and self-critiquing is not an unreasonable series of steps.
I have done some work with parallel promopts and various "roles" for different LLM interlocutors toward the same task. While it does sometimes go off the rails, it seems clear that multiple prompts with role-based instructions do achieve a greater level of analytical rigor than a single prompt.
- intended 3y agoI have to admit, that discussing this topic is forcing a precision in the vocabulary used to discuss reason. However, it is a TEDIOUS process. It’s simplest to just make something - the parallel prompts scenario for example. Can you leave your LLMs to their own business, and will they have a workable product at the end of it? As you said it would seem as if they have the “knowledge” of the task. It should not be an issue. However you will not get anywhere. The issue isnt in the LLM - the issue is in misunderstanding what is going on, and therefore expectations. LLMs don’t notice things - in the human manner you assumed they do. It isnt point out gaps, its repeating text patterns. Our habit of dealing with humans is filling in these gaps and supporting an assumption of ‘noticing’ or ‘improvement’. I think it is hard for us to consider changes in text, without seeing the changes in meaning as well. With LLMs you have to accept that it’s not seeing meaning, it’s just seeing correlation.
- resters 3y ago> LLMs don’t notice things - in the human manner you assumed they do. It isn't point out gaps, its repeating text patterns. Our habit of dealing with humans is filling in these gaps and supporting an assumption of ‘noticing’ or ‘improvement’. True. It is important to avoid anthropomorphizing LLMs, etc. I do this intentionally now and then but I agree it is dangerous to do it accidentally. > I think it is hard for us to consider changes in text, without seeing the changes in meaning as well. True, but since LLMs were trained on text sequences that had meaning, much of the meaning was accidentally embodied in the resulting model. Areas where LLMs seem to reason well happen to be the areas where the training data was sufficiently generalized and the language tokens used similarly enough... such as recipes. > With LLMs you have to accept that it’s not seeing meaning, it’s just seeing correlation. True. I think it is interesting how much it often feels like knowledge. I think we also have to be careful not to overly glorify human "knowledge" as something other than producing a pattern of output signals in response to a pattern of input signals.
- intended 3y agoI agree. I think precision in all cases makes it possible to pursue the technology, as opposed to almost dabbling in superstition and philosophy. However, it’s taken an inordinate amount of time to achieve even the finesse shared in this comment chain. It’s a challenging topic. We have to carve out levels of utility for systems that generate output given inputs. Some way to distinguish between what our wetware achieves, and what LLMs achieve - Without putting our models on a pedestal.