4 ms·

I’m very glad you set the argument this way, since it very neatly sets the stage for the core disagreement to be argued. > Simply very good information synthes

I’m very glad you set the argument this way, since it very neatly sets the stage for the core disagreement to be argued. > Simply very good information synthesizers > Simple very good syntactic synthesizers I argue that theorists tend to adhere to the former, and practitioners the latter. Consider prototyping: For an expert, it costs nothing (or near nothing) to check the output. You even start better, because you know how to ask the right questions to the tool in the first place. LLMs dont get the semantics right, they get syntactic correlations right. When a subject is common place (literature), the correlations are good enough to get the reasoning right. There is an IMPRESSIVE amount that gets done with just this. However, expecting to get reasoning right is the bridge too far. Wrong expectations of tools, result in bad management projects and failures. You can test this out right now. IF an LLM is able to do information synthesis, it is trivial to set up parallel prompts to work as teams. Try it out. I did, I know why it can’t work as a result. The simple question to ask is “whats your error rate, and your hallucination rate?” LLMs are always ‘hallucinating’. I do hope, that the difference eventually becomes academic. That there is so much training material, that correlation is equal to reasoning. However, it will not be reasoning/semantic prediction. Finally - the real world work has properties that are emergent. You can read every spec sheet you like, but when you start assembling components together, they are going to do weird things.

IanCal 3y ago
> IF an LLM is able to do information synthesis, it is trivial to set up parallel prompts to work as teams. Try it out. What do you mean? I've had GPT-4 (3.5 I think too) talking to itself, setup as a planner and someone critiquing a plan, the end result is much better. > However, expecting to get reasoning right is the bridge too far. Hmm, how do you position othello-gpt in this? It builds a world model and makes moves based on that, so it's not just correlation of input (x,y,z typically followed by A so respond A).
- intended 3y ago
  1) Set up multiple prompts to work together and make a web page. See how far that goes, before it’s nonsense talking to nonsense. 2) Production is proof. Get a working LLM based reasoning system that doesnt need its hand held. The process of failing is educational enough. For the record, I would love it if these things work.
resters 3y ago
Interesting comment. I think it is clear that GPT-4 contains a LOT of information. You can ask it explicit factual questions and it often gets the right answer. When it gets the answer wrong, its wording is typically syntactically correct English, or syntactically correct code. I'd argue that just because some of the errors it commits are "semantic errors" such as calling a method by an incorrect (but often similar) name in a segment of code or printing false statement in well-crafted prose, it nonetheless gets a lot of the semantics right. What is reasoning besides a set of language patterns that we define as valid reasoning? Imagine evaluating statements in various formal logics. Nonsense in one can be valid in another based on semantic rules alone. One could derive the model (model as in model-theoretic semantics) of a formal logical system by sampling a list of valid and invalid statements. LLMs are doing that kind of thing, it seems. There are gaps, but they are not necessarily gaps that the LLM itself cannot notice. For example, I will often ask GPT-4 to formulate a plan for something or to create a list of priorities/considerations for an undertaking. I will then ask it to draft an initial plan. After that I will ask it to review/critique its draft based on the initial goals. It typically points out exactly the kinds of gaps that a human would point to as deficiencies that indicate the LLM is not reasoning. In my view, this indicates that the "knowledge" of how to do the task was always aviailable to the LLM, but the interface (or some aspect of the internal implementation) did not allow the knowledge to be applied all at once. This is not necessarily dissimilar from human intellectual work, in which drafts and self-critiquing is not an unreasonable series of steps. I have done some work with parallel promopts and various "roles" for different LLM interlocutors toward the same task. While it does sometimes go off the rails, it seems clear that multiple prompts with role-based instructions do achieve a greater level of analytical rigor than a single prompt.
- intended 3y ago
  I have to admit, that discussing this topic is forcing a precision in the vocabulary used to discuss reason. However, it is a TEDIOUS process. It’s simplest to just make something - the parallel prompts scenario for example. Can you leave your LLMs to their own business, and will they have a workable product at the end of it? As you said it would seem as if they have the “knowledge” of the task. It should not be an issue. However you will not get anywhere. The issue isnt in the LLM - the issue is in misunderstanding what is going on, and therefore expectations. LLMs don’t notice things - in the human manner you assumed they do. It isnt point out gaps, its repeating text patterns. Our habit of dealing with humans is filling in these gaps and supporting an assumption of ‘noticing’ or ‘improvement’. I think it is hard for us to consider changes in text, without seeing the changes in meaning as well. With LLMs you have to accept that it’s not seeing meaning, it’s just seeing correlation.

IanCal 3y ago

> IF an LLM is able to do information synthesis, it is trivial to set up parallel prompts to work as teams. Try it out. What do you mean? I've had GPT-4 (3.5 I think too) talking to itself, setup as a planner and someone critiquing a plan, the end result is much better. > However, expecting to get reasoning right is the bridge too far. Hmm, how do you position othello-gpt in this? It builds a world model and makes moves based on that, so it's not just correlation of input (x,y,z typically followed by A so respond A).

intended 3y ago
1) Set up multiple prompts to work together and make a web page. See how far that goes, before it’s nonsense talking to nonsense. 2) Production is proof. Get a working LLM based reasoning system that doesnt need its hand held. The process of failing is educational enough. For the record, I would love it if these things work.

resters 3y ago

Interesting comment. I think it is clear that GPT-4 contains a LOT of information. You can ask it explicit factual questions and it often gets the right answer. When it gets the answer wrong, its wording is typically syntactically correct English, or syntactically correct code. I'd argue that just because some of the errors it commits are "semantic errors" such as calling a method by an incorrect (but often similar) name in a segment of code or printing false statement in well-crafted prose, it nonetheless gets a lot of the semantics right. What is reasoning besides a set of language patterns that we define as valid reasoning? Imagine evaluating statements in various formal logics. Nonsense in one can be valid in another based on semantic rules alone. One could derive the model (model as in model-theoretic semantics) of a formal logical system by sampling a list of valid and invalid statements. LLMs are doing that kind of thing, it seems. There are gaps, but they are not necessarily gaps that the LLM itself cannot notice. For example, I will often ask GPT-4 to formulate a plan for something or to create a list of priorities/considerations for an undertaking. I will then ask it to draft an initial plan. After that I will ask it to review/critique its draft based on the initial goals. It typically points out exactly the kinds of gaps that a human would point to as deficiencies that indicate the LLM is not reasoning. In my view, this indicates that the "knowledge" of how to do the task was always aviailable to the LLM, but the interface (or some aspect of the internal implementation) did not allow the knowledge to be applied all at once. This is not necessarily dissimilar from human intellectual work, in which drafts and self-critiquing is not an unreasonable series of steps. I have done some work with parallel promopts and various "roles" for different LLM interlocutors toward the same task. While it does sometimes go off the rails, it seems clear that multiple prompts with role-based instructions do achieve a greater level of analytical rigor than a single prompt.

intended 3y ago
I have to admit, that discussing this topic is forcing a precision in the vocabulary used to discuss reason. However, it is a TEDIOUS process. It’s simplest to just make something - the parallel prompts scenario for example. Can you leave your LLMs to their own business, and will they have a workable product at the end of it? As you said it would seem as if they have the “knowledge” of the task. It should not be an issue. However you will not get anywhere. The issue isnt in the LLM - the issue is in misunderstanding what is going on, and therefore expectations. LLMs don’t notice things - in the human manner you assumed they do. It isnt point out gaps, its repeating text patterns. Our habit of dealing with humans is filling in these gaps and supporting an assumption of ‘noticing’ or ‘improvement’. I think it is hard for us to consider changes in text, without seeing the changes in meaning as well. With LLMs you have to accept that it’s not seeing meaning, it’s just seeing correlation.