6 ms·

I think that if they would, that would be very surprising and indicative of a lot of wastefulness inside the model architecture. All these tests are simple sing

by tinco 3y ago

I think that if they would, that would be very surprising and indicative of a lot of wastefulness inside the model architecture. All these tests are simple single prompt experiments, so the LLM's get no chance to reason about their responses. They're just system 1 thinking, the equivalent of putting a gun to someone's head and asking them to solve a large division in 2 seconds. I bet a lot of these experiments would already solvable by putting the LLM in a simple loop with some helper prompts that make it restructure and validate its answers, form theories and get to explore multiple lines of thought. If an LLM would be able to do that in a single prompt, without a loop (so the LLM always answers in a predictable amount of time), then it would mean its entire reasoning structure is repeated horizontally through the layers of its architecture. That would be both limiting (i.e. limit the depth of the reasoning to the width of the network) and very expensive to train.

rf15 3y ago
they can't reason though, sadly - the premise does not hold.
- FrustratedMonky 3y ago
  So your premise is correct? Please back up the opposite. If you can, you should publish. These responses are logically the same as "No You".
  - deleted 3y ago
    [deleted]
- TeMPOraL 3y ago
  That's literally GP's point though - they don't reason because reasoning requires a loop, which you deny them and then say "see, it can't reason".
  - deleted 3y ago
    [deleted]
uoaei 3y ago
> They're just system 1 thinking, the equivalent of putting a gun to someone's head and asking them to solve a large division in 2 seconds. No, it's the equivalent of putting a gun to someone's head and asking them "what are my intentions?" Which is readily available to any being with a theory of mind.
- famouswaffles 3y ago
  LLMs don't fail those kind of tasks though. 4 is very good at keeping track of who knows what and why in a story. You can test this yourself.
- choudharism 3y ago
  I don’t think LLMs have theory of mind, but your point is not very strong. You can literally query ChatGPT right now and see that it can figure out intentions (both superficial and deep) of a gun is held to a head quite easily. Because, obviously, training data probably includes a decent amount of motivation breakdowns as a function of coercion. It doesn’t know why, but it knows what to say.
- FrustratedMonky 3y ago
  Don't think so. Put gun to persons ahead. Ask them to do a division. Then screaming at them "HOW DID YOU DO THAT, TELL ME NOW, OR YOU'RE TOAST". Even most humans would splutter and not be able to answer.
  - uoaei 3y ago
    The division problem is arbitrary and irrelevant to any notion of theory of mind. If you have a better example, you can feel free to offer one. I already did so above.
    - FrustratedMonky 3y ago
      The point is not about the example of division, it is about 'tell me your intentions'. A human also can't explain how their neurons calculated a division, or anything else, or 'intention', even if a gun is pointed at them. So, why is that a fault of AI if it can't explain how itself is working? It is not a requirement for AGI.
Androider 3y ago
The equivalent for a human would be an reflexive response to a question, the kind you could immediately answer after being woken up at 3am in the morning. That type of answer has been deeply trained into the human networks and also requires no deep insight. But if a human is allowed time and internal reasoning iterations, so should the LLM when determining if it has deep insight. Right now we're simply observing input -> output of LLMs, the equivalent of snap answers from a human. But nothing says it couldn't instead be an input -> extensive internal dialogue, maybe even between multiple expert models for seconds, minutes or hours, that are not at all visible to the prompter -> final insightful answer. Maybe future LLMs will say, "let me get back to you on that".
- I_Am_Nous 3y ago
  This reminds me of The Last Question by Isaac Asimov. I also think if we stopped expecting all LLMs to have an immediate answer, it would be relatively easy to shim some kind of "conscience" to direct the output in different ways. Similar to the safeties already in place in LLMs, but instead of it just saying "NO DON'T SAY THAT" it can dialog internally to change what the output is until it reaches what it believes to be the agreed upon best answer.
  - lifeisstillgood 3y ago
    It would have an emotional reaction to certain "thought constructs" and would be guided by that. Or we could just give them three laws
    - I_Am_Nous 3y ago
      With current LLMs the three laws might be tough to implement in a way that can't be prompt injected around. That's why I described the extra bits as a "conscience" which could enforce the three laws. Maybe the three laws are the internal conscience's context prompt while the main LLM is more able to think anything in general and then the output is tuned down by the conscience? Otherwise the laws will have to be implemented as weights or during training so the model explicitly knows the laws and would never even be capable of doing something against them.
two_in_one 3y ago
Just note that loop doesn't have to be visible from outside. It can be internal, with another driving thread asking right questions. Inner monologue. Then the summary is given back to user. This will give the model space for 'thinking' with internally generated text much large than the visible prompt + output. This way multi-step logic can be implemented.
- chii 3y ago
  > with another driving thread asking right questions. Inner monologue. spoilers warning: Is that basically the plot to westworld?
  - two_in_one 3y ago
    Inner monologue idea was around for a while. That's interesting that it actually materializes. It was thought that AGI will be operating with some abstracts, logic rules, probability calculations, etc. Not thinking in plain English.
lixy 3y ago
Yep, prototype exactly that this past week. With a strong instruction spec prompt from the start, you can have an AI come up with a much better answer by making sure it knows it has time to answer the questions and how it should approach the problem in stages. The great part is with clear enough directions it also knows how to evaluate whether its done or not.
- hskalin 3y ago
  Can you share the kind of prompts you used?
twobitshifter 3y ago
There are plenty of models that use introspection and check answers, that’s the idea behind let’s think about it step by step.
- tinco 3y ago
  I feel the "let's think about it step by step" is a bit of a hack. To circumvent the fact that there's no external loop you use the fact that it gets re-run on every token so you can store a bit of state in the tokens that it's already generated. Or am I misunderstanding something about that technique?
  - twobitshifter 3y ago
    You are right, it’s sometimes called zero shot chain of thought, but it’s a way of getting the type of thing you are describing to happen. The LLMs somehow process things in a perceived step by step to get a much improved answer. Whether the external loop or an llm imposed internal loop, does it matter? Are our own minds looping or just adding tokens?
    - tinco 3y ago
      Yeah that's true, and I do believe there's a good chance our minds are perpetually adding tokens. But our minds also have an efficient/effective way of dealing with the context cut off. We don't have (or we don't experience) a hard cut off of our memory context. Instead the tokens are increasingly lossily compressed as they age out of our memory, the lossiness amount being based both on time passed but also on some fancy value function. And that combined with a "system" (or trained/fine-tuned in) prompt that motivates the LLM to reason in a way that is conducive to working with that kind of memory would be a sort of single-shot AGI system. Where single-shot is lying a bit because it's just infinitely looping. I guess from that perspective it might make sense to test if such a thing is already happening in current LLM's and my dismissive attitude stems from the fact that I've played with them enough to know that they currently don't.
    - vidarh 3y ago
creer 3y ago
At the same time, if LLMs are based on all or enough human writings then don't they necessarily contain a theory of mind? A rather general, smoothed out and still neurotic one probably. But still just like an LLM can't be expected to have a specific knowledge of hydraulics, it also has read more hydraulics than even experts might be expected to. That's the entire issue about it, right? This issue of "is most of our mind basically just mixing and matching stuff we have seen, read, heard?" Do humans have some magical theory of mind that somehow stands ASIDE from all the "normal" learned stuff? Of course, yes, we do know one thing that's missing in LLMs, which is "loop and helpers" like you describe. Which I'm sure many people are currently hacking at - one way being for the LLM to talk to itself. But as for "a theory of mind", if enough writings served as input, then LLMs do have plenty of that. Another question is whether LLMs are raised to behave like humans (which might be where they most NEED some theory of mind). Of course not. The ones we know most about are only question answerers. The theory of mind they might have (that is not negated by the lack of loop and internal deliberation) may be overwhelmed by the pre- and post-processing: "no sex, no murder plots, talk to the human like they are 5, bla bla bla". And yet you can ask things like "Tell it like you are speaking to 5 year olds who want to have a fun time". Some theory of mind makes it through.
- neuronerdgirl 3y ago
  That's not what theory of mind is. Theory of mind is the understanding - not academic understanding, but functional, meaningful understanding that guides your behavior - that the other people (AIs?) around you have their own separate thoughts and feelings from your own. This is what leads to the stark behavioral differences as a child develops from ~3-6 years of age. Imagine a child is playing with their toys and puts one of them in a box out of the room, then later asks you to retrieve it for them. A 3 year old will assume that, because they know they put it in the box, that you know it is there as well. (Incidentally, this is where a lot of tantrums come from, because they can't understand why you aren't reading their mind and get frustrated.) A 6 year old will not make this assumption. While this is a simple example, the concept extrapolates to more complex issues as we develop cognitively and emotionally. Not only is theory of mind about recognizing that your thoughts and feelings are separate and distinct, but also being able to project your understanding into others to predict how they might be thinking and feeling in their own separate circumstance. This is the fundamental basis of empathy, or being able to predict and understand how others around you may be feeling given their unique current circumstances even if they are different from your own. LLMs have not demonstrated any of the above understanding. Again, I'm sure they could generate a definition for it, and on any given prompt they could probably even generate some generic-ish text that could pose as empathy, but so can a horoscope. But there is not the deep, continuous comprehension of the user as a separate thinking and feeling agent that is fundamental to the idea of theory of mind.

by tinco 3y ago

rf15 3y ago

they can't reason though, sadly - the premise does not hold.

FrustratedMonky 3y ago
So your premise is correct? Please back up the opposite. If you can, you should publish. These responses are logically the same as "No You".
- deleted 3y ago
  [deleted]
TeMPOraL 3y ago
That's literally GP's point though - they don't reason because reasoning requires a loop, which you deny them and then say "see, it can't reason".
- deleted 3y ago
  [deleted]

uoaei 3y ago

> They're just system 1 thinking, the equivalent of putting a gun to someone's head and asking them to solve a large division in 2 seconds. No, it's the equivalent of putting a gun to someone's head and asking them "what are my intentions?" Which is readily available to any being with a theory of mind.

famouswaffles 3y ago
LLMs don't fail those kind of tasks though. 4 is very good at keeping track of who knows what and why in a story. You can test this yourself.
choudharism 3y ago
I don’t think LLMs have theory of mind, but your point is not very strong. You can literally query ChatGPT right now and see that it can figure out intentions (both superficial and deep) of a gun is held to a head quite easily. Because, obviously, training data probably includes a decent amount of motivation breakdowns as a function of coercion. It doesn’t know why, but it knows what to say.
FrustratedMonky 3y ago
Don't think so. Put gun to persons ahead. Ask them to do a division. Then screaming at them "HOW DID YOU DO THAT, TELL ME NOW, OR YOU'RE TOAST". Even most humans would splutter and not be able to answer.
- uoaei 3y ago
  The division problem is arbitrary and irrelevant to any notion of theory of mind. If you have a better example, you can feel free to offer one. I already did so above.
  - FrustratedMonky 3y ago
    The point is not about the example of division, it is about 'tell me your intentions'. A human also can't explain how their neurons calculated a division, or anything else, or 'intention', even if a gun is pointed at them. So, why is that a fault of AI if it can't explain how itself is working? It is not a requirement for AGI.

Androider 3y ago

The equivalent for a human would be an reflexive response to a question, the kind you could immediately answer after being woken up at 3am in the morning. That type of answer has been deeply trained into the human networks and also requires no deep insight. But if a human is allowed time and internal reasoning iterations, so should the LLM when determining if it has deep insight. Right now we're simply observing input -> output of LLMs, the equivalent of snap answers from a human. But nothing says it couldn't instead be an input -> extensive internal dialogue, maybe even between multiple expert models for seconds, minutes or hours, that are not at all visible to the prompter -> final insightful answer. Maybe future LLMs will say, "let me get back to you on that".

I_Am_Nous 3y ago
This reminds me of The Last Question by Isaac Asimov. I also think if we stopped expecting all LLMs to have an immediate answer, it would be relatively easy to shim some kind of "conscience" to direct the output in different ways. Similar to the safeties already in place in LLMs, but instead of it just saying "NO DON'T SAY THAT" it can dialog internally to change what the output is until it reaches what it believes to be the agreed upon best answer.
- lifeisstillgood 3y ago
  It would have an emotional reaction to certain "thought constructs" and would be guided by that. Or we could just give them three laws
  - I_Am_Nous 3y ago
    With current LLMs the three laws might be tough to implement in a way that can't be prompt injected around. That's why I described the extra bits as a "conscience" which could enforce the three laws. Maybe the three laws are the internal conscience's context prompt while the main LLM is more able to think anything in general and then the output is tuned down by the conscience? Otherwise the laws will have to be implemented as weights or during training so the model explicitly knows the laws and would never even be capable of doing something against them.

two_in_one 3y ago

Just note that loop doesn't have to be visible from outside. It can be internal, with another driving thread asking right questions. Inner monologue. Then the summary is given back to user. This will give the model space for 'thinking' with internally generated text much large than the visible prompt + output. This way multi-step logic can be implemented.

chii 3y ago
> with another driving thread asking right questions. Inner monologue. spoilers warning: Is that basically the plot to westworld?
- two_in_one 3y ago
  Inner monologue idea was around for a while. That's interesting that it actually materializes. It was thought that AGI will be operating with some abstracts, logic rules, probability calculations, etc. Not thinking in plain English.

lixy 3y ago

Yep, prototype exactly that this past week. With a strong instruction spec prompt from the start, you can have an AI come up with a much better answer by making sure it knows it has time to answer the questions and how it should approach the problem in stages. The great part is with clear enough directions it also knows how to evaluate whether its done or not.

hskalin 3y ago
Can you share the kind of prompts you used?

twobitshifter 3y ago

There are plenty of models that use introspection and check answers, that’s the idea behind let’s think about it step by step.

tinco 3y ago
I feel the "let's think about it step by step" is a bit of a hack. To circumvent the fact that there's no external loop you use the fact that it gets re-run on every token so you can store a bit of state in the tokens that it's already generated. Or am I misunderstanding something about that technique?
- twobitshifter 3y ago
  You are right, it’s sometimes called zero shot chain of thought, but it’s a way of getting the type of thing you are describing to happen. The LLMs somehow process things in a perceived step by step to get a much improved answer. Whether the external loop or an llm imposed internal loop, does it matter? Are our own minds looping or just adding tokens?
  - tinco 3y ago
    Yeah that's true, and I do believe there's a good chance our minds are perpetually adding tokens. But our minds also have an efficient/effective way of dealing with the context cut off. We don't have (or we don't experience) a hard cut off of our memory context. Instead the tokens are increasingly lossily compressed as they age out of our memory, the lossiness amount being based both on time passed but also on some fancy value function. And that combined with a "system" (or trained/fine-tuned in) prompt that motivates the LLM to reason in a way that is conducive to working with that kind of memory would be a sort of single-shot AGI system. Where single-shot is lying a bit because it's just infinitely looping. I guess from that perspective it might make sense to test if such a thing is already happening in current LLM's and my dismissive attitude stems from the fact that I've played with them enough to know that they currently don't.
  - vidarh 3y ago

creer 3y ago

At the same time, if LLMs are based on all or enough human writings then don't they necessarily contain a theory of mind? A rather general, smoothed out and still neurotic one probably. But still just like an LLM can't be expected to have a specific knowledge of hydraulics, it also has read more hydraulics than even experts might be expected to. That's the entire issue about it, right? This issue of "is most of our mind basically just mixing and matching stuff we have seen, read, heard?" Do humans have some magical theory of mind that somehow stands ASIDE from all the "normal" learned stuff? Of course, yes, we do know one thing that's missing in LLMs, which is "loop and helpers" like you describe. Which I'm sure many people are currently hacking at - one way being for the LLM to talk to itself. But as for "a theory of mind", if enough writings served as input, then LLMs do have plenty of that. Another question is whether LLMs are raised to behave like humans (which might be where they most NEED some theory of mind). Of course not. The ones we know most about are only question answerers. The theory of mind they might have (that is not negated by the lack of loop and internal deliberation) may be overwhelmed by the pre- and post-processing: "no sex, no murder plots, talk to the human like they are 5, bla bla bla". And yet you can ask things like "Tell it like you are speaking to 5 year olds who want to have a fun time". Some theory of mind makes it through.

neuronerdgirl 3y ago
That's not what theory of mind is. Theory of mind is the understanding - not academic understanding, but functional, meaningful understanding that guides your behavior - that the other people (AIs?) around you have their own separate thoughts and feelings from your own. This is what leads to the stark behavioral differences as a child develops from ~3-6 years of age. Imagine a child is playing with their toys and puts one of them in a box out of the room, then later asks you to retrieve it for them. A 3 year old will assume that, because they know they put it in the box, that you know it is there as well. (Incidentally, this is where a lot of tantrums come from, because they can't understand why you aren't reading their mind and get frustrated.) A 6 year old will not make this assumption. While this is a simple example, the concept extrapolates to more complex issues as we develop cognitively and emotionally. Not only is theory of mind about recognizing that your thoughts and feelings are separate and distinct, but also being able to project your understanding into others to predict how they might be thinking and feeling in their own separate circumstance. This is the fundamental basis of empathy, or being able to predict and understand how others around you may be feeling given their unique current circumstances even if they are different from your own. LLMs have not demonstrated any of the above understanding. Again, I'm sure they could generate a definition for it, and on any given prompt they could probably even generate some generic-ish text that could pose as empathy, but so can a horoscope. But there is not the deep, continuous comprehension of the user as a separate thinking and feeling agent that is fundamental to the idea of theory of mind.