9 ms·

"LLMs are quickly going to be able to start explaining their own thought processes better than any human can explain their own." There is no "their" and there

by jmfldn 3y ago

"LLMs are quickly going to be able to start explaining their own thought processes better than any human can explain their own." There is no "their" and there is no "thought process" . There is something that produces text that appears to humans like there is something like thought going on (cf the Eliza Effect), but we must be wary of this anthropomorphising language. There is no self reflection, but if you ask an LLM program how "it" knows something it will produce some text.

nerpderp82 3y ago
What if you ask it to emit the reflexive output, then feed that reflexive output back into the LLM for the conscious answer? What if you ask it to synthesize multiple internal streams of thought, for an ensemble of interior monologues, then have all those argue with each other using logic and then present a high level answer from that panoply of answers?
- YawningAngel 3y ago
  What if you do? LLMs don't have reflexive output or internal streams of thought, they are simply (complex) processes that produce streams of tokens based on an inputted stream of tokens. They don't have a special response to tokens that indicate higher-level thinking to humans.
  - int_19h 3y ago
    If you direct the model output to itself and don't view it otherwise, how is it not an "internal stream of thought"?
  - TeMPOraL 3y ago
    LLMs seem to me to be the "internal streams of thought". I.e. it's not LLMs that are missing an internal process that humans have, but rather it's humans that have an entire process of conscious thinking built on top of something akin to LLM.
    - wilg 3y ago
      I agree completely and I think this is where a lot of people get tripped up. There's no reason to think an AGI needs to be an LLM alone, it might just be a key building block.
    - educaysean 3y ago
      Well put, and I agree. My belief is that if a typical person was drugged or otherwise induced to just blurt out their unfiltered thoughts out loud as it crossed their brain, the level of incohesion and false confidence on display would look a lot like an LLM hallucinating.
    - YawningAngel 3y ago
      That's possible I guess but is there positive evidence for that being the case?
      - TeMPOraL 3y ago
        The way I phrased it isn't exactly structured to admit any kind of evidence, so let me destructure it. My observation is that: - The argument that LLMs are missing introspection / inner voice is based on attempting to compare LLMs directly with human minds. - Human minds have conscious and unconscious parts; for many people, part of the boundary between unconscious and conscious mind manifests as the "inner voice" - the one that makes verbalized thoughts "appear" in their head (or rather perhaps become consciously observed). - Based entirely on my own experience in using GPT-3.5 and GPT-4, and my own introspection, I feel that GPT-4 bears a lot of resemblance to my inner voice in terms of functioning. - Therefore I propose that comparing LLMs directly to human minds is unproductive, and it's much more interesting/useful to compare them to the inner voice in human minds: the part of the boundary between unconscious and conscious that uses natural language for I/O.
  - nerpderp82 3y ago
    You seem to have high confidence in how LLMs work.
icholy 3y ago
Or maybe the human thought process isn't as sophisticated as we imagined.
- jmfldn 3y ago
  I'm not arguing for or against that. It's more the implications of sentience and selfhood implicit in the language many use around LLMs.
jpasmore 3y ago
As we don't know for sure what is happening 100% within a neural network, we can say we don't believe that they're thinking and we would still need to define the word thinking. Once LLM's can self-modify, the word "thinking" will be more accurate than it is today. And when Hinton says at MIT, "I find it very hard to believe that they don't have semantics when they consult problems like you know how I paint the rooms how I get all the rooms in my house to be painted white in two years time," I believe he's commenting on the ability of LLM's to think on some level.
- mcguire 3y ago
  In this case, I think we do if you will check out the paper (https://openaipublic.blob.core.windows.net/neuron-explainer/paper/index.html https://openaipublic.blob.core.windows.net/neuron-explainer/...). Their method is to 1. Show GPT-4 a GPT-produced text with the activation level of a specific neuron at the time it was producing that part of the text highlighted. They then ask GPT-4 for an explanation of what the neuron is doing. Text: "...mathematics is done _properly_, it...if it's done _right_. (Take ..." GPT produces "words and phrases related to performing actions correctly or properly". 2. Based on the explanation, get GPT to guess how strong the neuron activates on a new text. "Assuming that the neuron activates on words and phrases related to performing actions correctly or properly. GPT-4 guesses how strongly the neuron responds at each token: '...Boot. When done _correctly_, "Secure...'" 3. Compare those predictions to the actual activations of the neuron on the text to generate a score. So there is no introspection going on. They say, "We applied our method to all MLP neurons in GPT-2 XL [out of 1.5B?]. We found over 1,000 neurons with explanations that scored at least 0.8, meaning that according to GPT-4 they account for most of the neuron's top-activating behavior." But they also mention, "However, we found that both GPT-4-based and human contractor explanations still score poorly in absolute terms. When looking at neurons, we also found the typical neuron appeared quite polysemantic."
callesgg 3y ago
The text output of a llm is the thought process. In this context the main difference between humans and llms, is that llms can’t have internalized thoughts. There are of course other differences to, like the fact that humans have a wider gamut of input: visuals, sound, input from other bodily functions. And the fact that we have live training.
- creatonez 3y ago
  It's not clear whether or not LLMs have internal thoughts -- each token generation could absolutely have a ton of thought-like modelling in the hidden layers of the network. What is known is that these internal thoughts get erased each time a new token is generated. That is, it's starting from scratch from the contents of the text each time it generates a word. But you could postulate that similar prompt text leads to similar "thoughts" and/or navigation of the concept web, and therefore the thoughts are continuous in a sense.
  - callesgg 3y ago
    True, LLMs definetly has something that is "thought-like". But todays networks lacks the recursion(feedback where the output can go directly to the input) that is needed for the type of internalized thoughts that humans have. I guess this is one thing you are pointing at by mentioning the continuousnes of the internals of LLMs.
codehalo 3y ago
Pride comes before the fall, and the AI comes before humility.
- stevenhuang 3y ago
  Precisely. When someone states definitively what LLMs can or cannot do, that is when you know to immediately disregard them as the waffling of uninformed laymen lacking the necessary knowledge foundations (cognitive/neuroscience/philosophy) to even appreciate the uncertainty and finer points under discussion (all the open questions regarding human cognition etc). They don't know what they don't know and make unfounded assertions as result. Many would do to refrain from speaking so surely about matters they know nothing about, but that is the internet for you. https://en.wikipedia.org/wiki/Predictive_coding https://en.wikipedia.org/wiki/Predictive_coding
marshray 3y ago
> There is no self reflection, but if you ask an LLM program how "it" knows something it will produce some text. To be clear, you're saying that we should just dismiss out-of-hand any possibility that an LM AI might actually be able to explain its reasoning step-by-step? I find it kind of charming actually how so many humans are just so darn sure that they have their own special kind of cognition that could never be replicated. Not even with 175,000,000,000 calculations for every word generated.
- jmfldn 3y ago
  That's a strawman since I didn't argue anything about humans being special. I don't think there is anything necessarily inherently special about human intelligence, I'm just advocating for caution around the language we use to talk about current systems. All this talk of AGI and sentience and so on is premature and totally unfounded . It's pure sci fi, for now at least.
  - marshray 3y ago
    > I didn't argue anything about humans being special Above you said about AI LMs: > There is no "their" and there is no "thought process" So, unless you're claiming that humans lack a thought process as well, then you're arguing that humans are special. > All this talk of AGI and sentience and so on is premature and totally unfounded I don't see any mention of AGI or sentience in this thread? Also, I don't think anyone could read this transcript with GPT-4 and still claim that it's incapable of a significant degree of self-reflection and metacognition: https://news.ycombinator.com/item?id=35880148 https://news.ycombinator.com/item?id=35880148
    - jmfldn 3y ago
      I reject the words 'self' and 'cognition' in your comment. This is exactly what I'm talking about. A facsimile of them, maybe...
emporas 3y ago
Very true. In my opinion, in case there is a way to extract "Semantic Clouds of Words", i.e given a particular topic, navigate semantic clouds word by word, find some close neighbours of that word, jump to a neighbour of that word and so on, then LLMs might not seem that big of a deal. I think LLMs are "Semantic Clouds of Words" + grammar and syntax generator. Someone could just discard the grammar and syntax generator, just use the semantic cloud and create the grammar and syntax by himself. For example, in writing a legal document, a slightly educated person on the subject, could just use the relevant words put into an empty paper, fill in the blanks of syntax and grammar, alongside with the human reasoning which is far superior than any machine reasoning, till today at least. The process of editing the GPT* generated documents to fix reasoning is not a negligible task anyway. Sam Altman mentioned that: "the machine has some kind of reasoning", not a human reasoning ability by any means. My point is, that LLMs are two programs fused into one, "word clouds" and "syntax and grammar", sprinkled with some kind of poor reasoning. Their word clouding ability, is so unbelievable stronger than any human it fills me with awe every time i use it. Everything else is, just whatever!
- TeMPOraL 3y ago
  I think they're much more than that. Or rather, if they're "Semantic Cloud of Words", they're still a hundred thousand dimensional clouds of words, and in those hundred thousand dimensions, any relationship you can think of, no matter how obscure, ends up being reflected as proximity along some subset of dimensions. Looking at it this way, I honestly wouldn't be surprised if that's exactly how "System 1" (to borrow a term from Kahneman) in our brains works. What I'm saying is: > In my opinion, in case there is a way to extract "Semantic Clouds of Words", i.e given a particular topic, navigate semantic clouds word by word, find some close neighbours of that word, jump to a neighbour of that word and so on, then LLMs might not seem that big of a deal. It may be much more of a deal than we'd naively think - it seems to me that a lot of what we'd consider "thinking" and "reasoning" can be effectively implemented as proximity search in a high-dimensional enough vector space. In that case, such extracted "Semantic Cloud of Words" may turn out to represent the very structure of reasoning as humans do it - structure implicitly encoded in all the text that was used as training data for the LLMs.
circuit10 3y ago
Clearly there is some process going on to decide what word to pick. Why can’t we call that thinking?
greenhearth 3y ago
This is it. This comprehension of the chats is symptom of something like linguistic pareidolia. It's an enforced face that is composed of some probabilistic incidents and wistfulness.

by jmfldn 3y ago

nerpderp82 3y ago

What if you ask it to emit the reflexive output, then feed that reflexive output back into the LLM for the conscious answer? What if you ask it to synthesize multiple internal streams of thought, for an ensemble of interior monologues, then have all those argue with each other using logic and then present a high level answer from that panoply of answers?

YawningAngel 3y ago
What if you do? LLMs don't have reflexive output or internal streams of thought, they are simply (complex) processes that produce streams of tokens based on an inputted stream of tokens. They don't have a special response to tokens that indicate higher-level thinking to humans.
- int_19h 3y ago
  If you direct the model output to itself and don't view it otherwise, how is it not an "internal stream of thought"?
- TeMPOraL 3y ago
  LLMs seem to me to be the "internal streams of thought". I.e. it's not LLMs that are missing an internal process that humans have, but rather it's humans that have an entire process of conscious thinking built on top of something akin to LLM.
  - wilg 3y ago
    I agree completely and I think this is where a lot of people get tripped up. There's no reason to think an AGI needs to be an LLM alone, it might just be a key building block.
  - educaysean 3y ago
    Well put, and I agree. My belief is that if a typical person was drugged or otherwise induced to just blurt out their unfiltered thoughts out loud as it crossed their brain, the level of incohesion and false confidence on display would look a lot like an LLM hallucinating.
  - YawningAngel 3y ago
    That's possible I guess but is there positive evidence for that being the case?
    - TeMPOraL 3y ago
      The way I phrased it isn't exactly structured to admit any kind of evidence, so let me destructure it. My observation is that: - The argument that LLMs are missing introspection / inner voice is based on attempting to compare LLMs directly with human minds. - Human minds have conscious and unconscious parts; for many people, part of the boundary between unconscious and conscious mind manifests as the "inner voice" - the one that makes verbalized thoughts "appear" in their head (or rather perhaps become consciously observed). - Based entirely on my own experience in using GPT-3.5 and GPT-4, and my own introspection, I feel that GPT-4 bears a lot of resemblance to my inner voice in terms of functioning. - Therefore I propose that comparing LLMs directly to human minds is unproductive, and it's much more interesting/useful to compare them to the inner voice in human minds: the part of the boundary between unconscious and conscious that uses natural language for I/O.
- nerpderp82 3y ago
  You seem to have high confidence in how LLMs work.

icholy 3y ago

Or maybe the human thought process isn't as sophisticated as we imagined.

jmfldn 3y ago
I'm not arguing for or against that. It's more the implications of sentience and selfhood implicit in the language many use around LLMs.

jpasmore 3y ago

As we don't know for sure what is happening 100% within a neural network, we can say we don't believe that they're thinking and we would still need to define the word thinking. Once LLM's can self-modify, the word "thinking" will be more accurate than it is today. And when Hinton says at MIT, "I find it very hard to believe that they don't have semantics when they consult problems like you know how I paint the rooms how I get all the rooms in my house to be painted white in two years time," I believe he's commenting on the ability of LLM's to think on some level.

mcguire 3y ago
In this case, I think we do if you will check out the paper (https://openaipublic.blob.core.windows.net/neuron-explainer/paper/index.html https://openaipublic.blob.core.windows.net/neuron-explainer/...). Their method is to 1. Show GPT-4 a GPT-produced text with the activation level of a specific neuron at the time it was producing that part of the text highlighted. They then ask GPT-4 for an explanation of what the neuron is doing. Text: "...mathematics is done _properly_, it...if it's done _right_. (Take ..." GPT produces "words and phrases related to performing actions correctly or properly". 2. Based on the explanation, get GPT to guess how strong the neuron activates on a new text. "Assuming that the neuron activates on words and phrases related to performing actions correctly or properly. GPT-4 guesses how strongly the neuron responds at each token: '...Boot. When done _correctly_, "Secure...'" 3. Compare those predictions to the actual activations of the neuron on the text to generate a score. So there is no introspection going on. They say, "We applied our method to all MLP neurons in GPT-2 XL [out of 1.5B?]. We found over 1,000 neurons with explanations that scored at least 0.8, meaning that according to GPT-4 they account for most of the neuron's top-activating behavior." But they also mention, "However, we found that both GPT-4-based and human contractor explanations still score poorly in absolute terms. When looking at neurons, we also found the typical neuron appeared quite polysemantic."

callesgg 3y ago

The text output of a llm is the thought process. In this context the main difference between humans and llms, is that llms can’t have internalized thoughts. There are of course other differences to, like the fact that humans have a wider gamut of input: visuals, sound, input from other bodily functions. And the fact that we have live training.

creatonez 3y ago
It's not clear whether or not LLMs have internal thoughts -- each token generation could absolutely have a ton of thought-like modelling in the hidden layers of the network. What is known is that these internal thoughts get erased each time a new token is generated. That is, it's starting from scratch from the contents of the text each time it generates a word. But you could postulate that similar prompt text leads to similar "thoughts" and/or navigation of the concept web, and therefore the thoughts are continuous in a sense.
- callesgg 3y ago
  True, LLMs definetly has something that is "thought-like". But todays networks lacks the recursion(feedback where the output can go directly to the input) that is needed for the type of internalized thoughts that humans have. I guess this is one thing you are pointing at by mentioning the continuousnes of the internals of LLMs.

codehalo 3y ago

Pride comes before the fall, and the AI comes before humility.

stevenhuang 3y ago
Precisely. When someone states definitively what LLMs can or cannot do, that is when you know to immediately disregard them as the waffling of uninformed laymen lacking the necessary knowledge foundations (cognitive/neuroscience/philosophy) to even appreciate the uncertainty and finer points under discussion (all the open questions regarding human cognition etc). They don't know what they don't know and make unfounded assertions as result. Many would do to refrain from speaking so surely about matters they know nothing about, but that is the internet for you. https://en.wikipedia.org/wiki/Predictive_coding https://en.wikipedia.org/wiki/Predictive_coding

marshray 3y ago

> There is no self reflection, but if you ask an LLM program how "it" knows something it will produce some text. To be clear, you're saying that we should just dismiss out-of-hand any possibility that an LM AI might actually be able to explain its reasoning step-by-step? I find it kind of charming actually how so many humans are just so darn sure that they have their own special kind of cognition that could never be replicated. Not even with 175,000,000,000 calculations for every word generated.

jmfldn 3y ago
That's a strawman since I didn't argue anything about humans being special. I don't think there is anything necessarily inherently special about human intelligence, I'm just advocating for caution around the language we use to talk about current systems. All this talk of AGI and sentience and so on is premature and totally unfounded . It's pure sci fi, for now at least.
- marshray 3y ago
  > I didn't argue anything about humans being special Above you said about AI LMs: > There is no "their" and there is no "thought process" So, unless you're claiming that humans lack a thought process as well, then you're arguing that humans are special. > All this talk of AGI and sentience and so on is premature and totally unfounded I don't see any mention of AGI or sentience in this thread? Also, I don't think anyone could read this transcript with GPT-4 and still claim that it's incapable of a significant degree of self-reflection and metacognition: https://news.ycombinator.com/item?id=35880148 https://news.ycombinator.com/item?id=35880148
  - jmfldn 3y ago
    I reject the words 'self' and 'cognition' in your comment. This is exactly what I'm talking about. A facsimile of them, maybe...

emporas 3y ago

Very true. In my opinion, in case there is a way to extract "Semantic Clouds of Words", i.e given a particular topic, navigate semantic clouds word by word, find some close neighbours of that word, jump to a neighbour of that word and so on, then LLMs might not seem that big of a deal. I think LLMs are "Semantic Clouds of Words" + grammar and syntax generator. Someone could just discard the grammar and syntax generator, just use the semantic cloud and create the grammar and syntax by himself. For example, in writing a legal document, a slightly educated person on the subject, could just use the relevant words put into an empty paper, fill in the blanks of syntax and grammar, alongside with the human reasoning which is far superior than any machine reasoning, till today at least. The process of editing the GPT* generated documents to fix reasoning is not a negligible task anyway. Sam Altman mentioned that: "the machine has some kind of reasoning", not a human reasoning ability by any means. My point is, that LLMs are two programs fused into one, "word clouds" and "syntax and grammar", sprinkled with some kind of poor reasoning. Their word clouding ability, is so unbelievable stronger than any human it fills me with awe every time i use it. Everything else is, just whatever!

TeMPOraL 3y ago
I think they're much more than that. Or rather, if they're "Semantic Cloud of Words", they're still a hundred thousand dimensional clouds of words, and in those hundred thousand dimensions, any relationship you can think of, no matter how obscure, ends up being reflected as proximity along some subset of dimensions. Looking at it this way, I honestly wouldn't be surprised if that's exactly how "System 1" (to borrow a term from Kahneman) in our brains works. What I'm saying is: > In my opinion, in case there is a way to extract "Semantic Clouds of Words", i.e given a particular topic, navigate semantic clouds word by word, find some close neighbours of that word, jump to a neighbour of that word and so on, then LLMs might not seem that big of a deal. It may be much more of a deal than we'd naively think - it seems to me that a lot of what we'd consider "thinking" and "reasoning" can be effectively implemented as proximity search in a high-dimensional enough vector space. In that case, such extracted "Semantic Cloud of Words" may turn out to represent the very structure of reasoning as humans do it - structure implicitly encoded in all the text that was used as training data for the LLMs.

circuit10 3y ago

Clearly there is some process going on to decide what word to pick. Why can’t we call that thinking?

greenhearth 3y ago

This is it. This comprehension of the chats is symptom of something like linguistic pareidolia. It's an enforced face that is composed of some probabilistic incidents and wistfulness.