8 ms·

I worry that the "stochastic parrot" was premature, an idea sown early in development that will now carry along through any advances made. Basically there is t

I worry that the "stochastic parrot" was premature, an idea sown early in development that will now carry along through any advances made. Basically there is this innate idea that if the basic building blocks are simple systems with deterministic behavior, then the greater system can never be more than that. I've seen this is spades within the AI community, "It's just matrix multiplication! It's not capable of thinking or feeling!" Which to me always felt more like a hopeful statement rather than a factual one. These guys have no idea what consciousness is (nobody does) nor have any reference point for what exactly is "thinking" or "feeling". They can't prove I'm not a stochastic parrot anymore than they can prove whatever cutting edge LLM isn't. So while yes, present LLMs likely are just stochastic parrots, the same technology scaled might bring us a model that actually is "something that is something to be like", and we'll have everyone treating it with reckless carelessness because "its just a stochastic parrot".

cjalmeida 3y ago
Indeed. It's like saying life is just long strands of DNA composed of only four simple molecules. Or economy is just people trading their surpluses. Emergent behavior... emerges. It's hard to predict or explain from constituents. Scale changes everything.
- jschveibinz 3y ago
  [flagged]
ac2u 3y ago
Agreed. I think the stochastic parrot concept is useful to ground our expectations on LLMs for now, but it could outlive it's usefulness if there ends up being multiple jumps in sophistication similar to that of GPT-2 to 4 in the next 10 years. If that happens, then stochastic parrot as an argument as to why a machine isn't thinking can be made pretty useless if One chooses to drag the argument further into philosophy.
- Ireallyapart 3y ago
  [dead]
nathan_compton 3y ago
"These guys have no idea what consciousness is (nobody does)" Where do people get off saying no one has any idea what consciousness is? I agree that there is a significant sliver of a philosophical problem which remains stubborn (how precisely does physical activity produce qualia), but neuroscience knows quite a bit about what physical processes underlie our behavior from the behavior of individual neurons to the activity of the entire brain. I object to the wholesale dismissal of neuroscience because thinking about the brain relative to LLMs is genuinely informative about what sorts of things you could expect to be going on in an LLM. And, to my mind, a real appraisal of the differences between brains and LLMs makes the case pretty strongly that LLMs experience nothing and are, furthermore, fairly well characterized as stochastic parrots. "They can't prove I'm not a stochastic parrot anymore than they can prove whatever cutting edge LLM isn't." Prove is a very strong word, but I think its actually quite possible to demonstrate via scientific observation that you differ in many, significant, and relevant to the question of "being a stochastic parrot", ways, from LLMs. It astounds me that people routinely suggest that human brains and LLMs are somehow indistinguishable.
- ARandomerDude 3y ago
  Likewise, the nobody knows what consciousness is mindset is smuggling in the idea that in order to know something at all, you must know it comprehensively. I know exactly what consciousness is based on my experience with it, even though I could not possibly give a comprehensive account of everything consciousness entails. By analogy, I've been married for just shy of 20 years. I know my wife very well. I certainly do not know everything there is to know about her, but I do know her.
  - eimrine 3y ago
    > I know exactly what consciousness is based on my experience with it, even though I could not possibly give a comprehensive account of everything consciousness entails. What is the price of that kind of knowledge? You don't even know where is the border between your knowledge and your absence of knowledge. How much can you tell about consciousness without stepping on "absence of knowledge" field? Pretty nothing, isn't it? > By analogy, I've been married for just shy of 20 years. I know my wife very well. I certainly do not know everything there is to know about her, but I do know her. I have a better example. I speak English for 20 years if to start counting from my first English lesson when I learned my first English word. You can find plenty of silly mistakes in my comments. But at least I know what I can express or understand and what I can not.
PartiallyTyped 3y ago
Frankly speaking, I think this is more on us for thinking that we are special or that living forms are special. This is just our ego talking. Under the view that we are all just complexity arising out of an unfathomably large universe, then we can accept that LLMs are just that, like us, but weaker, and that is fine. They will improve, we can leverage them, we can live with them. It's almost as if we have created a new species that exists only abstractly; and arises out of silicon and electrons.
FabHK 3y ago
The notion of the stochastic parrot is that the system produces plausible sounding words, but does not understand, does not exhibit intelligence. Consciousness is orthogonal to the discussion (the term doesn't appear in the linked article).
- Ireallyapart 3y ago
  [dead]
ChatGTP 3y ago
I think you know what consciousness is ? It’s basically your life ?
me_me_me 3y ago
I have noticed people look at animal or machines and try to figure out if they are as complex as humans in order to figure out if they are conscious or not. But almost never you see logic applied other way around, maybe we are just bunch of simple mechanisms convinced we are something way more complex. There are quite few hints that the second option is the actual reality.
abeppu 3y ago
I disagree. The criticism is _not_ that basic building blocks cannot be combined to produce something richer. The issue is the "without any reference to meaning" part of the quoted definition from Bender in that article. Models which are _only_ trained on text do not have a grounding to relate linguistic forms to anything else. When you know what an apple is, it's in part because you've seen and touched and tasted and eaten one. The model only knows how people talk about apples, and which texts are plausible, but not which ones are true. But we're already getting past this with multi-modal models! Some really great work is being done which ties language processing with visual perception and in some cases robot action planning. A model can know how we talk about apples, can see where an apple is in a scene, can navigate to and retrieve an apple, etc. This lets us get at truth ("Is the claim 'the apple is on the book' true of this scene?") in a way which text-only models fundamentally cannot have. The point is, the way you get past the "stochastic parrot" phase requires qualitative structural changes to incorporate different kinds of information -- not just scaling up text-only models. > They can't prove I'm not a stochastic parrot anymore than they can prove whatever cutting edge LLM isn't. I can't prove you're not a stochastic parrot by only talking to you via text. But in person I can toss you an object and you can catch it which shows that you understand how to interact with a dynamic 3D environment. I can ask you a question about something in our shared environment, and you can give an answer which is _true_, rather than which is a plausible-sounding sentence. This is the difference between knowing what English texts or English conversations look like, versus knowing what states of the world are referred to by statements.
- Ireallyapart 3y ago
  [dead]
- NoGravitas 3y ago
  > I disagree. The criticism is _not_ that basic building blocks cannot be combined to produce something richer. The issue is the "without any reference to meaning" part of the quoted definition from Bender in that article. Right. People think the stochastic parrot description is about the Chinese Room thought experiment, but it's not. It's about the Thai Library thought experiment: https://medium.com/@emilymenonbender/thought-experiment-in-the-national-library-of-thailand-f2bf761a8a83 https://medium.com/@emilymenonbender/thought-experiment-in-t...
sfpotter 3y ago
Stochastic parrots have nothing directly to do with consciousness. You might consider reading the paper or at least the definition on Wikipedia more carefully. As far as your statement regarding consciousness goes, it's glib to say that no one has any reference point for what consciousness, thinking, or feeling are. We all have our own lived experience to draw on for intuition and guidance to inform our thinking, which is invaluable. We can relate our qualitative perception of these phenomena to other things in the world, where a reasonable person can form the hypothesis that "matrix multiplication" is unlikely to be conscious, to think, or to feel by dint of it being an abstract mathematical concept, since there is no precedence for an abstract mathematical concept exhibiting any of these qualities. Indeed, the only things in our lived experience which can plausibly be said to be conscious, to think, or to feel are biological organisms, of which a computer is not.
gisely 3y ago
It's weird seeing comments like this that argue simultaneously: 1) LLMs aren't stochastic parrots anymore! 2) You can't prove humans aren't stochastic parrots! It pretty clear the whole point is minimize the difference between us and AI, but it does feel like you are undermining you argument by trying to work it from both sides. It reminds me some accused of crime who say both "I didn't do it!" and "If did it, it wasn't wrong!". Humans aren't stochastic parrots. You can't "prove" this because it's not mathematical fact, but there is plenty evidence from study how to brain works to show this. Hell, it's even readily apparent from introspection if you'd bother to check. LLMs on the hand basically are stochastic parrots because they just autoregressive token predictors. They might become less so due to architectural changes made by the companies working on them, but it isn't going to just creep up on us like some goddamn emergence boogeyman.
- Workaccount2 3y ago
  Believe it or not, actual deep introspection (meditation, mindfulness) will make you realize you are more like a stochastic parrot, not less like one.
fnordpiglet 3y ago
The future is made by those that look past what is and see what might be and fail half way to achieving it. The rest are mired in their attachments and will never escape a murky prison of what today appears to be.
pseudotrash 3y ago
> These guys have no idea what consciousness is (nobody does) ... On that, there is a great "In Our Time" episode: https://open.spotify.com/episode/5oln4RwbhsKwjlZuxPuYYB?si=5mCXpqTIQ-qMzdgvmzZCgQ https://open.spotify.com/episode/5oln4RwbhsKwjlZuxPuYYB?si=5... Unless it is able to feel pain it remains a stochastic parrot and I wouldn't call it conscious or alive in any philosophical sense nor can one say it is capable of "feeling".
shimfish 3y ago
Roger Penrose wrote about this decades ago, arguing that computers cannot contain consciousness and neither can anything in currently understood physics. https://www.amazon.com/Emperors-New-Mind-Concerning-Computers-dp-0198784929/dp/0198784929/ https://www.amazon.com/Emperors-New-Mind-Concerning-Computer...
- dekhn 3y ago
  he never came up with any useful experiments that would demonstrate support for his position. Nor did he make a convincing theoretical argument.
cjbprime 3y ago
I think that some LLMs (mainly just GPT-4) should be considered as refutations to the Stochastic Parrot idea, which was published in March 2021 and claims no LLM can have "any model of the world". It was a reasonable (though perhaps overconfident) paper for authors who had only used GPT-3 to publish, but there is now ample evidence of world modeling, including published academic evidence, for GPT-4. I think the following claim from the paper is also deeply incorrect and confused: > an LM is a system for haphazardly stitching together sequences of linguistic forms it has observed in its vast training data, according to probabilistic information about how they combine, but without any reference to meaning: a stochastic parrot. Next token prediction is a function of tokens/words, but that doesn't preclude that prediction depending on meaning, and the best predictions obviously do depend on meaning. It is not clear, at least to me, that next token prediction leads to any kind of upper bound on intelligence. It is always possible to incorporate more of the descriptions of the world obtained through the training data into your predictions to improve them. But I think you've missed an important distinction. The stochastic parrot claim can be false, not because LLMs can or will ever feel or be conscious, but because they can (today) reason and solve novel problems (the capability is there, but it is unreliable). LLMs are not probabilistically regurgitating their training sets; they're applying the learning they took away from those training sets. I think GPT-4 can reason today, but I don't think it can feel or is conscious, and I don't expect it to be capable of those things in its current architecture.
Joeri 3y ago
In a sense LLM's are a Rorschach test for people's beliefs about consciousness. If you believe consciousness to be an emergent phenomenon derived from simple deterministic biological processes, then it is not a big leap to believe LLM's to be on a roadmap to consciousness. If you instead believe consciousness is a supernatural phenomenon, then you will discard the very idea of a computer having a consciousness, because a machine could never be imbued with one by mere algorithms. Tell me what your view is on the ability of LLM's to become AGI, and I'll tell you whether you believe in an immortal soul.
- piloto_ciego 3y ago
  Oh, I definitely don’t think this is correct because I at least for me the contra positive is not true. I believe machines that could be imbued with consciousness and I do not rule out that there could be supernatural elements to consciousness. Or at least things that fall outside the realm of strictly testable science.
itairal 3y ago
Totally agree. There is also a problem to me that "stochastic parrot" is too clever. It is too good of a name and evokes such a strong mental image. It is a great name for branding purposes but because of that it is a terrible name if we are actually trying to discover truth. It can't but help to become a blunt, unthinking, intellectual weapon and rhetorical device.
pizza234 3y ago
> Basically there is this innate idea that if the basic building blocks are simple systems with deterministic behavior, then the greater system can never be more than that. I've seen this is spades within the AI community I'm very surprised by this, because in essence, it's a flat-out denial of the emergence concept, no different from denying that atoms can ultimately lead to biological entities.

cjalmeida 3y ago

Indeed. It's like saying life is just long strands of DNA composed of only four simple molecules. Or economy is just people trading their surpluses. Emergent behavior... emerges. It's hard to predict or explain from constituents. Scale changes everything.

jschveibinz 3y ago
[flagged]

ac2u 3y ago

Agreed. I think the stochastic parrot concept is useful to ground our expectations on LLMs for now, but it could outlive it's usefulness if there ends up being multiple jumps in sophistication similar to that of GPT-2 to 4 in the next 10 years. If that happens, then stochastic parrot as an argument as to why a machine isn't thinking can be made pretty useless if One chooses to drag the argument further into philosophy.

Ireallyapart 3y ago
[dead]

nathan_compton 3y ago

"These guys have no idea what consciousness is (nobody does)" Where do people get off saying no one has any idea what consciousness is? I agree that there is a significant sliver of a philosophical problem which remains stubborn (how precisely does physical activity produce qualia), but neuroscience knows quite a bit about what physical processes underlie our behavior from the behavior of individual neurons to the activity of the entire brain. I object to the wholesale dismissal of neuroscience because thinking about the brain relative to LLMs is genuinely informative about what sorts of things you could expect to be going on in an LLM. And, to my mind, a real appraisal of the differences between brains and LLMs makes the case pretty strongly that LLMs experience nothing and are, furthermore, fairly well characterized as stochastic parrots. "They can't prove I'm not a stochastic parrot anymore than they can prove whatever cutting edge LLM isn't." Prove is a very strong word, but I think its actually quite possible to demonstrate via scientific observation that you differ in many, significant, and relevant to the question of "being a stochastic parrot", ways, from LLMs. It astounds me that people routinely suggest that human brains and LLMs are somehow indistinguishable.

ARandomerDude 3y ago
Likewise, the nobody knows what consciousness is mindset is smuggling in the idea that in order to know something at all, you must know it comprehensively. I know exactly what consciousness is based on my experience with it, even though I could not possibly give a comprehensive account of everything consciousness entails. By analogy, I've been married for just shy of 20 years. I know my wife very well. I certainly do not know everything there is to know about her, but I do know her.
- eimrine 3y ago
  > I know exactly what consciousness is based on my experience with it, even though I could not possibly give a comprehensive account of everything consciousness entails. What is the price of that kind of knowledge? You don't even know where is the border between your knowledge and your absence of knowledge. How much can you tell about consciousness without stepping on "absence of knowledge" field? Pretty nothing, isn't it? > By analogy, I've been married for just shy of 20 years. I know my wife very well. I certainly do not know everything there is to know about her, but I do know her. I have a better example. I speak English for 20 years if to start counting from my first English lesson when I learned my first English word. You can find plenty of silly mistakes in my comments. But at least I know what I can express or understand and what I can not.

PartiallyTyped 3y ago

Frankly speaking, I think this is more on us for thinking that we are special or that living forms are special. This is just our ego talking. Under the view that we are all just complexity arising out of an unfathomably large universe, then we can accept that LLMs are just that, like us, but weaker, and that is fine. They will improve, we can leverage them, we can live with them. It's almost as if we have created a new species that exists only abstractly; and arises out of silicon and electrons.

FabHK 3y ago

The notion of the stochastic parrot is that the system produces plausible sounding words, but does not understand, does not exhibit intelligence. Consciousness is orthogonal to the discussion (the term doesn't appear in the linked article).

Ireallyapart 3y ago
[dead]

ChatGTP 3y ago

I think you know what consciousness is ? It’s basically your life ?

me_me_me 3y ago

I have noticed people look at animal or machines and try to figure out if they are as complex as humans in order to figure out if they are conscious or not. But almost never you see logic applied other way around, maybe we are just bunch of simple mechanisms convinced we are something way more complex. There are quite few hints that the second option is the actual reality.

abeppu 3y ago

I disagree. The criticism is _not_ that basic building blocks cannot be combined to produce something richer. The issue is the "without any reference to meaning" part of the quoted definition from Bender in that article. Models which are _only_ trained on text do not have a grounding to relate linguistic forms to anything else. When you know what an apple is, it's in part because you've seen and touched and tasted and eaten one. The model only knows how people talk about apples, and which texts are plausible, but not which ones are true. But we're already getting past this with multi-modal models! Some really great work is being done which ties language processing with visual perception and in some cases robot action planning. A model can know how we talk about apples, can see where an apple is in a scene, can navigate to and retrieve an apple, etc. This lets us get at truth ("Is the claim 'the apple is on the book' true of this scene?") in a way which text-only models fundamentally cannot have. The point is, the way you get past the "stochastic parrot" phase requires qualitative structural changes to incorporate different kinds of information -- not just scaling up text-only models. > They can't prove I'm not a stochastic parrot anymore than they can prove whatever cutting edge LLM isn't. I can't prove you're not a stochastic parrot by only talking to you via text. But in person I can toss you an object and you can catch it which shows that you understand how to interact with a dynamic 3D environment. I can ask you a question about something in our shared environment, and you can give an answer which is _true_, rather than which is a plausible-sounding sentence. This is the difference between knowing what English texts or English conversations look like, versus knowing what states of the world are referred to by statements.

Ireallyapart 3y ago
[dead]
NoGravitas 3y ago
> I disagree. The criticism is _not_ that basic building blocks cannot be combined to produce something richer. The issue is the "without any reference to meaning" part of the quoted definition from Bender in that article. Right. People think the stochastic parrot description is about the Chinese Room thought experiment, but it's not. It's about the Thai Library thought experiment: https://medium.com/@emilymenonbender/thought-experiment-in-the-national-library-of-thailand-f2bf761a8a83 https://medium.com/@emilymenonbender/thought-experiment-in-t...

sfpotter 3y ago

Stochastic parrots have nothing directly to do with consciousness. You might consider reading the paper or at least the definition on Wikipedia more carefully. As far as your statement regarding consciousness goes, it's glib to say that no one has any reference point for what consciousness, thinking, or feeling are. We all have our own lived experience to draw on for intuition and guidance to inform our thinking, which is invaluable. We can relate our qualitative perception of these phenomena to other things in the world, where a reasonable person can form the hypothesis that "matrix multiplication" is unlikely to be conscious, to think, or to feel by dint of it being an abstract mathematical concept, since there is no precedence for an abstract mathematical concept exhibiting any of these qualities. Indeed, the only things in our lived experience which can plausibly be said to be conscious, to think, or to feel are biological organisms, of which a computer is not.

gisely 3y ago

It's weird seeing comments like this that argue simultaneously: 1) LLMs aren't stochastic parrots anymore! 2) You can't prove humans aren't stochastic parrots! It pretty clear the whole point is minimize the difference between us and AI, but it does feel like you are undermining you argument by trying to work it from both sides. It reminds me some accused of crime who say both "I didn't do it!" and "If did it, it wasn't wrong!". Humans aren't stochastic parrots. You can't "prove" this because it's not mathematical fact, but there is plenty evidence from study how to brain works to show this. Hell, it's even readily apparent from introspection if you'd bother to check. LLMs on the hand basically are stochastic parrots because they just autoregressive token predictors. They might become less so due to architectural changes made by the companies working on them, but it isn't going to just creep up on us like some goddamn emergence boogeyman.

Workaccount2 3y ago
Believe it or not, actual deep introspection (meditation, mindfulness) will make you realize you are more like a stochastic parrot, not less like one.

fnordpiglet 3y ago

The future is made by those that look past what is and see what might be and fail half way to achieving it. The rest are mired in their attachments and will never escape a murky prison of what today appears to be.

pseudotrash 3y ago

> These guys have no idea what consciousness is (nobody does) ... On that, there is a great "In Our Time" episode: https://open.spotify.com/episode/5oln4RwbhsKwjlZuxPuYYB?si=5mCXpqTIQ-qMzdgvmzZCgQ https://open.spotify.com/episode/5oln4RwbhsKwjlZuxPuYYB?si=5... Unless it is able to feel pain it remains a stochastic parrot and I wouldn't call it conscious or alive in any philosophical sense nor can one say it is capable of "feeling".

shimfish 3y ago

Roger Penrose wrote about this decades ago, arguing that computers cannot contain consciousness and neither can anything in currently understood physics. https://www.amazon.com/Emperors-New-Mind-Concerning-Computers-dp-0198784929/dp/0198784929/ https://www.amazon.com/Emperors-New-Mind-Concerning-Computer...

dekhn 3y ago
he never came up with any useful experiments that would demonstrate support for his position. Nor did he make a convincing theoretical argument.

cjbprime 3y ago

I think that some LLMs (mainly just GPT-4) should be considered as refutations to the Stochastic Parrot idea, which was published in March 2021 and claims no LLM can have "any model of the world". It was a reasonable (though perhaps overconfident) paper for authors who had only used GPT-3 to publish, but there is now ample evidence of world modeling, including published academic evidence, for GPT-4. I think the following claim from the paper is also deeply incorrect and confused: > an LM is a system for haphazardly stitching together sequences of linguistic forms it has observed in its vast training data, according to probabilistic information about how they combine, but without any reference to meaning: a stochastic parrot. Next token prediction is a function of tokens/words, but that doesn't preclude that prediction depending on meaning, and the best predictions obviously do depend on meaning. It is not clear, at least to me, that next token prediction leads to any kind of upper bound on intelligence. It is always possible to incorporate more of the descriptions of the world obtained through the training data into your predictions to improve them. But I think you've missed an important distinction. The stochastic parrot claim can be false, not because LLMs can or will ever feel or be conscious, but because they can (today) reason and solve novel problems (the capability is there, but it is unreliable). LLMs are not probabilistically regurgitating their training sets; they're applying the learning they took away from those training sets. I think GPT-4 can reason today, but I don't think it can feel or is conscious, and I don't expect it to be capable of those things in its current architecture.

Joeri 3y ago

In a sense LLM's are a Rorschach test for people's beliefs about consciousness. If you believe consciousness to be an emergent phenomenon derived from simple deterministic biological processes, then it is not a big leap to believe LLM's to be on a roadmap to consciousness. If you instead believe consciousness is a supernatural phenomenon, then you will discard the very idea of a computer having a consciousness, because a machine could never be imbued with one by mere algorithms. Tell me what your view is on the ability of LLM's to become AGI, and I'll tell you whether you believe in an immortal soul.

piloto_ciego 3y ago
Oh, I definitely don’t think this is correct because I at least for me the contra positive is not true. I believe machines that could be imbued with consciousness and I do not rule out that there could be supernatural elements to consciousness. Or at least things that fall outside the realm of strictly testable science.

itairal 3y ago

Totally agree. There is also a problem to me that "stochastic parrot" is too clever. It is too good of a name and evokes such a strong mental image. It is a great name for branding purposes but because of that it is a terrible name if we are actually trying to discover truth. It can't but help to become a blunt, unthinking, intellectual weapon and rhetorical device.

pizza234 3y ago

> Basically there is this innate idea that if the basic building blocks are simple systems with deterministic behavior, then the greater system can never be more than that. I've seen this is spades within the AI community I'm very surprised by this, because in essence, it's a flat-out denial of the emergence concept, no different from denying that atoms can ultimately lead to biological entities.