6 ms·

The more I think about it the more I'm convinced I am basically just predicting/saying my next word whenever I speak.

morkalork 3y ago
Why not? Your brain is already making predictions about what you expect to see and hear as a part of your perception of reality anyways.
- quad_eye_oh 3y ago
  Right, but LLMs suggest that learning how to predict and only training on predictions is sufficient to learn anything and to have emergent generative abilities. What if learning to predict the upcoming input is all that's all that is needed for general human intelligence? What if it is all that any animals do?
  - 20after4 3y ago
    This right here. I actually am strongly starting to believe that this is indeed what's going on. I read the book Kingdom of Speech a few years ago and that also left me with the perspective that perhaps language has a lot more to do with how we think and perceive the world than most people like to admit. The book has been heavily criticized but I believe it made an interesting point about language.
    - moffkalast 3y ago
      I mean hasn't it been proven on multiple occasions that humans who never learn a language during their life also don't develop higher cognitive skills and stay on the level of a child? Clearly language plays a notable part in reasoning, and it's frankly no surprise that we're seeing this behaviour from language models. It's also quite interesting how foundation models and fine tuning appear analogous to being born with a brain that already has six million years' worth of weights in it (trained implicitly through random changes and natural selection), which are then adapted over the course of a lifetime when environment relevant data is gradually obtained.
quad_eye_oh 3y ago
This is a thought that LLMs caused me to have: the emotion of surprise is just the brain model of the world discovering that it had been poorly calibrated, or making poor predictions.
- Hendrikto 3y ago
  Sounds like the free energy principle [0]: > The free energy principle is based on the Bayesian idea of the brain as an “inference engine.” Under the free energy principle, systems pursue paths of least surprise, or equivalently, minimize the difference between predictions based on their model of the world and their sense and associated perception. [0]: https://en.m.wikipedia.org/wiki/Free_energy_principle https://en.m.wikipedia.org/wiki/Free_energy_principle
mewse 3y ago
The question is whether you knew how that sentence was going to end when you started writing it, or indeed whether I knew that I was going to add this comma-separated adjunct when I started writing the preceding clause, and I cannot honestly say at this precise moment of typing whether the final word in this sentence is going to end up being 'yes' or 'no'.
- Mertax 3y ago
  I do wonder how well (quickly) the whole thought is formed in your head and it’s just the encoding into language that tricks you into thinking you didn’t know what the word would be. Having learned another language, the moment you start to feel “fluent” is when you start speaking first in the 2nd language and aren’t using your first language as an intermediate step to translate to your 2nd language.
- diffxx 3y ago
  For me it's a little more subtle. When I sit down to write, say, an email, I usually have a specific point that I want to make but I usually haven't yet figured out exactly how I want to get there. I'm not even sure it is even possible to conceive of writing without doing it word by word. Perhaps the difference though is that we have the ability to go forward and backwards in the sense that we can usually think faster than we can type or write which allows our thoughts to advance beyond the tokens we have produced. This allows for a kind of real time editing process that incorporates both past and possible future information. But I imagine there us probably a way to enhance LLMs to branch and edit before producing the next word materially (if they aren't already doing that). To go even more meta, there is an analogy I'm trying to make right now in which I am visualizing a road and thinking about how describing the road relates to the process of writing. In my mind's eye, I can see the full length of the road and all of its contours but I can't actually describe the individual stretches of the road coherently without enumerating them. Something similar happens with writing. I can visualize what I want to say far beyond the next word, but it's true that the actual process of writing goes word to word, much like how the process of token selection is described for an LLM. The question is whether the LLM has an analogous conception of where it is going. Going back to the process above, sometimes I know where I am going and haven't yet figured out how to articulate it yet. It is through the process of writing that I am able to articulate that thought. But the thought preceded my articulation of it. I don't know to what extent LLMs have coherent thoughts that they are articulating or if that even makes sense for the type of intelligence they project. My suspicion is that they don't have additional sensory inputs beyond language that give thoughts the immaterial shape that then is expressed in language. Without that, I am skeptical that they will truly get beyond regurgitating and/or remixing what has already been fed to them textually. That doesn't diminish how amazing they are, but I am somewhat more in the Brooks/Knuth camp that they are impressive and surprising, but there is something that ultimately leaves me a bit cold about them.
cageface 3y ago
Maybe in casual conversation but that's not how I experience my though process about anything non trivial at all. I usually spend a lot of time thinking about the concept in non verbal terms and that process involves recalling images and sensory information in fairly abstract terms and then through what feels like several iterations it starts to coalesce into something I can encode in language. I think we can all agree that these LLMs are surprisingly good at generating text that is often coherent but I don't see how you can discard all those extra inputs and claim you have the same process.
- panarky 3y ago
  > thinking about the concept in non verbal ... abstract terms The abstract terms we think about are concepts, and we think about multiple concepts, at various levels of abstraction, and their relationships to each other, before getting a sense of what we want to say or write. Only then do we begin speaking or writing, grouping concepts into paragraphs, breaking them down into sentences and words. And there's evidence that LLMs do something similar, creating embeddings for both big ideas and small details, modeling how the small details combine into larger concepts, discovering the relationships between concepts, and only then generating a probabilistic sequence of tokens to express those deeper concepts.
  - cageface 3y ago
    Right but they're limited to concepts that are derived from linguistic tokens. Human reasoning can include nonverbal inputs. For example asking GPT what would be a reasonable way to stack a list of arbitrary objects on top of each other.
    - namaria 3y ago
      Lately I've been think that staple of science fiction, the android robot, is the only way to get AGI. I thought a lot about computer vision when that was the bleeding edge of AI a few years back. It strikes me as obviously wrong to try training models on two dimensional arrays of pixels and expect them to develop visual skills. Why don't we give them two cameras and arms so they can learn about physics and start differentiating 3 dimensional input? I think when we have robots capable of interacting with the world as humans do we might be able to train them to develop near human levels of intelligence.
astrea 3y ago
But you and your brain are much more than the language-forming aspect.
- moffkalast 3y ago
  Yes, most of the brain is a glorified microcontroller for running and interpreting data from a billion skin sensors and driving hundreds of muscle actuators, doing depth and object inference from binocular images, locating sounds from the twin microphone array, etc. Language and especially reasoning is a very small part of it all.
- JoshTko 3y ago
  Language is the compression of our entire human experience into words, so I'd argue language captures the majority of the thinking aspect of the human experience.
ChatGTP 3y ago
I believe this view of humans is because we work in increasingly deterministic environments (the internet) using predictable methods of communication, text. We’ve kind of made the world a bit boring and deterministic, places where almost perfect is knowledge is obtainable and so everything feels more and more predictable. Your day probably consists of using Google, talking on platforms that don’t change much, solving already solved coding problems and communicating with others about office politics problems that we’ve all spoke about over and over again. We literally are just chatbots in this world.
zmmmmm 3y ago
Honestly I feel like one reason people are struggling with this is they can't accept a critical part of the truth : most people are stochastic parrots themselves most of the time. True, creative genuine deep thinking is an exceptional state of thinking for us.
- DJHenk 3y ago
  Years ago I transitioned from a developer role to a manager role and suddenly I had to do a lot more talking. Not all the talking needs to be a deeply involved exchange of complex ideas, a lot of it serves a different purpose. Sometimes it can be a simple as filling up the time in a pleasant way with a group of people that may or may not know each other that well. After getting some experience with this I noticed that I had developed a talking on/off button in my head. I could just simply turn it on and start talking. I could generate words that sounded good together and fit the purpose of the moment. But They just seemed to come from a different place in my brain than my conscious mind. Because that was not involved in this process at all. The only job my mind had was to turn the button off again at the right moment, for the rest it was free to think whatever it wanted. (I transferred back to development a couple of years later.)
  - namaria 3y ago
    The very fact that we can be conscious about our ability to generate language and work it separately from the rest of our consciousness tells me that chatbots might emulate our language generation ability. We shouldn't invert it to conclude that means they can reason about it.
    - TeMPOraL 3y ago
      > We shouldn't invert it to conclude that means they can reason about it. Agreed. However, I think it's somewhat accepted view that the bulk of work constituting reasoning happens subconsciously, with the conscious mind playing the role of a censor/gatekeeper, and occasionally handholding during when reasoning through tougher problems.
- 3y ago
anon84873628 3y ago
The sad truth is that we are all NPCs without free will. "Thoughts" are a non-physical phenomenon that, seemingly, arise from physical activity in the brain. The thoughts (our "experience") come after the electrochemical reaction that manifests them. How could these non-physical phenomena then turn around and influence the physical chemistry? They can't. By the time we experience a thought, the physical state for it has been represented, and that state causes the next one after it. Like a stone bouncing down a hill - chaotic but still deterministic. We're just along for the ride.
- antihipocrat 3y ago
  The act of being conscious of our own thoughts and using them to influence our physical state is skill practiced by many people with success.
  - anon84873628 3y ago
    Nope, that's just what it feels like we're doing. It's a good trick!
- namaria 3y ago
  Reducing a debate that has been raging for millennia across several disciplines to three lines of text tells me you don't have enough of a grasp on the subject to draw any conclusions about it, let alone resolve it altogether.
  - Biganon 3y ago
    Care to enlighten us a little? What is there even to debate? The position expressed by your parent comment is so uncomfortable that it's constantly being challenged, but that doesn't make it wrong. Or are "thoughts" some sort of magical fluff not bound by the laws of physics? Are we constantly pulling ourselves by our bootstraps? Giving birth to entirely new brain activity out of nothingness? This is just woo woo nonsense. Until someone can find an explanation that makes more sense than "natural selection slowly gave apes the ability to think abstractly and they started thinking about themselves", I won't believe in any kind of free will, and now the meta is reaching the point where we create entirely new brains out of silicon. "They just repeat stuff!" So do we
namaria 3y ago
Can you think without words? Can you imagine how it feels to swim, something you might have done, and extend it to fly like a bird? Can you read a novel and feel emotions about characters that don't exist beyond text descriptions of their actions? Language is a way to encode thought. It's not what thoughts are made of.
- copperx 3y ago
  > Language is a way to encode thought. It's not what thoughts are made of. What about people who think verbally and don't have a "mind's eye"? In that case, thoughts and their encoding might be closer to 1:1.
  - namaria 3y ago
    Are images and words the only way to think? Are these boundaries even fundamental or are they arbitrary? How does a blind deaf person think? Do you ever think in terms of sensations or movements? Can you remember a roller coaster ride, a car trip? Can you think of a taste? I find hard to conceive of people who can think only in terms of language.
goatlover 3y ago
Does that reasoning work when applied to what you just wrote? If you’re just predicting the next word, then what makes you confident in the conclusion that you’re just predicting your next word?
- JoshTko 3y ago
  Yes. This was my second draft. I reviewed my first draft which became an input in my second draft which gained more clarify after reviewing the first draft.
  - goatlover 3y ago
    The way people usually say it is that you revised your first draft upon reflection, because it didn't quite get across the point you wanted to make, or something to that effect. So what makes the way you state it more true than the way people typically state it?
    - JoshTko 3y ago
      I can switch from word generation and evaluation. I saw the score of 1st draft was below my target. And I used the insights from the eval of the 1st draft to inform the 2nd draft.

Why not? Your brain is already making predictions about what you expect to see and hear as a part of your perception of reality anyways.

quad_eye_oh 3y ago
Right, but LLMs suggest that learning how to predict and only training on predictions is sufficient to learn anything and to have emergent generative abilities. What if learning to predict the upcoming input is all that's all that is needed for general human intelligence? What if it is all that any animals do?
- 20after4 3y ago
  This right here. I actually am strongly starting to believe that this is indeed what's going on. I read the book Kingdom of Speech a few years ago and that also left me with the perspective that perhaps language has a lot more to do with how we think and perceive the world than most people like to admit. The book has been heavily criticized but I believe it made an interesting point about language.
  - moffkalast 3y ago
    I mean hasn't it been proven on multiple occasions that humans who never learn a language during their life also don't develop higher cognitive skills and stay on the level of a child? Clearly language plays a notable part in reasoning, and it's frankly no surprise that we're seeing this behaviour from language models. It's also quite interesting how foundation models and fine tuning appear analogous to being born with a brain that already has six million years' worth of weights in it (trained implicitly through random changes and natural selection), which are then adapted over the course of a lifetime when environment relevant data is gradually obtained.

quad_eye_oh 3y ago

This is a thought that LLMs caused me to have: the emotion of surprise is just the brain model of the world discovering that it had been poorly calibrated, or making poor predictions.

Hendrikto 3y ago
Sounds like the free energy principle [0]: > The free energy principle is based on the Bayesian idea of the brain as an “inference engine.” Under the free energy principle, systems pursue paths of least surprise, or equivalently, minimize the difference between predictions based on their model of the world and their sense and associated perception. [0]: https://en.m.wikipedia.org/wiki/Free_energy_principle https://en.m.wikipedia.org/wiki/Free_energy_principle

mewse 3y ago

The question is whether you knew how that sentence was going to end when you started writing it, or indeed whether I knew that I was going to add this comma-separated adjunct when I started writing the preceding clause, and I cannot honestly say at this precise moment of typing whether the final word in this sentence is going to end up being 'yes' or 'no'.

Mertax 3y ago
I do wonder how well (quickly) the whole thought is formed in your head and it’s just the encoding into language that tricks you into thinking you didn’t know what the word would be. Having learned another language, the moment you start to feel “fluent” is when you start speaking first in the 2nd language and aren’t using your first language as an intermediate step to translate to your 2nd language.
diffxx 3y ago
For me it's a little more subtle. When I sit down to write, say, an email, I usually have a specific point that I want to make but I usually haven't yet figured out exactly how I want to get there. I'm not even sure it is even possible to conceive of writing without doing it word by word. Perhaps the difference though is that we have the ability to go forward and backwards in the sense that we can usually think faster than we can type or write which allows our thoughts to advance beyond the tokens we have produced. This allows for a kind of real time editing process that incorporates both past and possible future information. But I imagine there us probably a way to enhance LLMs to branch and edit before producing the next word materially (if they aren't already doing that). To go even more meta, there is an analogy I'm trying to make right now in which I am visualizing a road and thinking about how describing the road relates to the process of writing. In my mind's eye, I can see the full length of the road and all of its contours but I can't actually describe the individual stretches of the road coherently without enumerating them. Something similar happens with writing. I can visualize what I want to say far beyond the next word, but it's true that the actual process of writing goes word to word, much like how the process of token selection is described for an LLM. The question is whether the LLM has an analogous conception of where it is going. Going back to the process above, sometimes I know where I am going and haven't yet figured out how to articulate it yet. It is through the process of writing that I am able to articulate that thought. But the thought preceded my articulation of it. I don't know to what extent LLMs have coherent thoughts that they are articulating or if that even makes sense for the type of intelligence they project. My suspicion is that they don't have additional sensory inputs beyond language that give thoughts the immaterial shape that then is expressed in language. Without that, I am skeptical that they will truly get beyond regurgitating and/or remixing what has already been fed to them textually. That doesn't diminish how amazing they are, but I am somewhat more in the Brooks/Knuth camp that they are impressive and surprising, but there is something that ultimately leaves me a bit cold about them.

cageface 3y ago

Maybe in casual conversation but that's not how I experience my though process about anything non trivial at all. I usually spend a lot of time thinking about the concept in non verbal terms and that process involves recalling images and sensory information in fairly abstract terms and then through what feels like several iterations it starts to coalesce into something I can encode in language. I think we can all agree that these LLMs are surprisingly good at generating text that is often coherent but I don't see how you can discard all those extra inputs and claim you have the same process.

panarky 3y ago
> thinking about the concept in non verbal ... abstract terms The abstract terms we think about are concepts, and we think about multiple concepts, at various levels of abstraction, and their relationships to each other, before getting a sense of what we want to say or write. Only then do we begin speaking or writing, grouping concepts into paragraphs, breaking them down into sentences and words. And there's evidence that LLMs do something similar, creating embeddings for both big ideas and small details, modeling how the small details combine into larger concepts, discovering the relationships between concepts, and only then generating a probabilistic sequence of tokens to express those deeper concepts.
- cageface 3y ago
  Right but they're limited to concepts that are derived from linguistic tokens. Human reasoning can include nonverbal inputs. For example asking GPT what would be a reasonable way to stack a list of arbitrary objects on top of each other.
  - namaria 3y ago
    Lately I've been think that staple of science fiction, the android robot, is the only way to get AGI. I thought a lot about computer vision when that was the bleeding edge of AI a few years back. It strikes me as obviously wrong to try training models on two dimensional arrays of pixels and expect them to develop visual skills. Why don't we give them two cameras and arms so they can learn about physics and start differentiating 3 dimensional input? I think when we have robots capable of interacting with the world as humans do we might be able to train them to develop near human levels of intelligence.

astrea 3y ago

But you and your brain are much more than the language-forming aspect.

moffkalast 3y ago
Yes, most of the brain is a glorified microcontroller for running and interpreting data from a billion skin sensors and driving hundreds of muscle actuators, doing depth and object inference from binocular images, locating sounds from the twin microphone array, etc. Language and especially reasoning is a very small part of it all.
JoshTko 3y ago
Language is the compression of our entire human experience into words, so I'd argue language captures the majority of the thinking aspect of the human experience.

ChatGTP 3y ago

I believe this view of humans is because we work in increasingly deterministic environments (the internet) using predictable methods of communication, text. We’ve kind of made the world a bit boring and deterministic, places where almost perfect is knowledge is obtainable and so everything feels more and more predictable. Your day probably consists of using Google, talking on platforms that don’t change much, solving already solved coding problems and communicating with others about office politics problems that we’ve all spoke about over and over again. We literally are just chatbots in this world.

zmmmmm 3y ago

Honestly I feel like one reason people are struggling with this is they can't accept a critical part of the truth : most people are stochastic parrots themselves most of the time. True, creative genuine deep thinking is an exceptional state of thinking for us.

DJHenk 3y ago
Years ago I transitioned from a developer role to a manager role and suddenly I had to do a lot more talking. Not all the talking needs to be a deeply involved exchange of complex ideas, a lot of it serves a different purpose. Sometimes it can be a simple as filling up the time in a pleasant way with a group of people that may or may not know each other that well. After getting some experience with this I noticed that I had developed a talking on/off button in my head. I could just simply turn it on and start talking. I could generate words that sounded good together and fit the purpose of the moment. But They just seemed to come from a different place in my brain than my conscious mind. Because that was not involved in this process at all. The only job my mind had was to turn the button off again at the right moment, for the rest it was free to think whatever it wanted. (I transferred back to development a couple of years later.)
- namaria 3y ago
  The very fact that we can be conscious about our ability to generate language and work it separately from the rest of our consciousness tells me that chatbots might emulate our language generation ability. We shouldn't invert it to conclude that means they can reason about it.
  - TeMPOraL 3y ago
    > We shouldn't invert it to conclude that means they can reason about it. Agreed. However, I think it's somewhat accepted view that the bulk of work constituting reasoning happens subconsciously, with the conscious mind playing the role of a censor/gatekeeper, and occasionally handholding during when reasoning through tougher problems.
3y ago

anon84873628 3y ago

The sad truth is that we are all NPCs without free will. "Thoughts" are a non-physical phenomenon that, seemingly, arise from physical activity in the brain. The thoughts (our "experience") come after the electrochemical reaction that manifests them. How could these non-physical phenomena then turn around and influence the physical chemistry? They can't. By the time we experience a thought, the physical state for it has been represented, and that state causes the next one after it. Like a stone bouncing down a hill - chaotic but still deterministic. We're just along for the ride.

antihipocrat 3y ago
The act of being conscious of our own thoughts and using them to influence our physical state is skill practiced by many people with success.
- anon84873628 3y ago
  Nope, that's just what it feels like we're doing. It's a good trick!
namaria 3y ago
Reducing a debate that has been raging for millennia across several disciplines to three lines of text tells me you don't have enough of a grasp on the subject to draw any conclusions about it, let alone resolve it altogether.
- Biganon 3y ago
  Care to enlighten us a little? What is there even to debate? The position expressed by your parent comment is so uncomfortable that it's constantly being challenged, but that doesn't make it wrong. Or are "thoughts" some sort of magical fluff not bound by the laws of physics? Are we constantly pulling ourselves by our bootstraps? Giving birth to entirely new brain activity out of nothingness? This is just woo woo nonsense. Until someone can find an explanation that makes more sense than "natural selection slowly gave apes the ability to think abstractly and they started thinking about themselves", I won't believe in any kind of free will, and now the meta is reaching the point where we create entirely new brains out of silicon. "They just repeat stuff!" So do we

namaria 3y ago

Can you think without words? Can you imagine how it feels to swim, something you might have done, and extend it to fly like a bird? Can you read a novel and feel emotions about characters that don't exist beyond text descriptions of their actions? Language is a way to encode thought. It's not what thoughts are made of.

copperx 3y ago
> Language is a way to encode thought. It's not what thoughts are made of. What about people who think verbally and don't have a "mind's eye"? In that case, thoughts and their encoding might be closer to 1:1.
- namaria 3y ago
  Are images and words the only way to think? Are these boundaries even fundamental or are they arbitrary? How does a blind deaf person think? Do you ever think in terms of sensations or movements? Can you remember a roller coaster ride, a car trip? Can you think of a taste? I find hard to conceive of people who can think only in terms of language.

goatlover 3y ago

Does that reasoning work when applied to what you just wrote? If you’re just predicting the next word, then what makes you confident in the conclusion that you’re just predicting your next word?

JoshTko 3y ago
Yes. This was my second draft. I reviewed my first draft which became an input in my second draft which gained more clarify after reviewing the first draft.
- goatlover 3y ago
  The way people usually say it is that you revised your first draft upon reflection, because it didn't quite get across the point you wanted to make, or something to that effect. So what makes the way you state it more true than the way people typically state it?
  - JoshTko 3y ago
    I can switch from word generation and evaluation. I saw the score of 1st draft was below my target. And I used the insights from the eval of the 1st draft to inform the 2nd draft.