4 ms·

Yes, yes, more explorations in this direction. For a couple of years now, I've had this half-articulated sense that the uncanny ability of sufficiently-advance

Yes, yes, more explorations in this direction. For a couple of years now, I've had this half-articulated sense that the uncanny ability of sufficiently-advanced language models to back into convincing simulations of conscious thought entirely via predicting language tokens means something profound about the nature of language itself. I'm sure there are much smarter people than I thinking about this (and probably quite a bit of background reading that would help; Chomsky, perhaps McLuhan?) but it feels like, in parallel to everything going on in the development of LLMs, there's also something big about us waiting there under the surface.

skydhash 2y ago
> convincing simulations of conscious thought entirely via predicting language tokens means something profound about the nature of language itself. > there's also something big about us waiting there under the surface. I don't believe so. In "The Origins of Knowledge and Imagination" by Jacob Brownoski, he argues that human language have four unique characteristics: - We can separate information (data of what being described) from emotional content (how we're supposed to react). There's no longer a bijection between communication and action. - We can extend the time reference of the communication content. We talk about the past, we plan for the future. - We can refer to ourselves. So we examine what we've done and iterate over it until we fix the errors. We can see ourselves doing the action without actually doing it. - We can rearrange units of languages to have different meanings. The same words can have different meanings based on their order. So meaning depends not only on the words, but their sequence. And that goes from words to phrases to sequence of dialogs. The fourth point is the most important. LLMs by predicting languages tokens can give use the most common order for a particular context. And because we don't have that many words, their orders can be extracted from books and other written content. But then they fail for the higher levels, mostly because that's when everything get unique. As for the third point, by observing ourselves, our communication is constantly being based on reality, which grounds it in truth. And because we can extend the reference it's based on, that leads us to observe changes and model laws. The first point allows us to separate what things are from what we should do or feel based on their existence and absence. Instead of the LLMs fooling us, it's more us fooling ourselves, because by recognizing meaning in sentences, we try to extract meanings for longer sequences of text where there aren't any. Why? Because there is no "I" that has done the job of extracting information and using language to transmit it (while still cognizant of the imperfection of natural languages). LLMs are lossy compressions of ideas. Only the smallest survives and then it generates much more false ones.
- justinjlynn 2y ago
  Are you certain that you're not playing with words to arrive at a predetermined conclusion? What is this "I" to which you're referring and how can you demonstrate that "I" does not or cannot exist within systems such as these? Further, if you are to find something which qualifies as an "I" elsewhere, what makes that elsewhere fundamentally different and therefore capable of supporting and being an "I" and is that elsewhere such simply by definition or in and of itself? Further, if the language usage is indistinguishable from the language usage of an "I", is the difference of source meaningful? If so, why?
  - skydhash 2y ago
    The "I" is stemmed from the theory of the mind. We can only access our own mind and thus has no way to infer the thoughts of other. So we observe them and infer based on our own patterns. In a sense, we assume that others have the same mechanism that we possess, and thus we engage in interactions with them. So far, there is no demonstration of reasoning within systems such as these, it's all simulation of the communication channel themselves. > Further, if the language usage is indistinguishable from the language usage of an "I", is the difference of source meaningful? Is it indistinguishable? The first thing we look for in communication is consistency so that we can examine for intent. And this is after we determine the other party. Because we know the intent is not ours. But what I've seen of prompt engineering is that the communication intent always come from the person, not the models. Then it goes on to find the most likely continuation of this intent (based on the model training) and then it quickly become an echo chamber. It's search in lexical space and you can see the limits when it became a oscillating loop between the same set of reply. Because there's no "I don't know" damping.
- brianush1 2y ago
  Why does there need to be an "I" that uses language to transmit information? Language itself encodes information. I can read a piece of text and gain something from it. Where the text came from is irrelevant.
  - skydhash 2y ago
furstenheim 2y ago
100%, maybe intelligence is not as mysterious and extraordinary as we thought
leobg 2y ago
Chomsky, of all people? Chomsky rose to fame by attacking BF Skinner’s book “Verbal Behavior”. Which is the book that made exactly the case you’re making now, only some 60 years ago. Skinner would marvel at today’s LLMs. They are the most elegant proof that intelligence is not just shaped by external contingencies, but that it is identical with those contingencies.
ryandv 2y ago
To this list I would absolutely add Julian Jaynes' "The Origin of Consciousness in the Breakdown of the Bicameral Mind." > simulations of conscious thought entirely via prediction language tokens Jaynes goes so far as to assert that language generates consciousness, which is characterized by (amongst other features) its narrative structure, as well as its production of a metaphor of our selves that can inhabit a spatiotemporal mental space that serves as an analog for the physical world; the mental space where we imagine potential actions, play with ideas, predict future outcomes, and analyze concepts prior to taking action in the "real, actual" world. The generation of metaphors is inextricably linked to the psychotechnology (to pull a word from vocabulary discussed by John Vervaeke in his "Awakening from the Meaning Crisis" series) of language, which is the means by which one object can be described and elaborated by its similarity to another. As an etymological example: the Sanskrit word "bhu" which means "to grow" forms the basis of the modern English verb "to be," but predates lofty abstract notions such as that of "being," "ontology," or "existence." It's from the known and the familiar (plant or animal growth) that we can reach out into the unknown and the unfamiliar (the concept of being), using (psycho-)technologies such as language to extend our cognition in the same way a hammer or a bicycle extends our body. There is something here about language being the substrate of thought, and perhaps even consciousness in general as Jaynes would seem to assert in Book I of his 1976 work, where he spends a considerable amount of time discussing metaphor and language in connection to his definition of "consciousness." There are also questions of "intentionality" and whether or not computers and their internal representations can actually be "about" something in the way that our language and our ideas can be "about" something in the physical (or even ideal) world that we want to discuss. Searle and the "Chinese room" argument come to mind. Turing famously dodged this question in his paper "Computing Machinery and Intelligence" by substituting what is now called the "Turing test" in lieu of answering the question of whether or not "machines" can "think" (whatever those two words actually mean).
- mikewarot 2y ago
  >Jaynes goes so far as to assert that language generates consciousness The recent discussion of Helen Keller[1] and her description of learning the meaning of "I", strongly backs this assertion, on my opinion. I read her words as implying that you can't have consciousness without self identity. [1] https://news.ycombinator.com/item?id=40466814 https://news.ycombinator.com/item?id=40466814

skydhash 2y ago

> convincing simulations of conscious thought entirely via predicting language tokens means something profound about the nature of language itself. > there's also something big about us waiting there under the surface. I don't believe so. In "The Origins of Knowledge and Imagination" by Jacob Brownoski, he argues that human language have four unique characteristics: - We can separate information (data of what being described) from emotional content (how we're supposed to react). There's no longer a bijection between communication and action. - We can extend the time reference of the communication content. We talk about the past, we plan for the future. - We can refer to ourselves. So we examine what we've done and iterate over it until we fix the errors. We can see ourselves doing the action without actually doing it. - We can rearrange units of languages to have different meanings. The same words can have different meanings based on their order. So meaning depends not only on the words, but their sequence. And that goes from words to phrases to sequence of dialogs. The fourth point is the most important. LLMs by predicting languages tokens can give use the most common order for a particular context. And because we don't have that many words, their orders can be extracted from books and other written content. But then they fail for the higher levels, mostly because that's when everything get unique. As for the third point, by observing ourselves, our communication is constantly being based on reality, which grounds it in truth. And because we can extend the reference it's based on, that leads us to observe changes and model laws. The first point allows us to separate what things are from what we should do or feel based on their existence and absence. Instead of the LLMs fooling us, it's more us fooling ourselves, because by recognizing meaning in sentences, we try to extract meanings for longer sequences of text where there aren't any. Why? Because there is no "I" that has done the job of extracting information and using language to transmit it (while still cognizant of the imperfection of natural languages). LLMs are lossy compressions of ideas. Only the smallest survives and then it generates much more false ones.

justinjlynn 2y ago
Are you certain that you're not playing with words to arrive at a predetermined conclusion? What is this "I" to which you're referring and how can you demonstrate that "I" does not or cannot exist within systems such as these? Further, if you are to find something which qualifies as an "I" elsewhere, what makes that elsewhere fundamentally different and therefore capable of supporting and being an "I" and is that elsewhere such simply by definition or in and of itself? Further, if the language usage is indistinguishable from the language usage of an "I", is the difference of source meaningful? If so, why?
- skydhash 2y ago
  The "I" is stemmed from the theory of the mind. We can only access our own mind and thus has no way to infer the thoughts of other. So we observe them and infer based on our own patterns. In a sense, we assume that others have the same mechanism that we possess, and thus we engage in interactions with them. So far, there is no demonstration of reasoning within systems such as these, it's all simulation of the communication channel themselves. > Further, if the language usage is indistinguishable from the language usage of an "I", is the difference of source meaningful? Is it indistinguishable? The first thing we look for in communication is consistency so that we can examine for intent. And this is after we determine the other party. Because we know the intent is not ours. But what I've seen of prompt engineering is that the communication intent always come from the person, not the models. Then it goes on to find the most likely continuation of this intent (based on the model training) and then it quickly become an echo chamber. It's search in lexical space and you can see the limits when it became a oscillating loop between the same set of reply. Because there's no "I don't know" damping.
brianush1 2y ago
Why does there need to be an "I" that uses language to transmit information? Language itself encodes information. I can read a piece of text and gain something from it. Where the text came from is irrelevant.
- skydhash 2y ago

furstenheim 2y ago

100%, maybe intelligence is not as mysterious and extraordinary as we thought

leobg 2y ago

Chomsky, of all people? Chomsky rose to fame by attacking BF Skinner’s book “Verbal Behavior”. Which is the book that made exactly the case you’re making now, only some 60 years ago. Skinner would marvel at today’s LLMs. They are the most elegant proof that intelligence is not just shaped by external contingencies, but that it is identical with those contingencies.

ryandv 2y ago

To this list I would absolutely add Julian Jaynes' "The Origin of Consciousness in the Breakdown of the Bicameral Mind." > simulations of conscious thought entirely via prediction language tokens Jaynes goes so far as to assert that language generates consciousness, which is characterized by (amongst other features) its narrative structure, as well as its production of a metaphor of our selves that can inhabit a spatiotemporal mental space that serves as an analog for the physical world; the mental space where we imagine potential actions, play with ideas, predict future outcomes, and analyze concepts prior to taking action in the "real, actual" world. The generation of metaphors is inextricably linked to the psychotechnology (to pull a word from vocabulary discussed by John Vervaeke in his "Awakening from the Meaning Crisis" series) of language, which is the means by which one object can be described and elaborated by its similarity to another. As an etymological example: the Sanskrit word "bhu" which means "to grow" forms the basis of the modern English verb "to be," but predates lofty abstract notions such as that of "being," "ontology," or "existence." It's from the known and the familiar (plant or animal growth) that we can reach out into the unknown and the unfamiliar (the concept of being), using (psycho-)technologies such as language to extend our cognition in the same way a hammer or a bicycle extends our body. There is something here about language being the substrate of thought, and perhaps even consciousness in general as Jaynes would seem to assert in Book I of his 1976 work, where he spends a considerable amount of time discussing metaphor and language in connection to his definition of "consciousness." There are also questions of "intentionality" and whether or not computers and their internal representations can actually be "about" something in the way that our language and our ideas can be "about" something in the physical (or even ideal) world that we want to discuss. Searle and the "Chinese room" argument come to mind. Turing famously dodged this question in his paper "Computing Machinery and Intelligence" by substituting what is now called the "Turing test" in lieu of answering the question of whether or not "machines" can "think" (whatever those two words actually mean).

mikewarot 2y ago
>Jaynes goes so far as to assert that language generates consciousness The recent discussion of Helen Keller[1] and her description of learning the meaning of "I", strongly backs this assertion, on my opinion. I read her words as implying that you can't have consciousness without self identity. [1] https://news.ycombinator.com/item?id=40466814 https://news.ycombinator.com/item?id=40466814