3 ms·

>LLMs cannot function, by definition, without language. But we know there is an experience of being without language. Take a sufficiently big LLM, remove the i

by tux3 3y ago

>LLMs cannot function, by definition, without language. But we know there is an experience of being without language. Take a sufficiently big LLM, remove the input and the output, but let it run in a loop regardless, in a complete vacuum. Past the encoding and decoding layers, there is still math, computation, some sort of thinking happening, there is just no clear tie to human language anymore. The LLM with inputs and outputs sheared off is not useful to us, but if you keep it running, how would we know whether or not it has a sense of being, inside? If you presuppose that it did not before, it still doesn't after, of course. A stroke of the ventral pons can sometimes leave people entirely unresponsive. No language, no observable behavior. Except for some of them, who can still move their eyes. People with locked-in syndrome do not very much 'function', but they are as far as we can guess, still conscious. Being quadriplegic, they lose almost all external stimulus, they can't feel very much in that sense. They still have intelligence, and emotions, and thoughts, by extrapolation from those whose eyes still move to those more thoroughly disconnected. Furthermore, some people don't have an "inner voice". They simply don't use language for thinking, only for reading, hearing, and talking. These people have an inner experience that is pure thought and intuition, without translating it into any particular language in their head. So, we already knew that there can be experience of being without much or any language. Language happens, mostly, at the interface with the outside world. I think LLMs give us new data points to think about being, but not in this particular way. We already had being without language. Certainly small humans are born without knowing any language, language comes from the outside in, much in the way that the experience of being doesn't. >So now we suppose that thought, at least, does not require the experience of being that can be accomplished without language. That's reasonable to suppose in the abstract, but then applying that supposition to LLMs is just as untestable as your other hypothesis. We simply aren't aware of any way to know. Otherwise, I could apply the same supposition to people suffering of the most comprehensive forms of locked-in syndrome, who we are pretty sure still think, but show otherwise no externally observable proof of their inner experience of being. Just a disconnected brain, floating in the dark, sending action potentials in a complex loop. I think this reduces to a feeling that LLMs are categorically different, because they look so much simpler, and so different from us. If we were cleverer, perhaps we'd think of ourselves as relatively simple, too.

K0balt 3y ago
I think that LLMs are evidence that we are simple as well. The locked in LLM still uses tokens, which is language. It cannot operate without these tokens. My conjecture is that since LLMs are built on a technology designed to mimic biology (neural networks) that they probably work in a very similar way to us. But also that thinking symbolically, as an LLM does, is only part of the experience of existence. Other animals arguably experience self awareness, and have passed the tests that we use to demonstrate this. Even some insects. This non-linguistic part is what I am arguing that LLMs do not possess (discounting magic), ergo have no sense of self. It might be inherent in modelling a corporeal existence, or inherent to some other process of being. I do not posit however that this neural function of self could not be replicated. Actually, since it is apparently possessed by insects, I would suggest that it may actually be very, very simple, some kind of neural REPL perhaps. Regardless, LLMs in their current form do not seem to possess this sort of self awareness. Im pretty sure we could achieve it though if we worked on it a bit… but should we?
- tux3 3y ago
  >The locked in LLM still uses tokens, which is language. It cannot operate without these tokens. I think that's incorrect on a technicality. Input sentences are tokenized, but then the first step is to convert tokens into their high-dimensional embeddings. LLMs work with high dimensional piles of numbers, which start as embeddings in the first layer, and quickly become a nigh-incomprehensible weighted average of many different low-level computations, until something statistically interesting comes out the other side again >But also that thinking symbolically, as an LLM does, is only part of the experience of existence. I think the symbols are entirely an interface used to communicate with us. Two neural networks can be grafted to one another, making a deeper whole, and information will flow between the two using a shared floating-point representation far away from anything humans would recognize as language, tokens, symbols. LLMs internally are really piles of highly-dimensional numbers, which turns out to be really hard to interpret, despite efforts Whatever else they're doing that is not the part manipulating symbols, it's clearly a very different "existence" than the one we have. I agree that it doesn't really look anything like our experience of existence. But I think it's much harder to pin down the part that we think they don't have, in a way that won't be begging the question, or exclude some humans and some animals that we normally think of as conscious and feeling pain. >This non-linguistic part is what I am arguing that LLMs do not possess (discounting magic), ergo have no sense of self That's an interesting argument, but I think the devil is in the details, and whatever this non-linguistic part is, it will be very hard to define without risking begging the question, or without defining it as some sort of intangible property that cannot very well be proven to exist. In a strict sense, I think we can create in principle (and up to your definition of what language means) large multimodel transformer model that uses nothing recognizable as language, but nervertheless have vision inputs, a robotic frame, and are able to mimic some simple animal behaviors. We can surely train robots to pass whatever tests we use to demonstrate self-awareness, we can probably even indirectly get this behavior to emerge without explicitly 'training on the test set', but I think that wouldn't be convincing, it'd be a cheap trick. If we want to define the something else that isn't language, that we want to use to determine which beings have a sense of self, it needs to be robust enough that I can't create a simulacra that has the property, but that we strongly wouldn't expect to have a sense of self. (And robust enough that not having that thing really implies no sense of self) >I do not posit however that this neural function of self could not be replicated. Actually, since it is apparently possessed by insects, I would suggest that it may actually be very, very simple, some kind of neural REPL perhaps. I would worry that not only can it be replicated, but that it could be taken to the absurd by minmaxing the dumbest digital neural network that notionally has the non-language REPL property we want, until the property starts to look absurd. I'm thinking for instance of Integrated Information Theory (IIT), a really clever idea to define consciousness mathematically. It's a really fascinating proposition, but unfortunately very quickly reaches absurd conclusions, like some repetitions of simple digital circuits being vastly 'more conscious' than we are. I think whatever the non-language part is that we want to use to recognize a sense of self, it may run into the same problems as IIT. We simply can't put our fingers on whatever it is that we think makes us special, and any single thing we try to isolate can be replicated in a simpler model and taken to the absurd. >Im pretty sure we could achieve it though if we worked on it a bit… but should we? I'm sure we could outdo Nature in cruelty. My hope is that we care more about suffering than she does.

by tux3 3y ago

K0balt 3y ago

I think that LLMs are evidence that we are simple as well. The locked in LLM still uses tokens, which is language. It cannot operate without these tokens. My conjecture is that since LLMs are built on a technology designed to mimic biology (neural networks) that they probably work in a very similar way to us. But also that thinking symbolically, as an LLM does, is only part of the experience of existence. Other animals arguably experience self awareness, and have passed the tests that we use to demonstrate this. Even some insects. This non-linguistic part is what I am arguing that LLMs do not possess (discounting magic), ergo have no sense of self. It might be inherent in modelling a corporeal existence, or inherent to some other process of being. I do not posit however that this neural function of self could not be replicated. Actually, since it is apparently possessed by insects, I would suggest that it may actually be very, very simple, some kind of neural REPL perhaps. Regardless, LLMs in their current form do not seem to possess this sort of self awareness. Im pretty sure we could achieve it though if we worked on it a bit… but should we?

tux3 3y ago
>The locked in LLM still uses tokens, which is language. It cannot operate without these tokens. I think that's incorrect on a technicality. Input sentences are tokenized, but then the first step is to convert tokens into their high-dimensional embeddings. LLMs work with high dimensional piles of numbers, which start as embeddings in the first layer, and quickly become a nigh-incomprehensible weighted average of many different low-level computations, until something statistically interesting comes out the other side again >But also that thinking symbolically, as an LLM does, is only part of the experience of existence. I think the symbols are entirely an interface used to communicate with us. Two neural networks can be grafted to one another, making a deeper whole, and information will flow between the two using a shared floating-point representation far away from anything humans would recognize as language, tokens, symbols. LLMs internally are really piles of highly-dimensional numbers, which turns out to be really hard to interpret, despite efforts Whatever else they're doing that is not the part manipulating symbols, it's clearly a very different "existence" than the one we have. I agree that it doesn't really look anything like our experience of existence. But I think it's much harder to pin down the part that we think they don't have, in a way that won't be begging the question, or exclude some humans and some animals that we normally think of as conscious and feeling pain. >This non-linguistic part is what I am arguing that LLMs do not possess (discounting magic), ergo have no sense of self That's an interesting argument, but I think the devil is in the details, and whatever this non-linguistic part is, it will be very hard to define without risking begging the question, or without defining it as some sort of intangible property that cannot very well be proven to exist. In a strict sense, I think we can create in principle (and up to your definition of what language means) large multimodel transformer model that uses nothing recognizable as language, but nervertheless have vision inputs, a robotic frame, and are able to mimic some simple animal behaviors. We can surely train robots to pass whatever tests we use to demonstrate self-awareness, we can probably even indirectly get this behavior to emerge without explicitly 'training on the test set', but I think that wouldn't be convincing, it'd be a cheap trick. If we want to define the something else that isn't language, that we want to use to determine which beings have a sense of self, it needs to be robust enough that I can't create a simulacra that has the property, but that we strongly wouldn't expect to have a sense of self. (And robust enough that not having that thing really implies no sense of self) >I do not posit however that this neural function of self could not be replicated. Actually, since it is apparently possessed by insects, I would suggest that it may actually be very, very simple, some kind of neural REPL perhaps. I would worry that not only can it be replicated, but that it could be taken to the absurd by minmaxing the dumbest digital neural network that notionally has the non-language REPL property we want, until the property starts to look absurd. I'm thinking for instance of Integrated Information Theory (IIT), a really clever idea to define consciousness mathematically. It's a really fascinating proposition, but unfortunately very quickly reaches absurd conclusions, like some repetitions of simple digital circuits being vastly 'more conscious' than we are. I think whatever the non-language part is that we want to use to recognize a sense of self, it may run into the same problems as IIT. We simply can't put our fingers on whatever it is that we think makes us special, and any single thing we try to isolate can be replicated in a simpler model and taken to the absurd. >Im pretty sure we could achieve it though if we worked on it a bit… but should we? I'm sure we could outdo Nature in cruelty. My hope is that we care more about suffering than she does.