>The locked in LLM still uses tokens, which is language. It cannot operate without these tokens. I think that's incorrect on a technicality. Input sentences ar

>The locked in LLM still uses tokens, which is language. It cannot operate without these tokens. I think that's incorrect on a technicality. Input sentences are tokenized, but then the first step is to convert tokens into their high-dimensional embeddings. LLMs work with high dimensional piles of numbers, which start as embeddings in the first layer, and quickly become a nigh-incomprehensible weighted average of many different low-level computations, until something statistically interesting comes out the other side again >But also that thinking symbolically, as an LLM does, is only part of the experience of existence. I think the symbols are entirely an interface used to communicate with us. Two neural networks can be grafted to one another, making a deeper whole, and information will flow between the two using a shared floating-point representation far away from anything humans would recognize as language, tokens, symbols. LLMs internally are really piles of highly-dimensional numbers, which turns out to be really hard to interpret, despite efforts Whatever else they're doing that is not the part manipulating symbols, it's clearly a very different "existence" than the one we have. I agree that it doesn't really look anything like our experience of existence. But I think it's much harder to pin down the part that we think they don't have, in a way that won't be begging the question, or exclude some humans and some animals that we normally think of as conscious and feeling pain. >This non-linguistic part is what I am arguing that LLMs do not possess (discounting magic), ergo have no sense of self That's an interesting argument, but I think the devil is in the details, and whatever this non-linguistic part is, it will be very hard to define without risking begging the question, or without defining it as some sort of intangible property that cannot very well be proven to exist. In a strict sense, I think we can create in principle (and up to your definition of what language means) large multimodel transformer model that uses nothing recognizable as language, but nervertheless have vision inputs, a robotic frame, and are able to mimic some simple animal behaviors. We can surely train robots to pass whatever tests we use to demonstrate self-awareness, we can probably even indirectly get this behavior to emerge without explicitly 'training on the test set', but I think that wouldn't be convincing, it'd be a cheap trick. If we want to define the something else that isn't language, that we want to use to determine which beings have a sense of self, it needs to be robust enough that I can't create a simulacra that has the property, but that we strongly wouldn't expect to have a sense of self. (And robust enough that not having that thing really implies no sense of self) >I do not posit however that this neural function of self could not be replicated. Actually, since it is apparently possessed by insects, I would suggest that it may actually be very, very simple, some kind of neural REPL perhaps. I would worry that not only can it be replicated, but that it could be taken to the absurd by minmaxing the dumbest digital neural network that notionally has the non-language REPL property we want, until the property starts to look absurd. I'm thinking for instance of Integrated Information Theory (IIT), a really clever idea to define consciousness mathematically. It's a really fascinating proposition, but unfortunately very quickly reaches absurd conclusions, like some repetitions of simple digital circuits being vastly 'more conscious' than we are. I think whatever the non-language part is that we want to use to recognize a sense of self, it may run into the same problems as IIT. We simply can't put our fingers on whatever it is that we think makes us special, and any single thing we try to isolate can be replicated in a simpler model and taken to the absurd. >Im pretty sure we could achieve it though if we worked on it a bit… but should we? I'm sure we could outdo Nature in cruelty. My hope is that we care more about suffering than she does.