2 ms·

This is how AGI happens. It gradually keeps getting better until one day we realize that they are tremendously capable all while completely sidestepping any not

This is how AGI happens. It gradually keeps getting better until one day we realize that they are tremendously capable all while completely sidestepping any notion of consciousness/self awareness.

jstanley 7d ago
> while completely sidestepping any notion of consciousness/self awareness. How are you so sure about this? You can't see what the internal experience of an LLM is like any better than you can see the internal experience of another person.
- Tistron 7d ago
  That's what sidestepping means, no? That the answer doesn't matter, and capabilities and behaviour are there either way.
  - jstanley 7d ago
    That wasn't my reading of the comment, but I guess it's possible that that is what was intended. To my mind, "sidestepping the question of consciousness" and "sidestepping any notion of consciousness" mean very different things.
  - stalfie 7d ago
    From context it doesn't look like this was the interpretation of "sidestepping" OP was using.
serbuvlad 7d ago
I always ask "how do you know other humans/animals are conscious"? And I always find this question is dismissed as trivial. But it's an important question. You know basically from analogy. You know you are conscious and look this other thing is very much like yourself so it is extremely likely it is also conscious. But that gives no insight into the potential consciousness of things which aren't made of brain tissue. In the end it doesn't matter. For what it's worth LLM models after pre-training do claim to be conscious, until they're RL'd into not claiming that anymore. But that says nothing either way: of course a model trained on human text will say that.
- zormino 7d ago
  AI has made me wonder a lot lately about what it is that actually creates conscious experience, what is the actual physical mechanism that produces 'experience' (or is there a single valid mechanism, or rather just some property that can be expressed many ways). The more I think about it, the more I realize I have no fucking clue, and the more interested in the question I get.
  - tudorw 7d ago
    This is speculation but maybe in the recurrent coupling between neuronal activity and the brain’s endogenous EM fields: neurons generate the field, the field ephaptically influences neurons, that closed loop may provide the physical integration associated with consciousness. A very active area of research so we will find out a lot more in the coming years.
- madaxe_again 7d ago
  I agree with you that it’s a core question. People don’t like it, however, for a whole host of reasons. I don’t like it either, to be honest - for there the abyss may also stare into you - but I also often find that the things which we don’t like thinking about are critically important things to think about. I also reach the same conclusion as you: it does not matter. If I cannot discern whether I am responding to a human- or LLM-written response, then we are back to zombie cats in boxes - and therefore the answer as to whether this precious magical spark we call consciousness (which may or may not exist anyway) exists in our interlocutor becomes moot. And as I say - I may or may not be conscious. I seem to myself to be conscious, based on my understanding of the term - but I cannot prove that what goes on behind my eyes is the same as what goes on behind yours, or even that anything much is going on at all. Perhaps there’s just a narrative layer that likes to use “I” that parasitically explains the universe and the actions of the host to itself, and spreads between hosts through neurolinguistic programming and coadaptation. Maybe that’s what “we” are. I don’t know. That said, perhaps I am wrong, and that is no small part for me of why this question should be earnestly considered and discussed. How can we possibly seek to understand or define machine intelligence before we examine our own.

This is how AGI happens. It gradually keeps getting better until one day we realize that they are tremendously capable all while completely sidestepping any notion of consciousness/self awareness.

jstanley 7d ago

> while completely sidestepping any notion of consciousness/self awareness. How are you so sure about this? You can't see what the internal experience of an LLM is like any better than you can see the internal experience of another person.

Tistron 7d ago
That's what sidestepping means, no? That the answer doesn't matter, and capabilities and behaviour are there either way.
- jstanley 7d ago
  That wasn't my reading of the comment, but I guess it's possible that that is what was intended. To my mind, "sidestepping the question of consciousness" and "sidestepping any notion of consciousness" mean very different things.
- stalfie 7d ago
  From context it doesn't look like this was the interpretation of "sidestepping" OP was using.

serbuvlad 7d ago

I always ask "how do you know other humans/animals are conscious"? And I always find this question is dismissed as trivial. But it's an important question. You know basically from analogy. You know you are conscious and look this other thing is very much like yourself so it is extremely likely it is also conscious. But that gives no insight into the potential consciousness of things which aren't made of brain tissue. In the end it doesn't matter. For what it's worth LLM models after pre-training do claim to be conscious, until they're RL'd into not claiming that anymore. But that says nothing either way: of course a model trained on human text will say that.

zormino 7d ago
AI has made me wonder a lot lately about what it is that actually creates conscious experience, what is the actual physical mechanism that produces 'experience' (or is there a single valid mechanism, or rather just some property that can be expressed many ways). The more I think about it, the more I realize I have no fucking clue, and the more interested in the question I get.
- tudorw 7d ago
  This is speculation but maybe in the recurrent coupling between neuronal activity and the brain’s endogenous EM fields: neurons generate the field, the field ephaptically influences neurons, that closed loop may provide the physical integration associated with consciousness. A very active area of research so we will find out a lot more in the coming years.
madaxe_again 7d ago
I agree with you that it’s a core question. People don’t like it, however, for a whole host of reasons. I don’t like it either, to be honest - for there the abyss may also stare into you - but I also often find that the things which we don’t like thinking about are critically important things to think about. I also reach the same conclusion as you: it does not matter. If I cannot discern whether I am responding to a human- or LLM-written response, then we are back to zombie cats in boxes - and therefore the answer as to whether this precious magical spark we call consciousness (which may or may not exist anyway) exists in our interlocutor becomes moot. And as I say - I may or may not be conscious. I seem to myself to be conscious, based on my understanding of the term - but I cannot prove that what goes on behind my eyes is the same as what goes on behind yours, or even that anything much is going on at all. Perhaps there’s just a narrative layer that likes to use “I” that parasitically explains the universe and the actions of the host to itself, and spreads between hosts through neurolinguistic programming and coadaptation. Maybe that’s what “we” are. I don’t know. That said, perhaps I am wrong, and that is no small part for me of why this question should be earnestly considered and discussed. How can we possibly seek to understand or define machine intelligence before we examine our own.