13 ms·

Do you believe models can go through the kind of phase transition you mention, after which they start modeling themselves as agents in a world (i.e., they beco

by cs702 4y ago

Do you believe models can go through the kind of phase transition you mention, after which they start modeling themselves as agents in a world (i.e., they become "conscious" or "self-aware," for lack of a better term), without requiring any new engineering breakthroughs or theoretical advances?

rtkwe 4y ago
We have such a poor understanding of our own consciousness that I think it's hard to answer that. We know a fair amount of what goes on in our heads and fundamentally it's fairly closely modelled by neural nets but we don't really know where (or even really if) it jumps from a complex but dead input output set of neurons to a (seemingly) self reflective consciousness.
- cs702 4y ago
  Yes, at the moment it's a matter of belief. That's why I asked gwern if he/she/they believe new breakthroughs will or won't be necessary.
  - rtkwe 4y ago
    I think there's probably additional scale or some internal function in neurons that we haven't captured through the current weigh and sum that we haven't captured. I don't think there's anything inherently irreproducible though.
efitz 4y ago
When we start building models to train and optimize other models, I would expect emergent behavior. If we did something foolish like building yhe system in such a way that it could write and execute code, then the emergent behavior could do things as nasty as anything else on the internet. We have also seen emergent behavior in programmable hardware [1] so I tend to be pessimistic about the limits of what such a system could do. Also it could pay human minions to interact with the physical world if needed. [1] https://www.damninteresting.com/on-the-origin-of-circuits https://www.damninteresting.com/on-the-origin-of-circuits
sdwr 4y ago
Not gwern, but I believe absolutely yes. I bet it could be done by chaining current models together, using text files as intermediate state. Awareness of self is just another game to optimize in an orthogonal (timewise) direction. Enough models and containers floating around would probably build an AI "molecule" by chance, like amino acids in primordial soup or whatever. People are harping on the chinese room angle, but thats all irrelephant
gwern 4y ago
I think there is a small possibility that optimal scaling as we understand it right now (scaling compute + task/data diversity + parameter count in an optimal fashion), could just emerge (or scale) consciousness. We at present do not know what leads to emergence in the many now-documented examples of emergent capabilities in large NNs (see https://arxiv.org/abs/2206.07682#google https://arxiv.org/abs/2206.07682#google https://gwern.net/doc/ai/scaling/emergence/index https://gwern.net/doc/ai/scaling/emergence/index ), cannot predict either their timing or their presence, and cannot even be sure that there is not 'hidden scaling' where the capability would exhibit smooth scaling if only we were prompting for it correctly (several tasks show flat scaling when prompted 'normally' and then smooth scaling when an inner-monologue prompt is used). Similarly, 'inverse scaling' turns out to be 'U-shaped' when tested on PaLM ( https://arxiv.org/abs/2211.02011v3#google https://arxiv.org/abs/2211.02011v3#google ), but no one had any good way of predicting what tasks would show inverse scaling nor whether PaLM would be adequate. There's a lot we do not know about DL scaling right now, which means we definitely do not know how consciousness does or doesn't work in DL scaling. Not that we have any idea how to benchmark it right now, nor is anyone doing so, so of course it may not need to emerge at all... There's also concerning scaling trends like power-seeking or self-preservation ( https://arxiv.org/abs/2212.09251#anthropic https://arxiv.org/abs/2212.09251#anthropic ), results which may sting a little more right now if you've been reading the Bing Sydney transcripts - at present, those outputs are 'just' imitation/memorization but there is no way to know at what threshold the memorization is replaced by generalization and becomes genuine agency. (Sufficiently advanced imitation is indistinguishable from the real thing.)
- cs702 4y ago
  Thank you. That makes sense to me. FWIW, it sure feels as if there's some unknown probability of self-awareness/consciousness emerging as a consequence of more computation + larger models + greater task diversity, but as you point out, we really don't know, and cannot predict the occurrence or timing of such an emergence. It would be a "black swan," as defined by Taleb.

by cs702 4y ago

rtkwe 4y ago

We have such a poor understanding of our own consciousness that I think it's hard to answer that. We know a fair amount of what goes on in our heads and fundamentally it's fairly closely modelled by neural nets but we don't really know where (or even really if) it jumps from a complex but dead input output set of neurons to a (seemingly) self reflective consciousness.

cs702 4y ago
Yes, at the moment it's a matter of belief. That's why I asked gwern if he/she/they believe new breakthroughs will or won't be necessary.
- rtkwe 4y ago
  I think there's probably additional scale or some internal function in neurons that we haven't captured through the current weigh and sum that we haven't captured. I don't think there's anything inherently irreproducible though.

efitz 4y ago

When we start building models to train and optimize other models, I would expect emergent behavior. If we did something foolish like building yhe system in such a way that it could write and execute code, then the emergent behavior could do things as nasty as anything else on the internet. We have also seen emergent behavior in programmable hardware [1] so I tend to be pessimistic about the limits of what such a system could do. Also it could pay human minions to interact with the physical world if needed. [1] https://www.damninteresting.com/on-the-origin-of-circuits https://www.damninteresting.com/on-the-origin-of-circuits

sdwr 4y ago

Not gwern, but I believe absolutely yes. I bet it could be done by chaining current models together, using text files as intermediate state. Awareness of self is just another game to optimize in an orthogonal (timewise) direction. Enough models and containers floating around would probably build an AI "molecule" by chance, like amino acids in primordial soup or whatever. People are harping on the chinese room angle, but thats all irrelephant

gwern 4y ago

I think there is a small possibility that optimal scaling as we understand it right now (scaling compute + task/data diversity + parameter count in an optimal fashion), could just emerge (or scale) consciousness. We at present do not know what leads to emergence in the many now-documented examples of emergent capabilities in large NNs (see https://arxiv.org/abs/2206.07682#google https://arxiv.org/abs/2206.07682#google https://gwern.net/doc/ai/scaling/emergence/index https://gwern.net/doc/ai/scaling/emergence/index ), cannot predict either their timing or their presence, and cannot even be sure that there is not 'hidden scaling' where the capability would exhibit smooth scaling if only we were prompting for it correctly (several tasks show flat scaling when prompted 'normally' and then smooth scaling when an inner-monologue prompt is used). Similarly, 'inverse scaling' turns out to be 'U-shaped' when tested on PaLM ( https://arxiv.org/abs/2211.02011v3#google https://arxiv.org/abs/2211.02011v3#google ), but no one had any good way of predicting what tasks would show inverse scaling nor whether PaLM would be adequate. There's a lot we do not know about DL scaling right now, which means we definitely do not know how consciousness does or doesn't work in DL scaling. Not that we have any idea how to benchmark it right now, nor is anyone doing so, so of course it may not need to emerge at all... There's also concerning scaling trends like power-seeking or self-preservation ( https://arxiv.org/abs/2212.09251#anthropic https://arxiv.org/abs/2212.09251#anthropic ), results which may sting a little more right now if you've been reading the Bing Sydney transcripts - at present, those outputs are 'just' imitation/memorization but there is no way to know at what threshold the memorization is replaced by generalization and becomes genuine agency. (Sufficiently advanced imitation is indistinguishable from the real thing.)

cs702 4y ago
Thank you. That makes sense to me. FWIW, it sure feels as if there's some unknown probability of self-awareness/consciousness emerging as a consequence of more computation + larger models + greater task diversity, but as you point out, we really don't know, and cannot predict the occurrence or timing of such an emergence. It would be a "black swan," as defined by Taleb.