I think there is a small possibility that optimal scaling as we understand it right now (scaling compute + task/data diversity + parameter count in an optimal f

I think there is a small possibility that optimal scaling as we understand it right now (scaling compute + task/data diversity + parameter count in an optimal fashion), could just emerge (or scale) consciousness. We at present do not know what leads to emergence in the many now-documented examples of emergent capabilities in large NNs (see https://arxiv.org/abs/2206.07682#google https://arxiv.org/abs/2206.07682#google https://gwern.net/doc/ai/scaling/emergence/index https://gwern.net/doc/ai/scaling/emergence/index ), cannot predict either their timing or their presence, and cannot even be sure that there is not 'hidden scaling' where the capability would exhibit smooth scaling if only we were prompting for it correctly (several tasks show flat scaling when prompted 'normally' and then smooth scaling when an inner-monologue prompt is used). Similarly, 'inverse scaling' turns out to be 'U-shaped' when tested on PaLM ( https://arxiv.org/abs/2211.02011v3#google https://arxiv.org/abs/2211.02011v3#google ), but no one had any good way of predicting what tasks would show inverse scaling nor whether PaLM would be adequate. There's a lot we do not know about DL scaling right now, which means we definitely do not know how consciousness does or doesn't work in DL scaling. Not that we have any idea how to benchmark it right now, nor is anyone doing so, so of course it may not need to emerge at all... There's also concerning scaling trends like power-seeking or self-preservation ( https://arxiv.org/abs/2212.09251#anthropic https://arxiv.org/abs/2212.09251#anthropic ), results which may sting a little more right now if you've been reading the Bing Sydney transcripts - at present, those outputs are 'just' imitation/memorization but there is no way to know at what threshold the memorization is replaced by generalization and becomes genuine agency. (Sufficiently advanced imitation is indistinguishable from the real thing.)