Yeah, but we're not talking about alien blobs, we're talking about pre-trained transformers. I'm 100% certain that if you make them bigger and better then all y

Yeah, but we're not talking about alien blobs, we're talking about pre-trained transformers. I'm 100% certain that if you make them bigger and better then all you will have is a bigger better pre-trained transformer. Scale it up (and sprinkle some magic fairy dust on it?) and it'll become sentient seems to be the thought process. Didn't work for CYC, and not going to work here either. We need architecture, not scale or efficiency or bells and whistles. Get rid of pre-training and design an architecture and learning algorithm that will learn continuously and incrementally from it's own actions and mistakes (i.e. prediction failures) and we'll start to get somewhere.