I agree, but consider this: there will hopefully be a new paradigm that uses fewer parameters but some mechanism like transfer learning to allow more modular tr

I agree, but consider this: there will hopefully be a new paradigm that uses fewer parameters but some mechanism like transfer learning to allow more modular training and composition of models. In an old job, I wrote many US patents and if I still worked there, then I would persue this idea. Now, I am just throwing this idea up in the air hoping it may be useful.