That's what I'm thinking about a lot. Something like the models "activate" just some subset of parameters when working (if I understand the new models correctly

That's what I'm thinking about a lot. Something like the models "activate" just some subset of parameters when working (if I understand the new models correctly). So that model could activate parts of context which are relevant for the task at hand