My impression is that while domain specific models do better in their domain. They do not generalize well and cannot be transferred to different domains without

My impression is that while domain specific models do better in their domain. They do not generalize well and cannot be transferred to different domains without retraining. Hence comes mixture of experts (domain specifics). The trick is to juggle them all. Which needs another quite capable 'manager' model. The good thing is that they can be trained separately. Also at inference there in no need to call them all. Existing models, like GPT-4, can be used to create datasets for experts, or enhance/balance existing datasets.