2 ms·
Yeah so far looking at the progression since 70s, domain-specific attempts stagnant while just-increase-the-input-and-core attempts get us where we are now
by mcmoor 3y ago
Yeah so far looking at the progression since 70s, domain-specific attempts stagnant while just-increase-the-input-and-core attempts get us where we are now
- two_in_one 3y agoMy impression is that while domain specific models do better in their domain. They do not generalize well and cannot be transferred to different domains without retraining. Hence comes mixture of experts (domain specifics). The trick is to juggle them all. Which needs another quite capable 'manager' model. The good thing is that they can be trained separately. Also at inference there in no need to call them all. Existing models, like GPT-4, can be used to create datasets for experts, or enhance/balance existing datasets.