4 ms·
Democracy of descendant models that have been trained separately by partitioning the identified clusters with strong capabilities from an ancestor model, so, in
by pizza 3y ago
Democracy of descendant models that have been trained separately by partitioning the identified clusters with strong capabilities from an ancestor model, so, in effect, they are modular, and can be learned to be combined competitively.
- deely3 3y agoHeh. I understand all these words separately. Btw, to which of the question of parent comment this is an answer?
- omeze 3y agoAncestor model = pre-trained model that used a large diverse corpus Descendant models = models fine tuned from an ancestor on one particular domain, e.g. by partitioning your training data by subject or source Democracy = some weighted mix of the descendant models is used to find the next token