2 ms·
> The fascinating thing is that capabilities of these models emerge at certain sizes and nobody knows why Is there a more technical dive into this statememt?
by nologic01 3y ago
> The fascinating thing is that capabilities of these models emerge at certain sizes and nobody knows why
Is there a more technical dive into this statememt?
In particular, is it an emergent statistical property of the model and/or the training data size or an emergent illusion of the human observer of model outputs (the way we visually perceive fluid motion once a frame rate exceeds a certain threshold).
The nature of this "emergence" is interesting both from a theoretical and a practical point of view.
The "large" model size (I put the word in quotes, because large compared to what equivalent task?) might be somehow intrinsic, in which case it will spark a race to make such sized computational resources more common place. But it might also be an imcomplete understanding of the model class.
- NhanH 3y agoThere is a paper saying that around 6-7 billion params, something happened that makes the larger transformer qualitative different than the smaller one. I forgot which paper it was though
- gwern 3y agoWei et al 2022 might be a good starting point: https://gwern.net/doc/ai/scaling/emergence/index#wei-et-al-2022-section https://gwern.net/doc/ai/scaling/emergence/index#wei-et-al-2...
- nologic01 3y agoGood reference. The sparsity of the data points (presumably due to the huge training costs) doesn't reveal too much at present but as the authors suggest the phenomenon might be reproduced in smaller models (which would make it easier to study in greater detail).