4 ms·
There was some recent research that was confirming capabilities improve smoothly as a function of parameter count yes. Their point was a lot of research that h
by ianbutler 3y ago
There was some recent research that was confirming capabilities improve smoothly as a function of parameter count yes.
Their point was a lot of research that had been showing jumps in performance at certain "breakpoints" for lack of a better word were the result of badly selected metrics versus a case of suddenly emergent behaviour.
The nice thing about that research is it suggests that if you are able to try something on a smaller model it will scale nicely to a bigger model.
- sigmoid10 3y agoThis has been known for several years. Especially zero and few-shot task performance scales extremely well with number of parameters. But more recently it was shown that you can actually trade parameters with training data volume and training time as well once you go into the billions of parameters. So while it takes more time to train, you can habe an equally powerful model with much fewer parameters and thus faster inference times.
- deleted 3y ago[deleted]
- ianbutler 3y agoThe paper was fairly recent, https://arxiv.org/abs/2304.15004 https://arxiv.org/abs/2304.15004. It was more thoroughly confirming what was generally agreed upon while debunking other reasons. The key insight from the abstract, "Specifically, nonlinear or discontinuous metrics produce apparent emergent abilities, whereas linear or continuous metrics produce smooth, continuous predictable changes in model performance." Yup that is another recent and interesting development for sure!
- sigmoid10 3y agoThe authors are a bit disingenuous here. They insinuate that GPT3's performance shows unpredictable behaviour change at certain scales using their weirdly constructed metrics (which may or may not be true - see below), while the original GPT3 paper already showed how these amazing "emergent" capabilities scale with parameters in a very predictable way: https://arxiv.org/pdf/2005.14165.pdf https://arxiv.org/pdf/2005.14165.pdf Also note that the plots in the appendix contain some obvious errors, so you definitely want to wait for a peer reviewed version of this paper (if it ever survives review).
- ianbutler 3y agoSorry, I think you've misunderstood, they're saying that's exactly the point. Those weird metrics are what they're debunking, not supporting. Per my snippet from the abstract in my last comment. Their point was a lot of papers use those weird metrics and it contributes to the appearance of emergent ability, when in reality its just the bad metrics. Nothing you've said so far disagrees with either my understanding or the conclusion of the paper I linked.
- sigmoid10 3y agoI think you misunderstood. The authors created the very issue they are "debunking." They took GPT3, slapped on some random metrics and showed that these metrics don't show scaling behaviour correctly, while the original publication of GPT3 actually did it correctly in the first place.
- deleted 3y ago[deleted]
- diffeomorphism 3y agoThe figures in the paper you linked below are log_10 scale on one axis but not the other. Thus even without quite understanding what all these figures measure, it seems to me that answer the question about "linear growth" is a hard no: double the size will not be twice as good. Or am I misunderstanding how to read these graphs?