4 ms·

The paper was fairly recent, https://arxiv.org/abs/2304.15004 https://arxiv.org/abs/2304.15004. It was more thoroughly confirming what was generally agreed upon

The paper was fairly recent, https://arxiv.org/abs/2304.15004 https://arxiv.org/abs/2304.15004. It was more thoroughly confirming what was generally agreed upon while debunking other reasons. The key insight from the abstract, "Specifically, nonlinear or discontinuous metrics produce apparent emergent abilities, whereas linear or continuous metrics produce smooth, continuous predictable changes in model performance." Yup that is another recent and interesting development for sure!

sigmoid10 3y ago
The authors are a bit disingenuous here. They insinuate that GPT3's performance shows unpredictable behaviour change at certain scales using their weirdly constructed metrics (which may or may not be true - see below), while the original GPT3 paper already showed how these amazing "emergent" capabilities scale with parameters in a very predictable way: https://arxiv.org/pdf/2005.14165.pdf https://arxiv.org/pdf/2005.14165.pdf Also note that the plots in the appendix contain some obvious errors, so you definitely want to wait for a peer reviewed version of this paper (if it ever survives review).
- ianbutler 3y ago
  Sorry, I think you've misunderstood, they're saying that's exactly the point. Those weird metrics are what they're debunking, not supporting. Per my snippet from the abstract in my last comment. Their point was a lot of papers use those weird metrics and it contributes to the appearance of emergent ability, when in reality its just the bad metrics. Nothing you've said so far disagrees with either my understanding or the conclusion of the paper I linked.
  - sigmoid10 3y ago
    I think you misunderstood. The authors created the very issue they are "debunking." They took GPT3, slapped on some random metrics and showed that these metrics don't show scaling behaviour correctly, while the original publication of GPT3 actually did it correctly in the first place.
    - deleted 3y ago
      [deleted]

sigmoid10 3y ago

The authors are a bit disingenuous here. They insinuate that GPT3's performance shows unpredictable behaviour change at certain scales using their weirdly constructed metrics (which may or may not be true - see below), while the original GPT3 paper already showed how these amazing "emergent" capabilities scale with parameters in a very predictable way: https://arxiv.org/pdf/2005.14165.pdf https://arxiv.org/pdf/2005.14165.pdf Also note that the plots in the appendix contain some obvious errors, so you definitely want to wait for a peer reviewed version of this paper (if it ever survives review).

ianbutler 3y ago
Sorry, I think you've misunderstood, they're saying that's exactly the point. Those weird metrics are what they're debunking, not supporting. Per my snippet from the abstract in my last comment. Their point was a lot of papers use those weird metrics and it contributes to the appearance of emergent ability, when in reality its just the bad metrics. Nothing you've said so far disagrees with either my understanding or the conclusion of the paper I linked.
- sigmoid10 3y ago
  I think you misunderstood. The authors created the very issue they are "debunking." They took GPT3, slapped on some random metrics and showed that these metrics don't show scaling behaviour correctly, while the original publication of GPT3 actually did it correctly in the first place.
  - deleted 3y ago
    [deleted]