3 ms·
Why is there a surprise when a larger model works better? And call it emergent?
by m3kw9 4y ago
Why is there a surprise when a larger model works better? And call it emergent?
- typon 4y agoThere's no a priori reason to assume that models would see a step function improvement in abilities as their size increases. Perhaps the null hypothesis would be that we'd observe a linear increase in capability. Empirical evidence shows that's not the case, and that's why studying this phenomena is interesting.
- deleted 4y ago[deleted]
- majormajor 4y agoThe judgement criteria itself looks step-like for many of these things. Word unscrambling, for instance: does a 15 percentage point increase in accuracy suggest that you're 15x better at it (from close to 0 to 15%ish)? Or could you do that by being modestly better, but that's just enough to cross the threshold to solve the simplest 15% of words in the task challenge you're being graded on? We're also using a log scale on the X axis there but a linear one for the Y axis which amplifies the "steppiness"
- bob1029 4y agoThis is a fair point. Things like arithmetic either work or they don't. Is partial credit issued as a linear interpolation between the predicted value and expected value? If not, then you necessarily will have step-like behavior in the data.
- int_19h 4y agoThe surprise is that it's qualitatively better. Like being able to talk and follow instructions where smaller models couldn't do either at all.