4 ms·
There's no a priori reason to assume that models would see a step function improvement in abilities as their size increases. Perhaps the null hypothesis would b
by typon 3y ago
There's no a priori reason to assume that models would see a step function improvement in abilities as their size increases. Perhaps the null hypothesis would be that we'd observe a linear increase in capability. Empirical evidence shows that's not the case, and that's why studying this phenomena is interesting.
- deleted 3y ago[deleted]
- majormajor 3y agoThe judgement criteria itself looks step-like for many of these things. Word unscrambling, for instance: does a 15 percentage point increase in accuracy suggest that you're 15x better at it (from close to 0 to 15%ish)? Or could you do that by being modestly better, but that's just enough to cross the threshold to solve the simplest 15% of words in the task challenge you're being graded on? We're also using a log scale on the X axis there but a linear one for the Y axis which amplifies the "steppiness"
- bob1029 3y agoThis is a fair point. Things like arithmetic either work or they don't. Is partial credit issued as a linear interpolation between the predicted value and expected value? If not, then you necessarily will have step-like behavior in the data.