3 ms·
It seems a little silly to pretend there’s a scaling “law” without plotting any points or doing a projection. Without the mathiness, we could instead say that n
by skybrian 2y ago
It seems a little silly to pretend there’s a scaling “law” without plotting any points or doing a projection. Without the mathiness, we could instead say that new models keep getting better and we don’t know how long that trend will continue.
- ctoth 2y ago> It seems a little silly to pretend there’s a scaling “law” without plotting any points or doing a projection. Isn't this Kaplan 2020 or Hoffmann 2022?
- skybrian 2y agoYes, those are scaling laws, but when we see vendors improving their models without increasing model size or training longer, they don't apply. There are apparently other ways to improve performance and we don't know the laws for those. (Sometimes people track the learning curve for an industry in other ways, though.)
- ford 2y ago"Law" might not be the right word - but there's no denying it's scaling with compute/data/model size. I suppose law happens after continued evidence over years.