2 ms·
> Are these prominent events? Some of these models are just trained a little longer or cooked with different data? >I think it would be better to list pivotable
by NitpickLawyer 2y ago
> Are these prominent events? Some of these models are just trained a little longer or cooked with different data?
>I think it would be better to list pivotable events, i.e. significant changes in architecture or approach, or a benchmark being surpassed. Otherwise this is just a list of models released by big tech companies irrespective of their importance.
I don't think it's that easy to dismiss changes in recipes, data and training regime, tbh. There have been plenty of "huh, that's strange" moments with LLMs even without major breakthroughs in the overall architecture. The case of one model being really strong at chess while others from the same family (larger to boot) aren't is such a "huh" moment for me. Or the new "reasoning" thing, where it's obvious the new models are better at some tasks than the previous gen. Training steps is another interesting one, where two teams - one fine-tuning on 100k+ samples and one fine-tuning on 1k samples w/ 15 epochs and getting similar performance is also a good example. Or the (extremely readable) simple paper coming out of meta fair ~6 mo ago where they found that repeating some (5-10%) of the samples leads to much better generalisation. All of these are "huh" worthy, and I don't think we thoroughly understand why they happen, or why they sometimes happen and not other times (i.e. w/ 20+T training sets on massively scaled LLMs like the purported gpt5).