2 ms·
It’s kind of funny to me that these organizations are naming FLOP thresholds and crowning MMLU as the relevant evaluation metric. Seems that several of them hav
by janalsncm 3y ago
It’s kind of funny to me that these organizations are naming FLOP thresholds and crowning MMLU as the relevant evaluation metric. Seems that several of them have copy-pasted similar thresholds. As compute becomes cheaper, these thresholds will become cheaper and cheaper. Perhaps we will look back on them as quaint and nearsighted.
- kolinko 3y agoWell, FLOP make sense, kind of - you can make computing as cheap as you want, but without algorithmic improvements the FLOP count will stay the same. And algorithmic improvements will probably be discrete - happening once in a while, not continuous. There is also a mathematical limit to how efficient we can make the training - we don’t know the limit yet, but there is a point below which we won’t get.