Well, FLOP make sense, kind of - you can make computing as cheap as you want, but without algorithmic improvements the FLOP count will stay the same. And algori

Well, FLOP make sense, kind of - you can make computing as cheap as you want, but without algorithmic improvements the FLOP count will stay the same. And algorithmic improvements will probably be discrete - happening once in a while, not continuous. There is also a mathematical limit to how efficient we can make the training - we don’t know the limit yet, but there is a point below which we won’t get.