This is almost exactly half the tokens (or twice the model size) that the Chinchilla scaling laws would predict (in fact, Table 3 in the Chinchilla paper gives

This is almost exactly half the tokens (or twice the model size) that the Chinchilla scaling laws would predict (in fact, Table 3 in the Chinchilla paper gives 3.7T tokens, almost exactly this number, as optimal for a 175B parameter model). Some comments on https://news.ycombinator.com/item?id=35891444 https://news.ycombinator.com/item?id=35891444 back when it was mostly assumed Palm2 was Chinchilla optimal said it was surprising that they hadn't taken inference into account (lowering the size, training for longer, for almost the same training FLOPs but faster inference -- good blog on this here: https://www.harmdevries.com/post/model-size-vs-compute-overhead/ https://www.harmdevries.com/post/model-size-vs-compute-overh...). So what gives?