"Surprisingly, their scaling law analysis still focuses on training FLOPs instead of training + inference FLOPs." It's kind of weird. In the conclusion, they s

"Surprisingly, their scaling law analysis still focuses on training FLOPs instead of training + inference FLOPs." It's kind of weird. In the conclusion, they say >With PaLM 2, we have independently verified the scaling laws from Hoffmann et al. (2022) at large scales; we have shown that training tokens should grow at roughly the same rate as the number of model parameters. then a few lines later >In effect, we find that it is generally more efficient to train a smaller model with more tokens, for a fixed inference and training budget. Without more architecture details, it's hard to tell what they're going on about.