Depends on your goal, if it's to overtake OpenAI as having the best model overall it makes sense to optimize for training loss alone (assuming a fixed upfront c

Depends on your goal, if it's to overtake OpenAI as having the best model overall it makes sense to optimize for training loss alone (assuming a fixed upfront compute budget). Optimizing for inference to achieve the same loss would require more compute overall so you're either paying upfront with higher training costs or kicking the can down the road to inference. News articles estimates of GPT4 cost seem to peg it at ~8 months of inference to achieve 1:1 cost with training. Life span of these models is TBD but it's a pretty safe bet we'll have new ones by then. Of course GPT3.5 is still getting used but probably won't cross 2:1ish in its lifetime. Might as well roll the dice and kick the can down the road if you're Google, I imagine they would happily pay an extra 500k/day in inference compute to be market leaders, whats 183mill for them? But if they don't get any real market share or the model sucks they saved substantially on training. > It makes me think they are Chinchilla-optimal, They elaborate in the appendix but they empirically determine PaLM-optimal, which concurs with Chinchilla-optimal (more or less).