Yeah, you’re totally right. I actually wrote a follow up essay about that: https://finbarr.ca/llms-not-trained-enough/ https://finbarr.ca/llms-not-trained-enou

Yeah, you’re totally right. I actually wrote a follow up essay about that: https://finbarr.ca/llms-not-trained-enough/ https://finbarr.ca/llms-not-trained-enough/ I think the conversations were partly (largely?) a snapshot in time. I was talking to people in February/March, and all of this was much less thought through at the time. But you’re totally right. You want something like Llama, where you train a smaller model longer than Chinchilla would predict.