Not even close. You can look at the Llama v2 training curves and its clear that even 7B models are still under-trained and they themselves have only been traine

Not even close. You can look at the Llama v2 training curves and its clear that even 7B models are still under-trained and they themselves have only been trained on 2T tokens. A 70B or whatever size/model amalgamation GPT-4 is also likely under-trained. Apart from just plain under-training, there's that whole 'grokking' phenomenon that's been observed in smaller models, where it looks like they're not really improving for a long time, but _eventually_ suddenly undergo a massive improvement. I don't know if anyone has been willing to set enough cash on fire to see if grokking can happen in a large LLM and/or how long it would need to be trained for. There's also still a lot of juice left in improving the quality of datasets and in training on larger data sets. OpenAI, having built GPT-4, probably want to work on some applications of it given that they have enough of a lead on everyone else. I think there's definitely a few "burn piles of cash to train a better model" buttons at OpenAI right now, but they have no reason to use them when they're clearly in the lead anyway. There's also the actual limiting factor: there are only so many A100s/H100s in the world, but the amount is growing.