Also, Llama 2 is still a few percentage points below GPT-4. Which is not close, because performance is logarithmic in training compute. Each additional percent

Also, Llama 2 is still a few percentage points below GPT-4. Which is not close, because performance is logarithmic in training compute. Each additional percentage point of performance requires exponentially greater investment in compute during pretraining. Llama 2 was pretrained on 2 trillion tokens -- a significant investment in compute, for sure, but still not enough to get close to GPT-4.