My understanding and suspicion is mostly less than you think. Llama 3 architecture has the following changes on GPT-2: 1. delete the absolute positional encodi

My understanding and suspicion is mostly less than you think. Llama 3 architecture has the following changes on GPT-2: 1. delete the absolute positional encoding and replace with RoPE 2. delete all biases in all layers (in LayerNorms, they turn into RMSNorm) 3. GeLU -> SwiGLU non-linearity in the MLP 4. longer context length 5. architecture hyperparameter changes, e.g. slightly different aspect ratios And there was a paper that I can't find the reference to anymore that claimed that if you train long enough, the gap becomes even lower. Possibly because the absolutely positional encoding has enough time to train more fully, where as the RoPE layer benefits from the "inductive bias" it adds in the earlier stages of training. But I don't have full confidence on the above claim, maybe someone has tried or has better/concrete reference.