Zuck mentioned they also cut entire classes of data from training Llama 2 that they included in Llama 3 as Llama 2 was intended to be used for social / meta rel

Zuck mentioned they also cut entire classes of data from training Llama 2 that they included in Llama 3 as Llama 2 was intended to be used for social / meta related services. So things like code were almost entirely skipped. He only mentioned code, but wouldn't be surprised if math etc was treated similarly. My assumption here is adding these missing subject matter areas had a larger impact than raw token counts.