4 ms·
Not really close, because performance is logarithmic in training compute. That is, each additional percentage point of performance requires exponentially great
by cs702 3y ago
Not really close, because performance is logarithmic in training compute.
That is, each additional percentage point of performance requires exponentially greater investment in compute during pretraining.
Llama 2 was pretrained on 2 trillion tokens -- a significant investment in compute, for sure, but still not enough to get close to GPT-4.
And this is only one benchmark.