3 ms·
It said to be trained on 75 Billion tokens with 8 A100 in 7 days. However, in the paper they say that for each of the 5 epochs, they sample 512 tokens from eac
by zwaps 3y ago
It said to be trained on 75 Billion tokens with 8 A100 in 7 days.
However, in the paper they say that for each of the 5 epochs, they sample 512 tokens from each paper randomly.
This would come down to an effective training size of 12 billion tokens.
- zwaps 3y agoAlso, if I am not reading this wrong, the paper shows that when Fine-tuning on a task, this additional Medical training (7 days on A100) essentially does nothing (benefit of maybe 1% over standard LLaMA). Zero shot comparisons are not shown. This looks like a failed experiment to me?