4 ms·
From Page 7: > In the case of GPT-2 training, random initialization requires 64×10^9 tokens to reach a perplexity of 12, while weight subcloning accomplishes t
by DarmokJalad1701 3y ago
From Page 7:
> In the case of GPT-2 training, random initialization requires 64×10^9 tokens to reach a perplexity of 12, while weight subcloning accomplishes this in just 64×10^9 tokens, again demonstrating a 4× training speedup
... what?
- mirekrusin 3y agoLooks like typo, from chart it seems to be around 20x10^9 for weight subcloning (doesn't look like 16x10^9 <<which would be 4x>> either).