3 ms·
> C = 6 * 1.4T * 7B = 58.8T FLOP-seconds. Sounds off by a few orders of magnitude since a RTX3090 can smash through about 78TFLOP/s. I trained nanoGPT on a re
by quickthrower2 3y ago
> C = 6 * 1.4T * 7B = 58.8T FLOP-seconds.
Sounds off by a few orders of magnitude since a RTX3090 can smash through about 78TFLOP/s.
I trained nanoGPT on a rented RTX3090 which is a much tinier model and takes about 12 hours on 100k steps (I need to check the params again but maybe 10M tokens tops)
And the reason is it is not a single flop per parameter per token. You have a deep architecture, compute intensive attention heads and back propagations to do as well.