4 ms·
"We know the number of floating point operations per token for inference is approximately twice the number of parameters" Does someone have a source for this?
by cubefox 4y ago
"We know the number of floating point operations per token for inference is approximately twice the number of parameters"
Does someone have a source for this?
(By the way, it is unknown how many parameters GPT-3.5 has, the foundation model which powers finetuned models like ChatGPT and text-davinci-003. GPT-3 had 175 billion parameters, but per the Hoffmann et al Chinchilla paper it wasn't trained compute efficiently, i.e. it had too many parameters relative to its amount of training data. It seems likely that GPT-3.5 was trained on more data with fewer parameters, similar to Chinchilla. GPT-3: 175B parameters, 300B tokens; Chinchilla: 70B parameters, 1.4T tokens.)
- vishal0123 4y agohttps://arxiv.org/pdf/2001.08361.pdf https://arxiv.org/pdf/2001.08361.pdf. See the C_forward formula approxiamtion.
- cubefox 4y agoThank you. Though it isn't quite clear to me whether the additive part is negligible?
- sytelus 4y agoFrom eq 2.2, additive part is usually in few 10s of millions. So, for N > 1B, approximation should be good but it doesn't work. For example, GPT3 inference flops is actually 3.4E+18 so the ratio is 19,000 not 2.
- vishal0123 4y agoFrom the paper > For contexts and models with d_model > n_ctx/12, the context-dependent computational cost per token is a relatively small fraction of the total compute. For GPT3, n_ctx is 4096 and d_model is 12228 >> 4096/12.