3 ms·
Well doesn’t the compute time for transformers scale roughly quadratically with model size? Would it make sense for power consumption to also scale roughly qua
by dartos 3y ago
Well doesn’t the compute time for transformers scale roughly quadratically with model size?
Would it make sense for power consumption to also scale roughly quadratically?
- tomjohnneill 3y agoI'm not sure. The figures I've seen suggest that GPT3 required 10x more energy to train than GPT2 (e.g. https://www.nnlabs.org/power-requirements-of-large-language-models/#:~:text=Power%20Consumption%20of%20Different%20Large%20Language%20Models&text=In%20comparison%2C%20GPT%2D3%2C,more%20energy%20than%20GPT%2D2 https://www.nnlabs.org/power-requirements-of-large-language-....), so I think a roughly 1-2 order of magnitude increase in energy usage from GPT2 to GPT3.5 makes sense.