3 ms·
It's not clear to me how you could spend a few trillion on training, unless you spread out the investment over a decade+. Where will all the hardware come from?
by IncreasePosts 3y ago
It's not clear to me how you could spend a few trillion on training, unless you spread out the investment over a decade+. Where will all the hardware come from? It's not like nvidia or whoever can just 100x their output next year.
- benreesman 3y agoI think that costs have scaled with N^2 on every version of GPT meriting a new major version number and that Altman’s public statements around seven trillion are almost exactly what the computer science would say it would cost to be on the wrong side of a polynomial at one more turn of the crank.
- HarHarVeryFunny 3y agoAltman's attempts to raise money for a new "$7T" chip consortium is exactly that - for a new chip consortium - this is NOT him trying to raise money for OpenAI's next training run (which is anyways already paid for and likely complete at this stage). I'm curious what "N" (what model measure is this?) you think spending is scaling in N^2 fashion with? And, what values of this N are you using for GPT-3, 4 and 5?