4 ms·

Am I reading this correctly: Training time was 4282407hrs. At, conservatively, 200w gpus, that's (4282407*200)/1_000_000_000 GWh ~= 1 GWh. At 10c/kWh that's $1

Am I reading this correctly: Training time was 4282407hrs. At, conservatively, 200w gpus, that's (4282407*200)/1_000_000_000 GWh ~= 1 GWh. At 10c/kWh that's $100,000 ? So if you have a single eqv GPU at home, it's 500yrs of training time and $100k in energy costs. Or, in practice, 3000 gpus for 2mo. The AI industry has to hope the world doesnt change fast enough for these models to be useless. EDIT: price is $100k

TaylorAlexander 2y ago
Thanks for the figures. I suppose with expenses like that, they will be motivated to research methods of updating models which have already been trained. Edit: I see the price was updated
hackerlight 2y ago
1 GWh is 1 million kWh, multiplied by $0.1 that should give $100k in energy costs?
- mjburgess 2y ago
  Yes, thanks. I had assumed I had been off by a factor somewhere. Yet, 100k seems small -- the total cost of production is in the 10mil+ range.
  - Etheryte 2y ago
    100k is small, but you only get away with 100k if you nail everything perfect the first time around — something that we all know does not really happen. I think compiling is a good parallel to training, imagine if compiling your whole software project cost 100k if you did it from scratch. Sure there's incremental builds etc, but the cost is steep no matter which way you look at it.
jsheard 2y ago
Numbers like these really don't bode well for the longer term prospects of open source models, I doubt the current strategy of waiting expectantly for a corporation to spoonfeed us yet another $100,000 model for free is going to work forever. That $100k is conservative too, it doesn't include the cost of buying/renting the hardware, or the compute time spent on experimental training runs, or the cost of data acquisition, labeling and cleaning, or the cost of RLHF fine-tuning.
- mikehollinger 2y ago
  > Numbers like these really don't bode well for the long-term prospects of open source models, I doubt the current strategy of waiting expectantly for a corporation to spoonfeed us yet another $100,000 model for free is going to work forever. I would add “in their current form” and agree. There’s three things that can change here: 1. Moore’s law: The worldwide economy is built around the steady progression of cheaper compute. Give it 36 months and your problem becomes a $25,000 problem. 2. Quantization and smaller models: There’ll likely become specializations of the various models (is this the beginning of the “Monolith vs Microservices” debate? 3. E2E Training isn’t for everyone: Finetunes and Alignment are more important than an end to end training run, IF we can coerce the behaviors we want into the models by finetuning them. That along with quantized models (imho) unlocked vision models which are now in the “plateau of producivity” in the gartner hype cycle compared to a few years ago. So as an example today, I can grab a backbone and pretrained weights for an object detector, and with relatively little data (from a few lines to a few 10’s of lines of code, and 50 to 500 images) and relatively little wall clock time and energy (say 5 to 15 minutes) on a PC, I can create a customized object detector that can detect -my- specific objects pretty well. I might need to revise it a few times, but it’ll work pretty well. Why would we not see the same sort of progression with transformer architectures? It hinges on someone creating the model weights for the “greater good,” or us figuring out how to do distributed training for open source in a “seti@home” style (long live the blockchain, anyone?).
  - jsheard 2y ago
    Yeah, there's no accounting for breakthroughs in training efficiency. I wouldn't count on Moores Law though, the amount of compute you can put into these problems is effectively unbounded so more efficient silicon just means those with money can train even bigger models. 3D rendering is a decent analogy, Moores Law has made it easy to render something comparable to the first Toy Story movie, but Pixar poured those gains back into more compute and is using it to do things you definitely can't afford to.
- philjohn 2y ago
  I wonder if a kind of Seti@Home approach could work - although I'm guessing the limited VRAM in most consumer cards compared to an H100, as well as the much slower "virtual WAN interconnect" versus the mellanox goodies that nVidia clusters enjoy would be too big an obstacle?
mmoskal 2y ago
Assuming $30k GPU with 3yr deprecation, it's additionally $1.14/h. Much more than energy.
Grimblewald 2y ago
Things like petals (https://github.com/bigscience-workshop/petals https://github.com/bigscience-workshop/petals) exist, distributed computing over willing participants. Right now corporate cash is being rammed into the space so why not snap it up while you can, but the moment it dries up projects like petals will see more of the love they deserve. I envision a future where crypto-style booms happen over tokens useful for purchasing priority computational time, which is earned by providing said computational time. This way researchers can daisy-chain their independent smaller rigs together into something with gargantuan capabilities.

TaylorAlexander 2y ago

Thanks for the figures. I suppose with expenses like that, they will be motivated to research methods of updating models which have already been trained. Edit: I see the price was updated

hackerlight 2y ago

1 GWh is 1 million kWh, multiplied by $0.1 that should give $100k in energy costs?

mjburgess 2y ago
Yes, thanks. I had assumed I had been off by a factor somewhere. Yet, 100k seems small -- the total cost of production is in the 10mil+ range.
- Etheryte 2y ago
  100k is small, but you only get away with 100k if you nail everything perfect the first time around — something that we all know does not really happen. I think compiling is a good parallel to training, imagine if compiling your whole software project cost 100k if you did it from scratch. Sure there's incremental builds etc, but the cost is steep no matter which way you look at it.

jsheard 2y ago

Numbers like these really don't bode well for the longer term prospects of open source models, I doubt the current strategy of waiting expectantly for a corporation to spoonfeed us yet another $100,000 model for free is going to work forever. That $100k is conservative too, it doesn't include the cost of buying/renting the hardware, or the compute time spent on experimental training runs, or the cost of data acquisition, labeling and cleaning, or the cost of RLHF fine-tuning.

mikehollinger 2y ago
> Numbers like these really don't bode well for the long-term prospects of open source models, I doubt the current strategy of waiting expectantly for a corporation to spoonfeed us yet another $100,000 model for free is going to work forever. I would add “in their current form” and agree. There’s three things that can change here: 1. Moore’s law: The worldwide economy is built around the steady progression of cheaper compute. Give it 36 months and your problem becomes a $25,000 problem. 2. Quantization and smaller models: There’ll likely become specializations of the various models (is this the beginning of the “Monolith vs Microservices” debate? 3. E2E Training isn’t for everyone: Finetunes and Alignment are more important than an end to end training run, IF we can coerce the behaviors we want into the models by finetuning them. That along with quantized models (imho) unlocked vision models which are now in the “plateau of producivity” in the gartner hype cycle compared to a few years ago. So as an example today, I can grab a backbone and pretrained weights for an object detector, and with relatively little data (from a few lines to a few 10’s of lines of code, and 50 to 500 images) and relatively little wall clock time and energy (say 5 to 15 minutes) on a PC, I can create a customized object detector that can detect -my- specific objects pretty well. I might need to revise it a few times, but it’ll work pretty well. Why would we not see the same sort of progression with transformer architectures? It hinges on someone creating the model weights for the “greater good,” or us figuring out how to do distributed training for open source in a “seti@home” style (long live the blockchain, anyone?).
- jsheard 2y ago
  Yeah, there's no accounting for breakthroughs in training efficiency. I wouldn't count on Moores Law though, the amount of compute you can put into these problems is effectively unbounded so more efficient silicon just means those with money can train even bigger models. 3D rendering is a decent analogy, Moores Law has made it easy to render something comparable to the first Toy Story movie, but Pixar poured those gains back into more compute and is using it to do things you definitely can't afford to.
philjohn 2y ago
I wonder if a kind of Seti@Home approach could work - although I'm guessing the limited VRAM in most consumer cards compared to an H100, as well as the much slower "virtual WAN interconnect" versus the mellanox goodies that nVidia clusters enjoy would be too big an obstacle?

mmoskal 2y ago

Assuming $30k GPU with 3yr deprecation, it's additionally $1.14/h. Much more than energy.

Grimblewald 2y ago

Things like petals (https://github.com/bigscience-workshop/petals https://github.com/bigscience-workshop/petals) exist, distributed computing over willing participants. Right now corporate cash is being rammed into the space so why not snap it up while you can, but the moment it dries up projects like petals will see more of the love they deserve. I envision a future where crypto-style booms happen over tokens useful for purchasing priority computational time, which is earned by providing said computational time. This way researchers can daisy-chain their independent smaller rigs together into something with gargantuan capabilities.