5 ms·
Is the author conflating the cost to train an AI model with the cost to use the resulting models, here? I can certainly believe that model training cost is exp
by rco8786 4y ago
Is the author conflating the cost to train an AI model with the cost to use the resulting models, here?
I can certainly believe that model training cost is exponential as the # of parameters goes exponential. But that is a one time(-ish) cost relative to actually using those models.
Or am I completely off base here?
- habibur 4y agoI pretty much always wanted to know the cost of running those models, not training those. That should be order of magnitude less.
- jackmott42 4y agostable diffusion will generate an image in ~30 seconds on an NVidia 3060, so the cost is on the order of cents per image, power use on the order of 170 watts during the generation, or 0.0014166666667 kwh. @ around 5 cents per kwh
- rileymat2 4y agoYou also have to calculate the life expectancy of the card against 30 seconds of its life as an ongoing cost.
- MagnumOpus 4y agoWhich is also on the order of 0.01c per query (answering 5 million queries in the lifetime of a $500 card).
- smulc 4y agoJust ask ChatGPT what the cost is? ;)
- npongratz 4y agoSure, but just remember that it can't do math, so any mathy answer it gives is just bullshit.
- fock 4y agoWell, the number of parameters is directly related to the size of the machine to keep the model around. And so while you can host GPT-2 on a single consumer GPU, for ChatGPT you allegedly need 4-8-10k-a-piece GPUs. Which also consume 8-16 the power of your consumer GPU. So if you continue that, the cost to society will be too high to bear for this kind of games. (and in my eyes already is, as the outputs of large generative models are 100% used to generate more stuff to consume for the same amount of already information-overloaded people)
- ryan93 4y agoA single international flight literally consumes a swimming pools worth of fuel. a few GPUs is nothing especially as we transition to wind and solar.
- fock 4y agowell, then stop flying and the useless content-creation and let's get back to our "few GPUs" for consumption, when living, food and flying have transitioned to wind and solar. Then people can start setting up their text-based Holodeck-precursor consuming 1kW during usage. Just be clear: the alternative to letting a future GPT-derivative write your novel is to go to the library and pick a book. That works now and has negligible emissions. A future GPT (currently you won't get a consistent long form text) might do the same - optimistically it will do it on only 2-3kWh of energy (which amounts to the cost of creating a physical book). Granted, you only can sustain a human on that for a day, but then the humans don't go away because of AI. Conclusion to me: the things are fascinating and so are H-bombs... Question: which kind of time do you expect people to spend with these models? Which human needs will they satisfy? How will their energy consumption then still be negligible?
- layer8 4y agoIn the long run, a model that isn’t regularly updated with new data won’t be very viable for many purposes. The (re)training cost therefore will have some relevance.
- ahoho 4y agoInference costs are non-trivial, and I wouldn’t be surprised if the cost of running ChatGPT (given the 3M/day figure) has surpassed that of training it. Without optimizations, training only uses ~3 times the memory as inference, so exponential parameter/cost scaling still affects both. There’s ongoing research to reduce the computational costs of inference, but to my knowledge they only offer linear improvements (although I wouldn’t bet against more substantial reductions in the near future, particularly as these techniques are compounded).
- gizmo 4y agoOpenAI has stated that their costs per query are a few cents. Not nothing, but nothing outrageous either. The cost will go down to < 1c within a couple of years I'm sure. The cost may be prohibitive to use LLMs for absolutely everything, but the universe of applications where companies can afford to spend 1c on an LLM answer is very large.
- smoldesu 4y ago> the universe of applications where companies can afford to spend 1c on an LLM answer is very large. I hear this a lot, but don't see it very often in practice. GPT and it's ilk are coming up on 3 years old now, and our most novel application thus far is the same textbox + response that Talk to Transformer had. That's sad. I've heard the sell before ("imagine AI spreadsheets!") but I don't think people are willing to pay for answers that are regularly wrong in the long run. If your bridge only works half the time, people probably won't be inclined to pay your toll anymore.