4 ms·
If you take Llama-3-400B, and 30x its data (hitting the data ceiling, AFAICT), 30x its size to match, and the hardware improves by, say, 3x, then you'll use up
by trott 2y ago
If you take Llama-3-400B, and 30x its data (hitting the data ceiling, AFAICT), 30x its size to match, and the hardware improves by, say, 3x, then you'll use up about a year's worth of energy from a typical nuclear power plant.
- mathsmath 2y agoI don’t know much about LLMs, but is it possible to throttle their training? Solar has gotten pretty cheap, and I’m just wondering if you can throttle up and down based on how much output the panels are producing.
- moi2388 2y agoOf course it is, but the trade-off is time.
- spiralk 2y agoIf its for training a new foundation model it is not that bad. It's still only a fraction of the energy compared to many human industries. I did rough math some time ago and found that that training llama-3-70B used the equivalent energy to 1/30 of a full loaded container ship going from China to the US. Even scaled up 100x and trained 10x longer, its seems like the energy consumption is relatively small compared to other industries. The fact that people are considering nuclear power for AI training is an advantage not a downside, imo. It should have a much lower CO2 footprint.
- adrianN 2y agoYou always have to compare the cost to the value it generates. A year of power from a nuclear plant might be used in more productive ways.
- ofrzeta 2y agoYes, and consider that with the current hype around AI and enough (venture) capital there will be several corporations competing for the best AI and suddenly we are at several "nuclear plants" or equivalent energy "consumption".
- spiralk 2y agoI don't see increasing demand for nuclear power as a disadvantage. We have nuclear material that can last humanity 1000s of years at least. The CO2 footprint is an issue but nuclear is much better than others. Personally, I think it's better we utilize more energy and discover new breakthroughs while society is relatively stable and functioning, because there's no guarantee that it will last. Population collapse seems imminent in more educated societies, even China and India are trending this way now. Without some level of AI assistance, humanity would likely lose a great deal of productive output. Also, if this path to AGI does not work out, its not as though the nuclear reactors will be wasted. People will find something else to do with the energy.
- spiralk 2y agoSure I agree, but if we compared value it generates per unit energy it would still probably be better than many non-essential industries: the entertainment industry, fashion industry, alcohol, etc. Even in the current state LLMs can provide more useful practical value compared to industries with higher energy and CO2 footprints.
- monero-xmr 2y agoIf someone is willing to pay, who cares? Energy has a price. Focus on regulating how energy is generated, and when prices climb the market will solve the problem. If instead you focus on using the government to outlaw demand, only failure will follow. I mean, didn't the government outlawing the demand for illegal drugs fail miserably? I believe drugs are cheaper, more potent, and more available than ever. Similarly, if there is demand for compute, then compute will occur. There is always a clearing price commiserate with the risk.
- amanaplanacanal 2y agoA carbon tax would be the most free market way to do it: tax fossil carbon as it comes out of the ground. The market can handle the rest. Can’t seem to make that happen politically, though.
- DrNosferatu 2y agoCareful with carbon tax plans, they can be regressive: https://blogs.worldbank.org/en/energy/what-carbon-tax-can-do-and-why-it-cannot-do-it-all https://blogs.worldbank.org/en/energy/what-carbon-tax-can-do...