4 ms·

>One estimate I saw was that training GPT3 released 500 tons of CO2 back in 2020 So absolute nothing in the grand scheme of things?

throwup238 3y ago
Yeah that’s the annual emissions for only 100 people at the global average or about 30 Americans.
margalabargala 3y ago
That's the amount that would be released by burning 50,000 gallons of gas, which is about that ten typical cars will burn throughout their entire lifespan. Done once, I agree, that's very little. But if each of those 600,000 other models used that much (or even a tenth that much), then that now becomes impactful. Releasing 500 tons of CO2 600,000 times over would amount to about 1% of all human global annual emissions.
- renewiltord 3y ago
  500 tons is like a few flights between SF and NYC dude. And those 600k models are mostly fine-tunes. If running your 4090 at home is too much then we're going to have to get rid of the gamers. This CO2 objection is such an innumerate objection. Just making 100 cars already is more than making one of these LLMs from scratch. A finetune is so cheap in comparison. In fact, I bet if you asked most LLM companies they'd gladly support a universal carbon tax with even dividend based on emissions and then you'd see who's actually emitting.
  - margalabargala 3y ago
    There are two groups here. One sees the high impact of the large model, and the growth of model training, and is concerned with how much that could increase in coming years. The other group assumes the first group is complaining about right now, and thinks they're being ridiculous. This whole thing reminds me of ten years ago when people were pointing out energy waste as a downside of bitcoin. "It's so little! Electricity prices will prevent it from ever becoming significant!" was the response that it was met with, just like people are saying in this thread. In 2023, crypto mining accounted for about 0.5% of humanity's electricity consumption. If AI model training follows a similar curve, then it's reasonable to be concerned.
    - mlyle 3y ago
      > If AI model training follows a similar curve, then it's reasonable to be concerned. Yes, but one can at least still imagine scenarios where AI training being 0.5% of electricity use could still be a net win. (I hope we're more efficient than that; but if we're training models that end up helping a little with humanity's great problems, using 1/200th of our electricity for it could be worth it).
      - margalabargala 3y ago
        The current crop of generative AIs seems well-poised to take over a significant amount of low-skill human labor. It does not seem well-poised to yield novel advancements in unrelated-to-AI fields, yet. Possibly genetics. But things like solving global warming, there is not any sort of path towards that for anything we're currently creating. It's not clear to me that spending 0.5% of electricity generation to put a solid chunk of the lower-middle-class out of work is worth it.
        mlyle 3y ago
        There was an important "if" there in what I said. That's why I didn't say that it was the case. Though, no matter what, LLMs are doing more useful work than looking for hash collisions. Can LLMs help us save energy? It doesn't seem to be such a ridiculous idea to me. And can they be an effort multiplier for others working on harder problems? Likely-- I am a high-skill worker and I routinely have lower-skill tasks that I can delegate to LLMs more easily than I could either do myself or delegate to other humans. (And, now and then, they're helpful for brainstorming in my chosen fields). I had a big manual to write communicating how to use something I've built. Giving GPT-4 some bulleted lists and a sample of my writing got about 2/3rds of it done. (I had to throw a fraction away, and make some small correctness edits). It took much less of my time than working with a doc writer usually does and probably yielded a better result. In turn, I'm back to my high-value tasks sooner. That is, LLMs may help attacking the great problems directly, or they may help us dedicate more effort to the great problems. (Or they may do nothing or may screw us all up in other ways).

throwup238 3y ago

Yeah that’s the annual emissions for only 100 people at the global average or about 30 Americans.

margalabargala 3y ago

That's the amount that would be released by burning 50,000 gallons of gas, which is about that ten typical cars will burn throughout their entire lifespan. Done once, I agree, that's very little. But if each of those 600,000 other models used that much (or even a tenth that much), then that now becomes impactful. Releasing 500 tons of CO2 600,000 times over would amount to about 1% of all human global annual emissions.

renewiltord 3y ago
500 tons is like a few flights between SF and NYC dude. And those 600k models are mostly fine-tunes. If running your 4090 at home is too much then we're going to have to get rid of the gamers. This CO2 objection is such an innumerate objection. Just making 100 cars already is more than making one of these LLMs from scratch. A finetune is so cheap in comparison. In fact, I bet if you asked most LLM companies they'd gladly support a universal carbon tax with even dividend based on emissions and then you'd see who's actually emitting.
- margalabargala 3y ago
  There are two groups here. One sees the high impact of the large model, and the growth of model training, and is concerned with how much that could increase in coming years. The other group assumes the first group is complaining about right now, and thinks they're being ridiculous. This whole thing reminds me of ten years ago when people were pointing out energy waste as a downside of bitcoin. "It's so little! Electricity prices will prevent it from ever becoming significant!" was the response that it was met with, just like people are saying in this thread. In 2023, crypto mining accounted for about 0.5% of humanity's electricity consumption. If AI model training follows a similar curve, then it's reasonable to be concerned.
  - mlyle 3y ago
    > If AI model training follows a similar curve, then it's reasonable to be concerned. Yes, but one can at least still imagine scenarios where AI training being 0.5% of electricity use could still be a net win. (I hope we're more efficient than that; but if we're training models that end up helping a little with humanity's great problems, using 1/200th of our electricity for it could be worth it).
    - margalabargala 3y ago
      The current crop of generative AIs seems well-poised to take over a significant amount of low-skill human labor. It does not seem well-poised to yield novel advancements in unrelated-to-AI fields, yet. Possibly genetics. But things like solving global warming, there is not any sort of path towards that for anything we're currently creating. It's not clear to me that spending 0.5% of electricity generation to put a solid chunk of the lower-middle-class out of work is worth it.
      - mlyle 3y ago
        There was an important "if" there in what I said. That's why I didn't say that it was the case. Though, no matter what, LLMs are doing more useful work than looking for hash collisions. Can LLMs help us save energy? It doesn't seem to be such a ridiculous idea to me. And can they be an effort multiplier for others working on harder problems? Likely-- I am a high-skill worker and I routinely have lower-skill tasks that I can delegate to LLMs more easily than I could either do myself or delegate to other humans. (And, now and then, they're helpful for brainstorming in my chosen fields). I had a big manual to write communicating how to use something I've built. Giving GPT-4 some bulleted lists and a sample of my writing got about 2/3rds of it done. (I had to throw a fraction away, and make some small correctness edits). It took much less of my time than working with a doc writer usually does and probably yielded a better result. In turn, I'm back to my high-value tasks sooner. That is, LLMs may help attacking the great problems directly, or they may help us dedicate more effort to the great problems. (Or they may do nothing or may screw us all up in other ways).