7 ms·

The model seems to be "build something fast, get users, engagement, and venture capital, hope you can grow fast enough to still be around after the Great AI cul

The model seems to be "build something fast, get users, engagement, and venture capital, hope you can grow fast enough to still be around after the Great AI cull". > offers over 0.6 million different pretrained open models. One estimate I saw was that training GPT3 released 500 tons of CO2 back in 2020. Out of those 600k models, at least hundreds are of a comparable complexity. I can only hope building large models does not become analogous to cryptocoin speculation, where resources are forever burned only in a quest to attract the greater fool. Those startups and researchers would better invest in smarter algorithms and approaches instead of trying to outpolute OpenAI, Meta and Microsoft.

cs702 2y ago
> "build something fast, get users, engagement, and venture capital, hope you can grow fast enough to still be around after the Great AI cull" Snowflake is a publicly traded company with a market cap of $50B and $4B of cash in hand. It has no need for venture capital money. It looks like a case of "Look Ma! I can do it too!"
ReptileMan 2y ago
>One estimate I saw was that training GPT3 released 500 tons of CO2 back in 2020 So absolute nothing in the grand scheme of things?
- throwup238 2y ago
  Yeah that’s the annual emissions for only 100 people at the global average or about 30 Americans.
- margalabargala 2y ago
  That's the amount that would be released by burning 50,000 gallons of gas, which is about that ten typical cars will burn throughout their entire lifespan. Done once, I agree, that's very little. But if each of those 600,000 other models used that much (or even a tenth that much), then that now becomes impactful. Releasing 500 tons of CO2 600,000 times over would amount to about 1% of all human global annual emissions.
  - renewiltord 2y ago
    500 tons is like a few flights between SF and NYC dude. And those 600k models are mostly fine-tunes. If running your 4090 at home is too much then we're going to have to get rid of the gamers. This CO2 objection is such an innumerate objection. Just making 100 cars already is more than making one of these LLMs from scratch. A finetune is so cheap in comparison. In fact, I bet if you asked most LLM companies they'd gladly support a universal carbon tax with even dividend based on emissions and then you'd see who's actually emitting.
    - margalabargala 2y ago
      There are two groups here. One sees the high impact of the large model, and the growth of model training, and is concerned with how much that could increase in coming years. The other group assumes the first group is complaining about right now, and thinks they're being ridiculous. This whole thing reminds me of ten years ago when people were pointing out energy waste as a downside of bitcoin. "It's so little! Electricity prices will prevent it from ever becoming significant!" was the response that it was met with, just like people are saying in this thread. In 2023, crypto mining accounted for about 0.5% of humanity's electricity consumption. If AI model training follows a similar curve, then it's reasonable to be concerned.
      - mlyle 2y ago
        > If AI model training follows a similar curve, then it's reasonable to be concerned. Yes, but one can at least still imagine scenarios where AI training being 0.5% of electricity use could still be a net win. (I hope we're more efficient than that; but if we're training models that end up helping a little with humanity's great problems, using 1/200th of our electricity for it could be worth it).
        margalabargala 2y ago
        The current crop of generative AIs seems well-poised to take over a significant amount of low-skill human labor. It does not seem well-poised to yield novel advancements in unrelated-to-AI fields, yet. Possibly genetics. But things like solving global warming, there is not any sort of path towards that for anything we're currently creating. It's not clear to me that spending 0.5% of electricity generation to put a solid chunk of the lower-middle-class out of work is worth it.
oceanplexian 2y ago
Flights from the Western USA to Hawaii are ~2 million tons a year, at least in 2017, wouldn’t be surprised if that number doubled. 500t to train a model at least seems like a more productive use of carbon than spending a few days on the beach. So I don’t think the carbon use of training models is that extreme.
- cornholio 2y ago
  GPT3 was a 175 bln parameters model. All the big boys are now doing trillions of parameters without a substantial chip efficiency increase. So we are talking about thousands of tons of carbon per model, repeated every year or two or however fast they become obsolete. To that we need to add embedded carbon in the entire hardware stack and datacenter, it quickly adds up. If it's just a handfull of companies doing it, fine, it's negligible versus benefits. If it starts to chase the marginal cost of the resources in requires, so that every mid to large company feels that a few million $ or so spent training their a model on their own dataset makes them more in competitive advantages, then it quickly spirals out of control hence the cryptocoin analogy. That's exactly what many AI startups are proposing.
  - kaibee 2y ago
    AI models don’t care if the electricity comes from renewable sources. Renewables are cheaper than fossil fuels at this point and getting cheaper still. I feel a lot better about a world where we consume 10x the energy but it comes from renewables than one where we only consume 2x but the lack of demand limits investment in renewables.
    - shadowgovt 2y ago
      It's also a great load to support with renewables because you can always do training as "bulk operations" on the margins. Just do them when renewable supply is high and demand is low; that energy can't be stored and would have been wasted anyway.
      - 2y ago
bee_rider 2y ago
I wonder what is greater, the CO2 produced by training AI models, the CO2 produced by researchers flying around to talk about AI models, or the CO2 produced by private jets funded by AI investments.
- 01HNNWZ0MV43FF 2y ago
  Institute a carbon tax and I'm sure we'll find out soon enough
  - bee_rider 2y ago
    For sure; I didn’t realize sensible systemic reforms were on the table. I’m not sure if any of these things would be the first on the chopping block if a carbon tax were implemented, but it is worth a shot.
    - TeMPOraL 2y ago
      They're probably above the median on the scale of actually useful human activities; there's a lot of stuff carbon tax would eat first.
      - mlyle 2y ago
        Yup, but even for the useful stuff, a greater price of carbon-intensive energy would change some about how you consider doing it.
shrubble 2y ago
So less than Taylor Swift over 12-18 months, since she burned 138t in the last 3 months: https://www.newsweek.com/taylor-swift-coming-under-fire-co2-emissions-1854856 https://www.newsweek.com/taylor-swift-coming-under-fire-co2-...
EVa5I7bHFq9mnYK 2y ago
I've seen estimates that training gpt3 consumed 10GWh, while inference by its millions of users consumes 1GWh per day, so inference Co2 costs dwarf training costs.

cs702 2y ago

> "build something fast, get users, engagement, and venture capital, hope you can grow fast enough to still be around after the Great AI cull" Snowflake is a publicly traded company with a market cap of $50B and $4B of cash in hand. It has no need for venture capital money. It looks like a case of "Look Ma! I can do it too!"

ReptileMan 2y ago

>One estimate I saw was that training GPT3 released 500 tons of CO2 back in 2020 So absolute nothing in the grand scheme of things?

throwup238 2y ago
Yeah that’s the annual emissions for only 100 people at the global average or about 30 Americans.
margalabargala 2y ago
That's the amount that would be released by burning 50,000 gallons of gas, which is about that ten typical cars will burn throughout their entire lifespan. Done once, I agree, that's very little. But if each of those 600,000 other models used that much (or even a tenth that much), then that now becomes impactful. Releasing 500 tons of CO2 600,000 times over would amount to about 1% of all human global annual emissions.
- renewiltord 2y ago
  500 tons is like a few flights between SF and NYC dude. And those 600k models are mostly fine-tunes. If running your 4090 at home is too much then we're going to have to get rid of the gamers. This CO2 objection is such an innumerate objection. Just making 100 cars already is more than making one of these LLMs from scratch. A finetune is so cheap in comparison. In fact, I bet if you asked most LLM companies they'd gladly support a universal carbon tax with even dividend based on emissions and then you'd see who's actually emitting.
  - margalabargala 2y ago
    There are two groups here. One sees the high impact of the large model, and the growth of model training, and is concerned with how much that could increase in coming years. The other group assumes the first group is complaining about right now, and thinks they're being ridiculous. This whole thing reminds me of ten years ago when people were pointing out energy waste as a downside of bitcoin. "It's so little! Electricity prices will prevent it from ever becoming significant!" was the response that it was met with, just like people are saying in this thread. In 2023, crypto mining accounted for about 0.5% of humanity's electricity consumption. If AI model training follows a similar curve, then it's reasonable to be concerned.
    - mlyle 2y ago
      > If AI model training follows a similar curve, then it's reasonable to be concerned. Yes, but one can at least still imagine scenarios where AI training being 0.5% of electricity use could still be a net win. (I hope we're more efficient than that; but if we're training models that end up helping a little with humanity's great problems, using 1/200th of our electricity for it could be worth it).
      - margalabargala 2y ago
        The current crop of generative AIs seems well-poised to take over a significant amount of low-skill human labor. It does not seem well-poised to yield novel advancements in unrelated-to-AI fields, yet. Possibly genetics. But things like solving global warming, there is not any sort of path towards that for anything we're currently creating. It's not clear to me that spending 0.5% of electricity generation to put a solid chunk of the lower-middle-class out of work is worth it.

oceanplexian 2y ago

Flights from the Western USA to Hawaii are ~2 million tons a year, at least in 2017, wouldn’t be surprised if that number doubled. 500t to train a model at least seems like a more productive use of carbon than spending a few days on the beach. So I don’t think the carbon use of training models is that extreme.

cornholio 2y ago
GPT3 was a 175 bln parameters model. All the big boys are now doing trillions of parameters without a substantial chip efficiency increase. So we are talking about thousands of tons of carbon per model, repeated every year or two or however fast they become obsolete. To that we need to add embedded carbon in the entire hardware stack and datacenter, it quickly adds up. If it's just a handfull of companies doing it, fine, it's negligible versus benefits. If it starts to chase the marginal cost of the resources in requires, so that every mid to large company feels that a few million $ or so spent training their a model on their own dataset makes them more in competitive advantages, then it quickly spirals out of control hence the cryptocoin analogy. That's exactly what many AI startups are proposing.
- kaibee 2y ago
  AI models don’t care if the electricity comes from renewable sources. Renewables are cheaper than fossil fuels at this point and getting cheaper still. I feel a lot better about a world where we consume 10x the energy but it comes from renewables than one where we only consume 2x but the lack of demand limits investment in renewables.
  - shadowgovt 2y ago
    It's also a great load to support with renewables because you can always do training as "bulk operations" on the margins. Just do them when renewable supply is high and demand is low; that energy can't be stored and would have been wasted anyway.
    - 2y ago

bee_rider 2y ago

I wonder what is greater, the CO2 produced by training AI models, the CO2 produced by researchers flying around to talk about AI models, or the CO2 produced by private jets funded by AI investments.

01HNNWZ0MV43FF 2y ago
Institute a carbon tax and I'm sure we'll find out soon enough
- bee_rider 2y ago
  For sure; I didn’t realize sensible systemic reforms were on the table. I’m not sure if any of these things would be the first on the chopping block if a carbon tax were implemented, but it is worth a shot.
  - TeMPOraL 2y ago
    They're probably above the median on the scale of actually useful human activities; there's a lot of stuff carbon tax would eat first.
    - mlyle 2y ago
      Yup, but even for the useful stuff, a greater price of carbon-intensive energy would change some about how you consider doing it.

shrubble 2y ago

So less than Taylor Swift over 12-18 months, since she burned 138t in the last 3 months: https://www.newsweek.com/taylor-swift-coming-under-fire-co2-emissions-1854856 https://www.newsweek.com/taylor-swift-coming-under-fire-co2-...

EVa5I7bHFq9mnYK 2y ago

I've seen estimates that training gpt3 consumed 10GWh, while inference by its millions of users consumes 1GWh per day, so inference Co2 costs dwarf training costs.