6 ms·

> While energy usage has not been disclosed, it’s estimated that GPT-3 consumed 936 MWh. I'm a huge fan of energy efficieny, but this figure isn't all that muc

> While energy usage has not been disclosed, it’s estimated that GPT-3 consumed 936 MWh. I'm a huge fan of energy efficieny, but this figure isn't all that much. Let's put things into perspective using some (probably somewhat inaccurate but probably somewhere in the ballpark) random internet sources, showing that this is less energy than a single long haul Boeing 747 flight. (CO2 footprint is probably different but somewhere in the ballpark.) https://science.howstuffworks.com/transport/flight/modern/question192.htm https://science.howstuffworks.com/transport/flight/modern/qu... > A plane like a Boeing 747 uses approximately 1 gallon of fuel (about 4 liters) every second. Over the course of a 10-hour flight, it might burn 36,000 gallons (150,000 liters). According to Boeing's Web site, the 747 burns approximately 5 gallons of fuel per mile (12 liters per kilometer). (I didn't find the right type of gallon where 36000 gallons == 150000 liters, but let's go with the liter figure anyway.) According to https://en.wikipedia.org/wiki/Energy_density https://en.wikipedia.org/wiki/Energy_density, 1 liter of kerosene has an energy content of 35 MJ. 150000 liters of kerosene have an energy content of 5250000 MJ. 5250000 MJ is 1458.333333 MWh. Add another dubious source just to check our calculations (this one uses a larger fuel tank, but we'll re-use the previous figure): https://www.withouthotair.com/c5/page_35.shtml https://www.withouthotair.com/c5/page_35.shtml > And fuel’s calorific value is 10 kWh per litre. (We learned that in Chapter 3.) 150000 l * 10 kWh/l = 1500000 kWh = 1500 MWh, so about the same.

zasdffaa 4y ago
You've compared one use of energy to another and found them about the same. Fair enough, but that's an utterly different issue to whether it "isn't all that much", which you have not justified. Not an attack on your figures or your claims, just saying you've not addressed the point you set out to address.
8fingerlouie 4y ago
Not to mention that training a model is a "one time deal", where a successfully trained model can be reused by a lot of client (devices). Considering the way AI can potentially bring benefits to humanity, i see it more like an investment. For comparison, Bitcoin in 2021 used 110 TWh, solving a problem we've either solved millenea ago, or could be solved using much less power with premined coins.
- klysm 4y ago
  I’m not arguing at all that bitcoin is better in this capacity, but I am curious about the estimated MWh required to mine all the metal for all the coins.
  - 8fingerlouie 4y ago
    While mining the metal probably hasn't been cheap, it's not as much a recurring cost as the cost of proof-of-work. Besides, in my part of the world at least, i haven't carried cash in a decade or more, and most people i know don't either. Money is increasingly only a digital thing that exists in your (also digital) bank account. Even "micro payments" between individuals is being handled here, for a decade or so, by MobilePay[1] on the domestic side, and PayPal on the international side, with Apple Pay looking to be a close contestant to both, at least for the 55-60% of the population using iPhones. [1]: https://en.wikipedia.org/wiki/MobilePay https://en.wikipedia.org/wiki/MobilePay
    - leppr 4y ago
      How do you handle the loss of privacy caused by using only non-cryptocurrency digital payments? Any mitigation measures one can take, or is it an all-or nothing proposition to have everything you ever buy logged? Can you see what other citizens buy, or is this privilege reserved to a few bankers, tech companies and government institutions?
      - 8fingerlouie 4y ago
        Government institutions (the tax man mostly) are trying their best to limit the flow of anonymous transactions. Withdrawing a relatively large amount of cash (>=$1500) requires you to sign a form stating your intended purpose with the cash. Depositing a large amount of cash (>$750) requires documentation on how you came by this cash. Crypto is nowhere as anonymous as everybody appears to believe. Every single transaction is on the blockchain, and can be traced from acquisition to being spent. Considering that all European exchanges are now obliged to report their customer details to the authorities (under the KYC of AMLD-V EU directive), it becomes a relatively simple matter to trace your money to crypto transactions. Fortunately, Europe isn’t hit as hard with the customer profiling as the US is, and the EU is rather intent on protecting privacy, so your data purchase history is available only to youths relevant parties. In case of MobilePay, the service provider can of course see the details, but all my bank sees is a text I’ve input when sending money, or what the sender has input. It still gets reported to the tax authorities.
leoff 4y ago
Following your logic, a pencil weights 0.006kg, considering E=mc*2, this is 150,000MWh. This AI uses less energy than a pencil.
- ben_w 4y ago
  If we made pencils in the LHC, but we don’t so it isn’t.
  - leoff 4y ago
    That's my point, you can't simply compare the energies like this.
    - ben_w 4y ago
      Your point is incorrect. The energy in the fuel used to fly a plane can also be used to power a gas turbine electrical generator and thereby a computer. They are, at least in principle, fungible.
      - dekhn 4y ago
        Their energy consumptions are fungible, but that doesn't mean the content or value of any particular training run or international flight is comparable. Some flights run totally empty (airlines need to keep their precious routes) which uses a marginally smaller amount of fuel than a full plane, but provides no real value (nobody was transported to a location they wanted to be in). Other flights are normal, but also carry an individual of great importance, or play a strategic role. Some training runs waste a bunch of power and then get thrown away. Others go on to serve users and make billions of dollars. The energy cost is only a proxy for the fully loaded "value" of what you create with your energy consumption. That's why it's not comparable.
adamsmith143 4y ago
>>> While energy usage has not been disclosed, it’s estimated that GPT-3 consumed 936 MWh. How was this estimated? Is that for the final model run, how about all the testing runs, funs that failed, parameter tuning runs etc. are those included?
sdenton4 4y ago
Many of the big players in ML also go out of their way to ensure their daughters are using renewable energy sources. The CO2 footprint thus ends up much lower than the airplane flight.

zasdffaa 4y ago

You've compared one use of energy to another and found them about the same. Fair enough, but that's an utterly different issue to whether it "isn't all that much", which you have not justified. Not an attack on your figures or your claims, just saying you've not addressed the point you set out to address.

8fingerlouie 4y ago

Not to mention that training a model is a "one time deal", where a successfully trained model can be reused by a lot of client (devices). Considering the way AI can potentially bring benefits to humanity, i see it more like an investment. For comparison, Bitcoin in 2021 used 110 TWh, solving a problem we've either solved millenea ago, or could be solved using much less power with premined coins.

klysm 4y ago
I’m not arguing at all that bitcoin is better in this capacity, but I am curious about the estimated MWh required to mine all the metal for all the coins.
- 8fingerlouie 4y ago
  While mining the metal probably hasn't been cheap, it's not as much a recurring cost as the cost of proof-of-work. Besides, in my part of the world at least, i haven't carried cash in a decade or more, and most people i know don't either. Money is increasingly only a digital thing that exists in your (also digital) bank account. Even "micro payments" between individuals is being handled here, for a decade or so, by MobilePay[1] on the domestic side, and PayPal on the international side, with Apple Pay looking to be a close contestant to both, at least for the 55-60% of the population using iPhones. [1]: https://en.wikipedia.org/wiki/MobilePay https://en.wikipedia.org/wiki/MobilePay
  - leppr 4y ago
    How do you handle the loss of privacy caused by using only non-cryptocurrency digital payments? Any mitigation measures one can take, or is it an all-or nothing proposition to have everything you ever buy logged? Can you see what other citizens buy, or is this privilege reserved to a few bankers, tech companies and government institutions?
    - 8fingerlouie 4y ago
      Government institutions (the tax man mostly) are trying their best to limit the flow of anonymous transactions. Withdrawing a relatively large amount of cash (>=$1500) requires you to sign a form stating your intended purpose with the cash. Depositing a large amount of cash (>$750) requires documentation on how you came by this cash. Crypto is nowhere as anonymous as everybody appears to believe. Every single transaction is on the blockchain, and can be traced from acquisition to being spent. Considering that all European exchanges are now obliged to report their customer details to the authorities (under the KYC of AMLD-V EU directive), it becomes a relatively simple matter to trace your money to crypto transactions. Fortunately, Europe isn’t hit as hard with the customer profiling as the US is, and the EU is rather intent on protecting privacy, so your data purchase history is available only to youths relevant parties. In case of MobilePay, the service provider can of course see the details, but all my bank sees is a text I’ve input when sending money, or what the sender has input. It still gets reported to the tax authorities.

leoff 4y ago

Following your logic, a pencil weights 0.006kg, considering E=mc*2, this is 150,000MWh. This AI uses less energy than a pencil.

ben_w 4y ago
If we made pencils in the LHC, but we don’t so it isn’t.
- leoff 4y ago
  That's my point, you can't simply compare the energies like this.
  - ben_w 4y ago
    Your point is incorrect. The energy in the fuel used to fly a plane can also be used to power a gas turbine electrical generator and thereby a computer. They are, at least in principle, fungible.
    - dekhn 4y ago
      Their energy consumptions are fungible, but that doesn't mean the content or value of any particular training run or international flight is comparable. Some flights run totally empty (airlines need to keep their precious routes) which uses a marginally smaller amount of fuel than a full plane, but provides no real value (nobody was transported to a location they wanted to be in). Other flights are normal, but also carry an individual of great importance, or play a strategic role. Some training runs waste a bunch of power and then get thrown away. Others go on to serve users and make billions of dollars. The energy cost is only a proxy for the fully loaded "value" of what you create with your energy consumption. That's why it's not comparable.

adamsmith143 4y ago

>>> While energy usage has not been disclosed, it’s estimated that GPT-3 consumed 936 MWh. How was this estimated? Is that for the final model run, how about all the testing runs, funs that failed, parameter tuning runs etc. are those included?

sdenton4 4y ago

Many of the big players in ML also go out of their way to ensure their daughters are using renewable energy sources. The CO2 footprint thus ends up much lower than the airplane flight.