4 ms·
This is somewhat well covered in prior research, in so far as the information required is available. Here are some recent evaluations of the carbon footprint of
by soraki_soladead 5y ago
This is somewhat well covered in prior research, in so far as the information required is available. Here are some recent evaluations of the carbon footprint of modern models:
https://arxiv.org/abs/1906.02243 https://arxiv.org/abs/1906.02243 (cited in the above paper)
> The U.S. Environmental Protection Agency (EPA)
provides average CO2 produced (in pounds per
kilowatt-hour) for power consumed in the U.S.
(EPA, 2018), which we use to convert power to
estimated CO2 emissions:
> CO2e = 0.954 pt
> This conversion takes into account the relative proportions of different energy sources (primarily natural gas, coal, nuclear and renewable) consumed
to produce energy in the United States.
(Other countries are also included in the paper.)
https://arxiv.org/abs/2104.10350 https://arxiv.org/abs/2104.10350
The authors note that in many cases accurately estimating the carbon footprint is difficult because the information required is not publicly or readily available. However, they do provide some additional data, improved calculations, as well as motivations beyond CO2 reduction.
- justicezyx 5y agoCannot you read the parent? Because of the opaqueness of the electrical infrastructure, it's not precise of these carbon fiber footprint measurements, because the data is simply not there. Therefore, a better measurement is just the electric usage...
- soraki_soladead 5y agoThe papers I provided go a bit beyond "electrical usage". One of them is cited by the paper in question. Yes, they are approximations in lieu of more accurate data but that doesn't invalidate them as a tool or a motivation for future work such as this one. Consider further that it's not just OpenAI's models in question: it's every practitioner who attempts to train similarly large models. These practitioners may not be using "green" data centers, even if we generously assume that OpenAI does. (Microsoft's 100% renewable target for data centers isn't until 2025. Read another way: they may be trying but they're not there yet.) The available data and approximations illustrate that it is not accurate to assume that the average data center are powered by 100% renewables and carbon neutral. Thus the only reasonable conclusion is that more efficient models will have a positive impact on CO2 which is the motivation of the paper. Even if you don't agree, it's not completely unfounded and is based on at least some research and data. At the end of the day, is this really worth fighting against? Who wants less energy efficient models?
- justicezyx 5y agoIndeed human cannot read... Not sure what you are rehashing the same content in these words...
- soraki_soladead 5y agoI have provided recent and relevant citations with data and detailed comments. You have provided? What? Attacks? I'm honestly not even sure. Given your comment history I think I'm done here.
- justicezyx 5y agoHmm, I meant that we are just talking about the same thing...
- ChefboyOG 5y agoThere are also some pretty cool open source projects dedicated to tracking this kind of thing: https://www.comet.ml/site/introducing-codecarbon-an-open-source-tool-to-help-track-the-co2-emissions-of-your-research/ https://www.comet.ml/site/introducing-codecarbon-an-open-sou...
- eterevsky 5y agoThe abstract of the first paper says "Remarkably, the choice of DNN, datacenter, and processor can reduce the carbon footprint up to ~100-1000X." If your aim is to optimize CO2 emissions you have a lot of variable at your disposal and the architecture of the network is just one of them. If the paper from the post indeed tried to evaluate and compare various kinds of optimization, then citing the CO2 emissions would be valid. But since it only discusses the improvements in the model itself, then it would be much more productive to just point to the reductions in the required memory, GPU/TPU-time etc. The readers can do the carbon math themselves depending on how carbon-neutral is their datacenter.