3 ms·
Carbon emissions arguments tend to ignore the value of what's being done as well. BERT and other transformers were meaningful experiments that were valuable in
by rococode 6y ago
Carbon emissions arguments tend to ignore the value of what's being done as well. BERT and other transformers were meaningful experiments that were valuable in furthering a major research direction and enabling more effective consumer and business applications. In that sense, it's like any other company doing R&D - of course energy will be used and of course there will be some inefficiencies.
I think it's quite misleading to compare the energy usage of an industry-wide research effort to individual consumption. The graphs look bad - "wow, 626,000 lbs! that's 284 metric tons of CO2! a plane flight is way less!" - but there's a fundamental difference between "progress on a problem being worked on by thousands of highly-paid researchers" and "I bought a car".
Meanwhile, the worst power plants are generating on the order of 10+ million tons of CO2 every year. There are at least a dozen of these in the US alone. Car factories are emitting hundreds of thousands of tons of CO2 (Tesla is somewhere around 150,000 tons a year, apparently, and it's designed to be efficient). Perhaps activism around CO2 emissions in ML training might be better focused on improving the efficiency of those things instead, seeing as a 1% improvement would outweigh the entirety of the NLP model training industry. It's certainly good to keep in mind the energy costs of training in case things balloon out of control, but right now the costs relative to the results seem small and not worth highlighting as some forgotten sin.
- 0thgen 6y agothis is a great comment, and shows why the original author’s “opportunity cost” argument is a double-edged sword
- refactor_master 6y agoStill though, it’s good that these numbers are brought into light, along with other “hidden” costs in the IT industry. Otherwise we’ll just spiral into whataboutism and “my own carbon footprint is totally fine, because somewhere out there a model is doing worse things”. It also goes to show the sheer scale of this research field (arguably in a double-edges sword way) if the general public was still thinking “nerds in a basement recognizing cats”.
- sontek 6y agoI think we all know that there is a carbon footprint cost to what we do. Its the reason google has been working on renewable energy datacenters for so long: https://www.google.com/about/datacenters/renewable/ https://www.google.com/about/datacenters/renewable/
- qsort 6y agoThis is exactly the problem I have with naive environmentalism. The most recent data I could find for the United States total carbon footprint was 5 billion (metric) tons in 2016. Total energy consumption of all computers, mobile phones, datacenters, servers etc, combined, isn't even a percentage point of that. Yes, CO2 emissions are a problem. You are not going to solve that problem by targeting sectors which entire footprint is not even a significant digit.
- labawi 6y agoI think the main was sidetracked. If we collectively emit as much CO₂ as 0.1 or 100 persons lifetime for a single model, every year .. who cares? I don't. But, what if Amazon wants it's own model with its own curation? Maybe we need different languages, maybe countries would like to have their own model with a different world-view. Why shouldn't a researcher train their own model, maybe experiment with different versions? Why should consumers be relegated to pre-trained model with inscrutable preconceptions?