3 ms·

This is ultimately a silly and misguided proposal, which focuses on the costs and not benefits. There is nothing special about training an AI; it is simply anot

by gwern 7y ago

This is ultimately a silly and misguided proposal, which focuses on the costs and not benefits. There is nothing special about training an AI; it is simply another thing to spend resources on, no intrinsically worse or better than any other way. A biology experiment has 'carbon footprint'. The ITER has a 'carbon footprint' (and probably one orders of magnitude larger than all AI research this year). HN burns CO2. Everything uses energy or it uses things which require energy, like human labor, and they involve other costs like opportunity cost which are just as real and important. They stand or fall on their net merits, not how much electricity they use. There is no need to go around criticizing people for 'green AI' by myopically focusing solely on an abstract electrical cost of training. And if there is, then that applies to everything which uses electricity, and is better handled by putting a carbon tax on energy sources, and letting the market find the most unprofitable uses of energy (which will probably not be AI research, I'll tell you that...) and stop it and substitute in more 'green' power sources for everything else. More importantly, if you are concerned about the costs of training AI, you should be concerned about the total costs as compared to the total benefits, not slicing out a completely arbitrary subset of costs and ranting about how many 'cars' it is equivalent to (which is not even strictly true in the first place considering that many data centers are located near cheap and renewable power like hydropower or nuclear power plants!) and shrugging away the issue that people consider these performance gains important and well-worth paying for. There are costs to a model which is worse than it could be. There are costs to models which run slower at deployment time even if they are faster to train. There are costs to models which cannot be used for transfer learning (as the criticized language models excel at, incidentally). And so on. What matters are the total costs, and corporations and researchers already pay considerable attention to that already. (Not a single one of their metrics - 'carbon emission', 'electricity usage', 'elapsed real time', 'number of parameters', 'FPO' - is an actual total cost!)

6gvONxR4sf7o 7y ago
>This position paper advocates a practical solution by making efficiency an evaluation criterion for research alongside accuracy and related measures. In addition, we propose reporting the financial cost or "price tag" of developing, training, and running models to provide baselines for the investigation of increasingly efficient methods. [emphasis mine] They suggest reporting cost metrics next to benefit metrics so readers can judge the tradeoff for themselves. There's no proposal to ignore the benefits.
- gwern 7y ago
  And, as I already pointed out, none of their suggests cost metrics are even close to a total cost, and many of them are seriously misleading, and they use them in misleading rhetorical ways (as I noted about their bogus estimates of CO2 emissions). Everything about this paper, from the name 'green AI' on down, is framed to put sole attention on a narrow arbitrary section of costs of political relevance, at the cost of a holistic evaluation. A single vague clause about 'other measures' doesn't change that, no matter how you italicize it. Motte-and-bailey.
tlb 7y ago
It is useful, though, to compare results of projects with different budgets. There are papers tackling the same problem with 4 orders of magnitude difference in computing. Including an objective metric in the results would be worthwhile. In this case, the metric I want is either TFLOPs (for a commercially available architecture) or kWh (for some in-house machine) rather than tons of CO2, which depends on their power source.
dxbydt 7y ago
> is ultimately a silly and misguided proposal No its not. Its analogous to quantifying an F test with watts. Nothing wrong with that. I can give a billion row dataset to a dozen people. John assumes normality, samples 100 rows, runs a linear regression, gets 70% R^2, under 1 second on a 1980s era computer. Mary doesn't assume normality, runs a glm via irwls, gets better explanatory power than John, still on 1980s computer, though she takes 5 seconds instead of 1. Then James comes along & runs a decision tree & is twice as good as Mary, but he now needs a second on a 1990s PC. Tony uses a random forest & Baker wants a 128 layer neural net. And so on & so forth, until we end up with burning enough energy that would otherwise power a village, to overfit some dataset that excel at some completely artificial leaderboard metric. At some point a grown up comes in & says you don't need to light up your bedroom with industrial stadium lighting if you need to do your homework, a 40w table lamp will suffice. Hell half the third world uses a candle & they are performing quite well, if not better. That's really all this is. The marginal gains in performance aren't justified compared to the amount of energy you end up expending. Half these models are brittle & have zero shelf life, ppl regularly throw away stuff they wrote just a year ago, so to what raison d'etre this pursuit ?
- JimmyAustin 7y ago
  The cure isn't proscribing "green ai", it's adding a carbon tax that embeds the cost of the carbon externality into the cost of power in general, then letting the free market make the decision.
- mantoto 7y ago
  Money is optimizing this already as the highest invest is computing resources.
- speedplane 7y ago
  > The marginal gains in performance aren't justified compared to the amount of energy you end up expending. Come on, we’ve been through this before. At least for the past 10 years if not much longer, performance improvements have not actually improved performance, they’ve improved developer productivity. If you create a language that’s half the speed, but developers can program twice as fast in it, it makes sense to use it. Processing power is much cheaper than a $150k+ developer. Also, being first to market can yield outsized gains on competitors.
m0zg 7y ago
I disagree with some of this. The main thrust of their paper is correct, IMO: cost of training (and inference) needs to be factored into the comparison when comparing models. You can't optimize what you don't measure, and currently many SOTA results have the training regimes which take a ton of compute and aren't really doable in a research context if you're not Google and can't just use racks full of idle TPUs to do your stuff for free in the trough of the diurnal cycle while customers aren't using them. Cost of training is currently ignored so much, some of the authors don't even mention it in their papers, yet it has a direct relationship with things like researcher productivity and reproducibility of results by mere mortals. I wouldn't call this "myopically focusing on the cost of training". I'd call this "acknowledging cost of training as a valid metric worthy of publishing and optimization". I hope everyone can agree with this. Once you start optimizing this in addition to your mAP or BLEU or top1 accuracy, the results could be multiplicative. I suspect this will ultimately come down to math: both having architectures which are amenable to efficient gradient propagation and having better optimizers.
- gwern 7y ago
  There is no reason given to think that anyone is ignoring the total cost of training and not factoring this in when comparing models. Do you think Google is funding all this research and just forgetting to consider the cost of training BERT? Do you think any researchers are ignoring the cost of renting GPUs or their university overhead? AI researchers talk all the time about costs! They're positively obsessed with it, in a way you don't see in any other area. (When was the last time you saw someone snicker about a CRISPR paper, 'yeah, nice results, but you know those reagents cost thousands of dollars, right?' Or covered in said paper?) It's quite a double standard. Someone wants to spend >$20 billion on ITER (which doesn't work), you never hear about it; someone sketches out that the most inefficient way to train a neural net model (using on-demand cloud GPUs) costs $1m, suddenly it's a huge deal, a boondoggle, researchers should be ashamed of themselves, it's a threat to research, and we need 'green AI'.
  - m0zg 7y ago
    I worked at Google for many years. Google has so much idle compute at any given moment, they tend not to care _at all_ about you using it. It's not a problem for a Google employee to just grab a few thousand cores on a whim and do whatever they want with those cores, and many people do use this freedom. No permission is required if your tasks are preemptible (which by design they mostly are, at Google). I'm not sure if the same applies to TPUs (I left before they became prevalent) but I'm sure for the right subset of people the right permissions could be procured. Most workloads follow the diurnal cycle, so at any given time anywhere between 20 and 80% of capacity is sitting idle. That's far more than you need to do most kinds of deep learning research. That's why you see work from them that would cost you $100K+ to replicate end-to-end (e.g. neural architecture search). At Google you get those resources for basically free, so it's not a huge issue to just spin up a rack full of TPUs. That's just how things are done over there. I don't get your opposition to introducing approximate exaflops spent in training as an additional metric by which to judge "goodness" of a model. It is well known that some models are much easier to train than others. It's also known that some optimizers converge faster than others. All this stuff directly affects reproducibility of research. We're getting to the point where some of the papers aren't reproducible unless you're at Google or DeepMind or OpenAI or Microsoft Research. I think you'll agree that this is pretty terrible. Between two models with approximately the same validation accuracy metrics and inference timings wouldn't you choose the one that's more economical to train? I know I would. Why aren't these metrics even mentioned most of the time? Beats me. Even better if the model requires less _data_ to train, but that seems to be less tractable than just estimating and surfacing the cost of training.
Barrin92 7y ago
> There is nothing special about training an AI; it is simply another thing to spend resources on If the goal is actual intelligence, as the 'I' in the name appears to suggest, then making resource consumption in the form of energy/compute a central part of the equation makes a fair deal of sense. It's arguably long overdue and might actually push research towards methods that actually do increase the capacity of agents to learn rather than just praying for faster chips and more data. The benefit here can be twofold in that more inclusivity increases the pace at which the field advances by including more researchers, but also just gives researchers an incentive to not treat compute/data as unlimited moving closer towards figuring out how the brain actually learns.

by gwern 7y ago

6gvONxR4sf7o 7y ago

>This position paper advocates a practical solution by making efficiency an evaluation criterion for research alongside accuracy and related measures. In addition, we propose reporting the financial cost or "price tag" of developing, training, and running models to provide baselines for the investigation of increasingly efficient methods. [emphasis mine] They suggest reporting cost metrics next to benefit metrics so readers can judge the tradeoff for themselves. There's no proposal to ignore the benefits.

gwern 7y ago
And, as I already pointed out, none of their suggests cost metrics are even close to a total cost, and many of them are seriously misleading, and they use them in misleading rhetorical ways (as I noted about their bogus estimates of CO2 emissions). Everything about this paper, from the name 'green AI' on down, is framed to put sole attention on a narrow arbitrary section of costs of political relevance, at the cost of a holistic evaluation. A single vague clause about 'other measures' doesn't change that, no matter how you italicize it. Motte-and-bailey.

tlb 7y ago

It is useful, though, to compare results of projects with different budgets. There are papers tackling the same problem with 4 orders of magnitude difference in computing. Including an objective metric in the results would be worthwhile. In this case, the metric I want is either TFLOPs (for a commercially available architecture) or kWh (for some in-house machine) rather than tons of CO2, which depends on their power source.

dxbydt 7y ago

> is ultimately a silly and misguided proposal No its not. Its analogous to quantifying an F test with watts. Nothing wrong with that. I can give a billion row dataset to a dozen people. John assumes normality, samples 100 rows, runs a linear regression, gets 70% R^2, under 1 second on a 1980s era computer. Mary doesn't assume normality, runs a glm via irwls, gets better explanatory power than John, still on 1980s computer, though she takes 5 seconds instead of 1. Then James comes along & runs a decision tree & is twice as good as Mary, but he now needs a second on a 1990s PC. Tony uses a random forest & Baker wants a 128 layer neural net. And so on & so forth, until we end up with burning enough energy that would otherwise power a village, to overfit some dataset that excel at some completely artificial leaderboard metric. At some point a grown up comes in & says you don't need to light up your bedroom with industrial stadium lighting if you need to do your homework, a 40w table lamp will suffice. Hell half the third world uses a candle & they are performing quite well, if not better. That's really all this is. The marginal gains in performance aren't justified compared to the amount of energy you end up expending. Half these models are brittle & have zero shelf life, ppl regularly throw away stuff they wrote just a year ago, so to what raison d'etre this pursuit ?

JimmyAustin 7y ago
The cure isn't proscribing "green ai", it's adding a carbon tax that embeds the cost of the carbon externality into the cost of power in general, then letting the free market make the decision.
mantoto 7y ago
Money is optimizing this already as the highest invest is computing resources.
speedplane 7y ago
> The marginal gains in performance aren't justified compared to the amount of energy you end up expending. Come on, we’ve been through this before. At least for the past 10 years if not much longer, performance improvements have not actually improved performance, they’ve improved developer productivity. If you create a language that’s half the speed, but developers can program twice as fast in it, it makes sense to use it. Processing power is much cheaper than a $150k+ developer. Also, being first to market can yield outsized gains on competitors.

m0zg 7y ago

I disagree with some of this. The main thrust of their paper is correct, IMO: cost of training (and inference) needs to be factored into the comparison when comparing models. You can't optimize what you don't measure, and currently many SOTA results have the training regimes which take a ton of compute and aren't really doable in a research context if you're not Google and can't just use racks full of idle TPUs to do your stuff for free in the trough of the diurnal cycle while customers aren't using them. Cost of training is currently ignored so much, some of the authors don't even mention it in their papers, yet it has a direct relationship with things like researcher productivity and reproducibility of results by mere mortals. I wouldn't call this "myopically focusing on the cost of training". I'd call this "acknowledging cost of training as a valid metric worthy of publishing and optimization". I hope everyone can agree with this. Once you start optimizing this in addition to your mAP or BLEU or top1 accuracy, the results could be multiplicative. I suspect this will ultimately come down to math: both having architectures which are amenable to efficient gradient propagation and having better optimizers.

gwern 7y ago
There is no reason given to think that anyone is ignoring the total cost of training and not factoring this in when comparing models. Do you think Google is funding all this research and just forgetting to consider the cost of training BERT? Do you think any researchers are ignoring the cost of renting GPUs or their university overhead? AI researchers talk all the time about costs! They're positively obsessed with it, in a way you don't see in any other area. (When was the last time you saw someone snicker about a CRISPR paper, 'yeah, nice results, but you know those reagents cost thousands of dollars, right?' Or covered in said paper?) It's quite a double standard. Someone wants to spend >$20 billion on ITER (which doesn't work), you never hear about it; someone sketches out that the most inefficient way to train a neural net model (using on-demand cloud GPUs) costs $1m, suddenly it's a huge deal, a boondoggle, researchers should be ashamed of themselves, it's a threat to research, and we need 'green AI'.
- m0zg 7y ago
  I worked at Google for many years. Google has so much idle compute at any given moment, they tend not to care _at all_ about you using it. It's not a problem for a Google employee to just grab a few thousand cores on a whim and do whatever they want with those cores, and many people do use this freedom. No permission is required if your tasks are preemptible (which by design they mostly are, at Google). I'm not sure if the same applies to TPUs (I left before they became prevalent) but I'm sure for the right subset of people the right permissions could be procured. Most workloads follow the diurnal cycle, so at any given time anywhere between 20 and 80% of capacity is sitting idle. That's far more than you need to do most kinds of deep learning research. That's why you see work from them that would cost you $100K+ to replicate end-to-end (e.g. neural architecture search). At Google you get those resources for basically free, so it's not a huge issue to just spin up a rack full of TPUs. That's just how things are done over there. I don't get your opposition to introducing approximate exaflops spent in training as an additional metric by which to judge "goodness" of a model. It is well known that some models are much easier to train than others. It's also known that some optimizers converge faster than others. All this stuff directly affects reproducibility of research. We're getting to the point where some of the papers aren't reproducible unless you're at Google or DeepMind or OpenAI or Microsoft Research. I think you'll agree that this is pretty terrible. Between two models with approximately the same validation accuracy metrics and inference timings wouldn't you choose the one that's more economical to train? I know I would. Why aren't these metrics even mentioned most of the time? Beats me. Even better if the model requires less _data_ to train, but that seems to be less tractable than just estimating and surfacing the cost of training.

Barrin92 7y ago

> There is nothing special about training an AI; it is simply another thing to spend resources on If the goal is actual intelligence, as the 'I' in the name appears to suggest, then making resource consumption in the form of energy/compute a central part of the equation makes a fair deal of sense. It's arguably long overdue and might actually push research towards methods that actually do increase the capacity of agents to learn rather than just praying for faster chips and more data. The benefit here can be twofold in that more inclusivity increases the pace at which the field advances by including more researchers, but also just gives researchers an incentive to not treat compute/data as unlimited moving closer towards figuring out how the brain actually learns.