3 ms·

There is no reason given to think that anyone is ignoring the total cost of training and not factoring this in when comparing models. Do you think Google is fu

by gwern 7y ago

There is no reason given to think that anyone is ignoring the total cost of training and not factoring this in when comparing models. Do you think Google is funding all this research and just forgetting to consider the cost of training BERT? Do you think any researchers are ignoring the cost of renting GPUs or their university overhead? AI researchers talk all the time about costs! They're positively obsessed with it, in a way you don't see in any other area. (When was the last time you saw someone snicker about a CRISPR paper, 'yeah, nice results, but you know those reagents cost thousands of dollars, right?' Or covered in said paper?) It's quite a double standard. Someone wants to spend >$20 billion on ITER (which doesn't work), you never hear about it; someone sketches out that the most inefficient way to train a neural net model (using on-demand cloud GPUs) costs $1m, suddenly it's a huge deal, a boondoggle, researchers should be ashamed of themselves, it's a threat to research, and we need 'green AI'.

m0zg 7y ago
I worked at Google for many years. Google has so much idle compute at any given moment, they tend not to care _at all_ about you using it. It's not a problem for a Google employee to just grab a few thousand cores on a whim and do whatever they want with those cores, and many people do use this freedom. No permission is required if your tasks are preemptible (which by design they mostly are, at Google). I'm not sure if the same applies to TPUs (I left before they became prevalent) but I'm sure for the right subset of people the right permissions could be procured. Most workloads follow the diurnal cycle, so at any given time anywhere between 20 and 80% of capacity is sitting idle. That's far more than you need to do most kinds of deep learning research. That's why you see work from them that would cost you $100K+ to replicate end-to-end (e.g. neural architecture search). At Google you get those resources for basically free, so it's not a huge issue to just spin up a rack full of TPUs. That's just how things are done over there. I don't get your opposition to introducing approximate exaflops spent in training as an additional metric by which to judge "goodness" of a model. It is well known that some models are much easier to train than others. It's also known that some optimizers converge faster than others. All this stuff directly affects reproducibility of research. We're getting to the point where some of the papers aren't reproducible unless you're at Google or DeepMind or OpenAI or Microsoft Research. I think you'll agree that this is pretty terrible. Between two models with approximately the same validation accuracy metrics and inference timings wouldn't you choose the one that's more economical to train? I know I would. Why aren't these metrics even mentioned most of the time? Beats me. Even better if the model requires less _data_ to train, but that seems to be less tractable than just estimating and surfacing the cost of training.
- gwern 7y ago
  From what I understand from other Googlers, there are internal markets designed to track and allocate compute resources; this is necessary because of their expense, and because of the low marginal value of many possible uses. > I worked at Google for many years. Google has so much idle compute at any given moment, they tend not to care _at all_ about you using it. It's not a problem for a Google employee to just grab a few thousand cores on a whim and do whatever they want with those cores, and many people do use this freedom. And it works out for them, does it not? Because the uses are useful on net, even if some of them seem speculative or silly or inefficient. If they weren't, if people were just constantly burning resources to do absolutely nothing, presumably things would change and those resources would be tracked by the markets too or other measures taken. Let Google as a whole decide where to optimize, end to end - don't push it onto individual researchers and guilt-trip them into worrying about whether they are 'green AI' or evil people who hate the planet. > I think you'll agree that this is pretty terrible. I would not. I think it's awesome that machine learning is seeing the tiniest crumbs of the resources other fields have (still grossly disproportionate to its potential) invested in it, and we are no longer in a field where all results are reproducible on a $200 gaming GPU run overnight, and this is delivering big gains in performance and surprising nontrivial demonstrations of scalability (which would not be true of, say, IBM Watson). Again, as I said, imagine applying this sort of standard to any other field of science. 'You can't discover the Higgs boson using just 250 watts and a $200 interferometer you bought off eBay, I think you'll agree that this is pretty terrible.' Reproducibility doesn't mean 'should be reproduced, frequently, by anyone, at arbitrarily small budgets'. > Between two models with approximately the same validation accuracy metrics and inference timings wouldn't you choose the one that's more economical to train? No, I'd choose the one with the lowest total cost for my purposes. Do you program only in assembler to make sure you write programs with only the most economical runtime costs? What a narrow way of seeing costs... I'm often much more concerned with which one is easier to modify, or to scale, or to reuse for transfer learning, or which runs acceptably for users. The model which is 'more economical to train' is only better on one of many axis, and when someone else like OA or DM pays for it, it is a sunk cost of zero relevance to me. (How much does it matter to me how long it took GPT-2 to train? Not very much! Because I download the trained model and finetune it for other things.) > Why aren't these metrics even mentioned most of the time? They typically are. Most DL papers will mention roughly the GPU/TPU count and wallclock time, or you can guess from the training graph or baselines. And that's really all you need to know as you evaluate trainability as one of many cost variables; stuff like CO2 emissions or flops are not.

by gwern 7y ago

m0zg 7y ago

I worked at Google for many years. Google has so much idle compute at any given moment, they tend not to care _at all_ about you using it. It's not a problem for a Google employee to just grab a few thousand cores on a whim and do whatever they want with those cores, and many people do use this freedom. No permission is required if your tasks are preemptible (which by design they mostly are, at Google). I'm not sure if the same applies to TPUs (I left before they became prevalent) but I'm sure for the right subset of people the right permissions could be procured. Most workloads follow the diurnal cycle, so at any given time anywhere between 20 and 80% of capacity is sitting idle. That's far more than you need to do most kinds of deep learning research. That's why you see work from them that would cost you $100K+ to replicate end-to-end (e.g. neural architecture search). At Google you get those resources for basically free, so it's not a huge issue to just spin up a rack full of TPUs. That's just how things are done over there. I don't get your opposition to introducing approximate exaflops spent in training as an additional metric by which to judge "goodness" of a model. It is well known that some models are much easier to train than others. It's also known that some optimizers converge faster than others. All this stuff directly affects reproducibility of research. We're getting to the point where some of the papers aren't reproducible unless you're at Google or DeepMind or OpenAI or Microsoft Research. I think you'll agree that this is pretty terrible. Between two models with approximately the same validation accuracy metrics and inference timings wouldn't you choose the one that's more economical to train? I know I would. Why aren't these metrics even mentioned most of the time? Beats me. Even better if the model requires less _data_ to train, but that seems to be less tractable than just estimating and surfacing the cost of training.

gwern 7y ago
From what I understand from other Googlers, there are internal markets designed to track and allocate compute resources; this is necessary because of their expense, and because of the low marginal value of many possible uses. > I worked at Google for many years. Google has so much idle compute at any given moment, they tend not to care _at all_ about you using it. It's not a problem for a Google employee to just grab a few thousand cores on a whim and do whatever they want with those cores, and many people do use this freedom. And it works out for them, does it not? Because the uses are useful on net, even if some of them seem speculative or silly or inefficient. If they weren't, if people were just constantly burning resources to do absolutely nothing, presumably things would change and those resources would be tracked by the markets too or other measures taken. Let Google as a whole decide where to optimize, end to end - don't push it onto individual researchers and guilt-trip them into worrying about whether they are 'green AI' or evil people who hate the planet. > I think you'll agree that this is pretty terrible. I would not. I think it's awesome that machine learning is seeing the tiniest crumbs of the resources other fields have (still grossly disproportionate to its potential) invested in it, and we are no longer in a field where all results are reproducible on a $200 gaming GPU run overnight, and this is delivering big gains in performance and surprising nontrivial demonstrations of scalability (which would not be true of, say, IBM Watson). Again, as I said, imagine applying this sort of standard to any other field of science. 'You can't discover the Higgs boson using just 250 watts and a $200 interferometer you bought off eBay, I think you'll agree that this is pretty terrible.' Reproducibility doesn't mean 'should be reproduced, frequently, by anyone, at arbitrarily small budgets'. > Between two models with approximately the same validation accuracy metrics and inference timings wouldn't you choose the one that's more economical to train? No, I'd choose the one with the lowest total cost for my purposes. Do you program only in assembler to make sure you write programs with only the most economical runtime costs? What a narrow way of seeing costs... I'm often much more concerned with which one is easier to modify, or to scale, or to reuse for transfer learning, or which runs acceptably for users. The model which is 'more economical to train' is only better on one of many axis, and when someone else like OA or DM pays for it, it is a sunk cost of zero relevance to me. (How much does it matter to me how long it took GPT-2 to train? Not very much! Because I download the trained model and finetune it for other things.) > Why aren't these metrics even mentioned most of the time? They typically are. Most DL papers will mention roughly the GPU/TPU count and wallclock time, or you can guess from the training graph or baselines. And that's really all you need to know as you evaluate trainability as one of many cost variables; stuff like CO2 emissions or flops are not.