From what I understand from other Googlers, there are internal markets designed to track and allocate compute resources; this is necessary because of their expe

From what I understand from other Googlers, there are internal markets designed to track and allocate compute resources; this is necessary because of their expense, and because of the low marginal value of many possible uses. > I worked at Google for many years. Google has so much idle compute at any given moment, they tend not to care _at all_ about you using it. It's not a problem for a Google employee to just grab a few thousand cores on a whim and do whatever they want with those cores, and many people do use this freedom. And it works out for them, does it not? Because the uses are useful on net, even if some of them seem speculative or silly or inefficient. If they weren't, if people were just constantly burning resources to do absolutely nothing, presumably things would change and those resources would be tracked by the markets too or other measures taken. Let Google as a whole decide where to optimize, end to end - don't push it onto individual researchers and guilt-trip them into worrying about whether they are 'green AI' or evil people who hate the planet. > I think you'll agree that this is pretty terrible. I would not. I think it's awesome that machine learning is seeing the tiniest crumbs of the resources other fields have (still grossly disproportionate to its potential) invested in it, and we are no longer in a field where all results are reproducible on a $200 gaming GPU run overnight, and this is delivering big gains in performance and surprising nontrivial demonstrations of scalability (which would not be true of, say, IBM Watson). Again, as I said, imagine applying this sort of standard to any other field of science. 'You can't discover the Higgs boson using just 250 watts and a $200 interferometer you bought off eBay, I think you'll agree that this is pretty terrible.' Reproducibility doesn't mean 'should be reproduced, frequently, by anyone, at arbitrarily small budgets'. > Between two models with approximately the same validation accuracy metrics and inference timings wouldn't you choose the one that's more economical to train? No, I'd choose the one with the lowest total cost for my purposes. Do you program only in assembler to make sure you write programs with only the most economical runtime costs? What a narrow way of seeing costs... I'm often much more concerned with which one is easier to modify, or to scale, or to reuse for transfer learning, or which runs acceptably for users. The model which is 'more economical to train' is only better on one of many axis, and when someone else like OA or DM pays for it, it is a sunk cost of zero relevance to me. (How much does it matter to me how long it took GPT-2 to train? Not very much! Because I download the trained model and finetune it for other things.) > Why aren't these metrics even mentioned most of the time? They typically are. Most DL papers will mention roughly the GPU/TPU count and wallclock time, or you can guess from the training graph or baselines. And that's really all you need to know as you evaluate trainability as one of many cost variables; stuff like CO2 emissions or flops are not.