3 ms·
> is ultimately a silly and misguided proposal No its not. Its analogous to quantifying an F test with watts. Nothing wrong with that. I can give a billion r
by dxbydt 7y ago
> is ultimately a silly and misguided proposal
No its not.
Its analogous to quantifying an F test with watts.
Nothing wrong with that. I can give a billion row dataset to a dozen people. John assumes normality, samples 100 rows, runs a linear regression, gets 70% R^2, under 1 second on a 1980s era computer. Mary doesn't assume normality, runs a glm via irwls, gets better explanatory power than John, still on 1980s computer, though she takes 5 seconds instead of 1. Then James comes along & runs a decision tree & is twice as good as Mary, but he now needs a second on a 1990s PC. Tony uses a random forest & Baker wants a 128 layer neural net. And so on & so forth, until we end up with burning enough energy that would otherwise power a village, to overfit some dataset that excel at some completely artificial leaderboard metric.
At some point a grown up comes in & says you don't need to light up your bedroom with industrial stadium lighting if you need to do your homework, a 40w table lamp will suffice. Hell half the third world uses a candle & they are performing quite well, if not better. That's really all this is.
The marginal gains in performance aren't justified compared to the amount of energy you end up expending. Half these models are brittle & have zero shelf life, ppl regularly throw away stuff they wrote just a year ago, so to what raison d'etre this pursuit ?
- JimmyAustin 7y agoThe cure isn't proscribing "green ai", it's adding a carbon tax that embeds the cost of the carbon externality into the cost of power in general, then letting the free market make the decision.
- mantoto 7y agoMoney is optimizing this already as the highest invest is computing resources.
- speedplane 7y ago> The marginal gains in performance aren't justified compared to the amount of energy you end up expending. Come on, we’ve been through this before. At least for the past 10 years if not much longer, performance improvements have not actually improved performance, they’ve improved developer productivity. If you create a language that’s half the speed, but developers can program twice as fast in it, it makes sense to use it. Processing power is much cheaper than a $150k+ developer. Also, being first to market can yield outsized gains on competitors.
- gwern 7y ago> to overfit some dataset that excel at some completely artificial leaderboard metric. This comment exemplifies the myopic narrowminded focus on costs and being dismissive of the benefits of greater resource investments which I am criticizing here. Said 'completely artificial' tasks like ImageNet actually turn out to transfer really well to the real world and are why AI is becoming so useful, and also spurred the research into how to create efficient NNs via distillation and compression which still cannot be done without training the large models first. You need to invest if you want efficiency. Performance precedes it, it doesn't follow it. As the programming saying goes: get it correct, then get it fast.