3 ms·

This concern is vague, frequently raised, and not especially useful. What are you advocating, specifically? That the amount of compute being thrown at problems

This concern is vague, frequently raised, and not especially useful. What are you advocating, specifically? That the amount of compute being thrown at problems is excessive relative to the economic benefit that Google (or whoever) derives from training models as long as they do? This is an empirical question, and it’s well understood how much compute is useful. Are you saying that advances in M.L. theory are coming too slowly because we focus too much on hardware? I spend maybe a third of my day job reading journal articles, and the rate of ideas coming out is blistering. This isn’t people banging GPUs together in desperation because nothing works. This is what happens when everything works better and better the more money you throw at it with no apparent end.

s_Hogg 7y ago
I accept all of what you're saying and also spend a lot of my day reading ML papers :) To boil it down a little perhaps, the point I'm making is that the incremental gain from new ideas coming out seems to be slowly decreasing, in my view - which is perfectly natural! It obviously makes sense that that would happen. Where that thought leads me though, is the idea that we may be slowly moving towards the exploitation half of the explore/exploit choice. I'm not saying greenfield ML research is gone by any means, only that I think a lot of stuff presented as such may in fact not be because in many cases it doesn't really fundamentally re-think the problem being addressed.
whoevercares 7y ago
Deep NN surely gives people a lot of spaces to come up or combine ideas from other domain. The problem is people take what NN could do for granted without really look into the details. As long as you could invent some fancy architecture and interpret your loss, no one cares the black box part. The higher than usual portion of system that lacks of theoretical foundation is indeed worrying.

s_Hogg 7y ago

I accept all of what you're saying and also spend a lot of my day reading ML papers :) To boil it down a little perhaps, the point I'm making is that the incremental gain from new ideas coming out seems to be slowly decreasing, in my view - which is perfectly natural! It obviously makes sense that that would happen. Where that thought leads me though, is the idea that we may be slowly moving towards the exploitation half of the explore/exploit choice. I'm not saying greenfield ML research is gone by any means, only that I think a lot of stuff presented as such may in fact not be because in many cases it doesn't really fundamentally re-think the problem being addressed.

whoevercares 7y ago

Deep NN surely gives people a lot of spaces to come up or combine ideas from other domain. The problem is people take what NN could do for granted without really look into the details. As long as you could invent some fancy architecture and interpret your loss, no one cares the black box part. The higher than usual portion of system that lacks of theoretical foundation is indeed worrying.