4 ms·
I would like to have some expert feedback on how meaningful this line of research is. The paper is quite straightforward, but the implications are unclear to me
by hcks 5y ago
I would like to have some expert feedback on how meaningful this line of research is. The paper is quite straightforward, but the implications are unclear to me. Is there a chance that backpropagation is superseded by this method for some use cases ? Another question is, if this result is as interesting as it seems to me, how come this is only published in 2022 given the past research effort in the area of deep learning ?
- beecafe 5y agoIt's not that novel, they're just computing the "directional derivative" in a random direction. It's similar to the research OpenAI has done on Evolution Strategies - basically, trading off a more noisy gradient but getting optimization with less inter process communication, achieving scale. This work is somewhere in between ES and normal backprop. Usually progress moves in a see-saw fashion, with scaling up (and the techniques needed) bringing improvement, then they becoming compressed/found unnecessary as the real cause is found, etc..
- iandanforth 5y agoI'm sure you know this but for others ES is gradient free. Population sampling for optimization is distinct from 'gradient based' techniques in the literature.