4 ms·
Ugh, exactly, it's so cool. I've been a deep learning practitioner for ~3 years now, and I feel like this notion has really been impressed upon me only recently
by captainclam 2y ago
Ugh, exactly, it's so cool. I've been a deep learning practitioner for ~3 years now, and I feel like this notion has really been impressed upon me only recently.
I've spent an awful lot of mental energy trying to conceive of how these things work, when really it comes down to "does increasing this parameter improve the performance on this task? Yes? Move the dial up a bit. No? Down a bit..." x 1e9.
And the cool part is that this yields such rich, interesting, sometimes even useful, structures!
I like to think of this cognitive primitive as the analogue to the idea that thermodynamics is just the sum of particles bumping into each other. At the end of the day, that really is just it, but the collective behavior is something else entirely.
- xanderlewis 2y ago> At the end of the day, that really is just it, but the collective behavior is something else entirely. Exactly. It’s not to say that neat descriptions like this are the end of the story (or even the beginning of it). If they were, there would be no need for this entire field of study. But they are cool, and can give you a really clear conceptualisation of something that can appear more like a sum of disjoint observations and ad hoc tricks than a discipline based on a few deep principles.
- JackFr 2y agoNAND gates by themselves are kind of dull, but it's pretty cool what you can do with a billion of them.
- kadushka 2y agoit comes down to "does increasing this parameter improve the performance on this task? Yes? Move the dial up a bit. No? Down a bit..." x 1e9 This is not how gradient based NN optimization works. What you described is called "random weight perturbation", a variant of evolutionary algorithms. It does not scale to networks larger than a few thousand parameters for obvious reasons. NNs are optimized by directly computing a gradient which tells us the direction to go to to reduce the loss on the current batch of training data. There's no trying up or down and seeing if it worked - we always know which direction to go. SGD and RWP are two completely different approaches to learning optimal NN weights.
- xanderlewis 2y agoI don’t think the author literally meant tweaking the parameters and seeing what happens; it’s probably an analogy meant to give a sense of how the gradient indicates what direction and to what degree the parameters should be tweaked. Basically, substitute ‘the gradient is positive’ for ‘increasing this parameter decreases performance’ and vice versa and it becomes correct.
- p1esk 2y agoThat substitution is the main difference between SGD and RWP. It’s like describing bubble sort when you meant to describe quick sort. Would not fly on an ML 101 exam, or in an ML job interview.
- xanderlewis 2y agoIt’s not like that at all. You couldn’t accidentally sound like you’re describing quick sort when describing bubble sort, or vice versa. I can’t think of any substitution of a few words that would do that. The meaning of the gradient is perfectly adequately described by the author. They weren’t describing an algorithm for computing it.
- a_random_canuck 2y agoI don’t think anyone is trying to pass an exam here, but just give an understandable overview to a general audience.
- captainclam 2y agoI guess you could say I don't know RWP from Adam! :D My og comment wasn't to accurately explain gradient optimization, I was just expressing a sentiment not especially aimed at experts and not especially requiring details. Though I'm afraid I subjected you to the same "cringe" I experience when I read pop sci/tech articles describe deep learning optimization as "the algorithm" being "rewarded" or "punished," haha.
- 2y ago