3 ms·

I think OP was talking more about upending the universal dogma of "statistical optimization to minimize feedforward loss functions via gradient descent." We now

by gcr 4y ago

I think OP was talking more about upending the universal dogma of "statistical optimization to minimize feedforward loss functions via gradient descent." We now have plenty more tricks of the trade (ReLUs were a big one, self-attention was another, making things convolutional was a third, and the smaller ones like dropout/batchnorm/LR schedules). However, in broad strokes, this paradigm remains relatively unchanged since LeCunn+Bengio+Hinton in the 80s. The fundamental idea informally stretches back to Rosenblatt's perceptrons in the 50s, which itself was built on McCulloch and Pitts' modeling of neurons as multiply-accumulate units from 1943. We've spent lots of time questioning these assumptions (e.g. branching off into Boltzmann machines or spiking neurons, which are really fascinating), but to my knowledge, there haven't been promising alternatives yet. We have "something that appears to work" but is this itself a local minimum? Biology found its own efficient processes that look a bit different after all. In my opinion, "fundamental breakthroughs" would answer questions like: - is there an inherent reason that humans learn through a process akin to statistical optimization, or is it a coincidence that feedforward-only gradient descent seems to work? - Are there other models for "neurons" that go beyond GEMM, like spikey activations? We know from biology that they're nonlinear; how might they be modeled better? - Are there ways of training effective agents beyond just the reinforcement Q-learning techniques that we've settled on? - What does it even mean for a model to be able to "reason;" does our current view of recursive models with hidden state really map to that philosophy one-to-one or is there more?

adammarples 4y ago
But he has a very good point. Transformers are such a quantum leap over anns, it's almost a whole new paradigm, Karpathy calls them a brand new way to do computation
- gcr 4y ago
  Right, but transformers are artificial neural networks though, so it's not clear to me what you mean. The leap from RNNs to the Transformer architecture in 2017ish is similar to vision networks' leap from fully-connected layers to convolutional layers in 1987ish. In both cases, the key is to change the architecture to exploit some implicit structure in the data that wasn't quite used effectively before. For images, each pixel is strongly related to its close neighbors, so convolution is a natural way to simplify models while capturing that locality. For language, the opposite is true - almost any part of the text implicitly reference anything that's been said previously, so you need representations that are somewhat position-independent. Approaches that can model that structure naturally work better than an RNN munching tokens one by one. Each one was certainly a leap forward that unlocked its respective field, but at the end of the day, it's "just" an architecture tweak, you know?

by gcr 4y ago

adammarples 4y ago

But he has a very good point. Transformers are such a quantum leap over anns, it's almost a whole new paradigm, Karpathy calls them a brand new way to do computation

gcr 4y ago
Right, but transformers are artificial neural networks though, so it's not clear to me what you mean. The leap from RNNs to the Transformer architecture in 2017ish is similar to vision networks' leap from fully-connected layers to convolutional layers in 1987ish. In both cases, the key is to change the architecture to exploit some implicit structure in the data that wasn't quite used effectively before. For images, each pixel is strongly related to its close neighbors, so convolution is a natural way to simplify models while capturing that locality. For language, the opposite is true - almost any part of the text implicitly reference anything that's been said previously, so you need representations that are somewhat position-independent. Approaches that can model that structure naturally work better than an RNN munching tokens one by one. Each one was certainly a leap forward that unlocked its respective field, but at the end of the day, it's "just" an architecture tweak, you know?