3 ms·

> There is innovation in discovering an architecture that makes it possible to learn P(next_word|previous_words), in addition to the computing techniques and ha

> There is innovation in discovering an architecture that makes it possible to learn P(next_word|previous_words), in addition to the computing techniques and hardware improvements required to make it work. Isn't that essentially what mjburgess said in the parent post? > LLMs are a hardware innovation: they make it possible for GPUs to compute this at scale across TBs of data... The algorithm isnt doing anything other than aligning computation to hardware

nerdponx 3y ago
Not really, and no. Torch and CUDA align computation to hardware. If it were just a matter of doing that, we would be fine with fully-connected MLP. And maybe that would work with orders of magnitude more data and compute than we currently throw at these models. But we are already pushing the cutting edge of those things to get useful results out of the specialized architecture. Choosing the right NN architecture is like feature engineering: the exact details don't matter that much, but getting the right overall structure can be the difference between learning a working model and failing to learn a working model, from the same source data with the same information content. Clearly our choice of inductive bias matters, and the transformer architecture is clearly an improvement over other designs. Surely you wouldn't argue that a CNN is "just" aligning computation to hardware, right? Transformers are clearly showing themselves as a reliably effective model architecture for text in the same way that CNNs are reliably effective for images.
- mjburgess 3y ago
  Err... no. MLPs are fundamentally sequential algorithms (backprop weight updating). All major innovations in NN design have been to find ways of designing the architecture to fit GPU compute paradigms. It was an innovation, in the 80s, to map image structure to weight structure that underpins CNNs. That isnt what made CNNs trainable though.. that was alexnet, and just go read the paper... its pretty upfront about how the NN architecture is designed to fit the GPU... that's the point of it
- rsfern 3y ago
  There’s some interesting work replacing scaled dot product attention and position embeddings with fixed format MLPs [0] - so I tend to lean towards thinking of classic transformers as having a reasonable enough inductive bias and the scalability to actually realize the amount of compute that’s needed 0: https://arxiv.org/abs/2105.08050 https://arxiv.org/abs/2105.08050

nerdponx 3y ago

Not really, and no. Torch and CUDA align computation to hardware. If it were just a matter of doing that, we would be fine with fully-connected MLP. And maybe that would work with orders of magnitude more data and compute than we currently throw at these models. But we are already pushing the cutting edge of those things to get useful results out of the specialized architecture. Choosing the right NN architecture is like feature engineering: the exact details don't matter that much, but getting the right overall structure can be the difference between learning a working model and failing to learn a working model, from the same source data with the same information content. Clearly our choice of inductive bias matters, and the transformer architecture is clearly an improvement over other designs. Surely you wouldn't argue that a CNN is "just" aligning computation to hardware, right? Transformers are clearly showing themselves as a reliably effective model architecture for text in the same way that CNNs are reliably effective for images.

mjburgess 3y ago
Err... no. MLPs are fundamentally sequential algorithms (backprop weight updating). All major innovations in NN design have been to find ways of designing the architecture to fit GPU compute paradigms. It was an innovation, in the 80s, to map image structure to weight structure that underpins CNNs. That isnt what made CNNs trainable though.. that was alexnet, and just go read the paper... its pretty upfront about how the NN architecture is designed to fit the GPU... that's the point of it
rsfern 3y ago
There’s some interesting work replacing scaled dot product attention and position embeddings with fixed format MLPs [0] - so I tend to lean towards thinking of classic transformers as having a reasonable enough inductive bias and the scalability to actually realize the amount of compute that’s needed 0: https://arxiv.org/abs/2105.08050 https://arxiv.org/abs/2105.08050