7 ms·

If this were written today, Karpathy would have to call it "The Unreasonable Effectiveness of Convolutions". Since 2015, convolutions, causal or dilated convolu

• phowon 8y ago
  Can you elaborate on how Transformers are "convolutions with attention"?
  • epberry 8y ago
    I found this to be a great read on what I think the parent is referring to: https://lilianweng.github.io/lil-log/2018/06/24/attention-attention.html#transformer https://lilianweng.github.io/lil-log/2018/06/24/attention-at...
    • phowon 8y ago
      But no part of that Transformer section makes any reference to convolutions.
    • gwern 8y ago
      Self-attention uses 1x1 convolutions, and while the original Transformer is fully-connected-only (I think?) the latest & greatest uses convolutions (https://arxiv.org/abs/1901.11117 https://arxiv.org/abs/1901.11117), and even if you're using the original, it's very often being used in conjunction with convolutions elsewhere. So you could argue that the real title would have to be 'The Unreasonable Effectiveness of Attention', but given all the other stuff convolutions have done without attention like WaveNet, I think it's currently fairer to go with convolutions than attention. But we'll see how it goes over the next 4 years...
      • phowon 8y ago
        A 1x1 convolution is such an edge case of convolution that it's really not worth discussing its inclusion as related to the success of the Transformer. Calling the Transformer "convolutions with attention" demonstrates a new-complete misunderstanding of the architecture. There's a reason Transformer's original paper is entitled "Attention is All You Need", because it throw out all the previous structures people assumed were necessarily to solving these problems (recurrence from RNNs, local-transformations for convolutions) and just threw multiple layers of large multi-headed attentions at the problem and got even better results.
        • gwern 8y ago
          > A 1x1 convolution is such an edge case of convolution that it's really not worth discussing its inclusion It is, nevertheless, still a convolution and calls to convolution code is how self-attention is implemented. Look inside a SAGAN or something and you'll see the conv2d calls. > Calling the Transformer "convolutions with attention" demonstrates a new-complete misunderstanding of the architecture. You're reading that in an overly narrow way and imputing to me something I didn't mean. And again, the original Transformer may not be used in conjunction with convolutions, but it often is, and the best current variant uses convolutions internally and so involves convolutions no matter how you want to slice it. Attention is a powerful construct, but convolutions are pretty darn powerful too, it turns out, even outside images.
          • phowon 8y ago
            >Look inside a SAGAN or something and you'll see the conv2d calls. ...Yes, because SAGANs operate on images, so the foundational operation is a convolution. >You're reading that in an overly narrow way and imputing to me something I didn't mean. You characterized the Transformer as "convolutions with attention". You then attributed the success of Transformer-based models to "the non-locality & easy optimization of convolutions". The "SOTA for most (all?) sequence-related tasks" applies the regular Transformer variants, not the Evolved Transformer which was published about 5 days ago. No one is denying that convolutions are useful across many domains. But no one seriously working in the domain of NLP would consider convolutions to be anywhere near the most novel or notable parts of the Transformer. (In case you do want to look it up, OpenAI's GPT also uses character-level convolutions for its word embeddings. However, BERT does not.)
• stochastic_monk 8y ago
  Most benchmarks in [0] use LSTMs. [0] https://nlpprogress.com https://nlpprogress.com
  • gwern 8y ago
    Not sure how up to date that is. How many of those things currently listed as SOTA have been compared to, say, BERT baselines?
    • carbocation 8y ago
      Under their language modeling section, they include the Transformer and have listed some papers from 2019, so it seems to be reasonably up to date.
      • stochastic_monk 8y ago
        Furthermore. transformer xl extends the convolutional architecture to use some recurrence. I think that claims that RNNs have been wholesale replaced by convolutions are premature.
        • carbocation 8y ago
          I can't speak for 'gwern, but I was also under the impression that CNNs had basically replaced RNNs for these tasks. I think my view is colored by Jeremy Howard's FastAI talks. Since he isn't big on publishing papers (though that is changing a bit via collaborations with his students), it may be true that the state of the art is no longer best represented by published papers. (I would argue that avoiding this ambiguity is an argument in favor of publication.)
          • stochastic_monk 8y ago
            It’s common to use CNNs in practice because they get you most of the way and are so much cheaper to train/run. They are, however, limited in context to the width of the receptive field.
• swairshah 8y ago
  I don't understand what you mean by 'non-locality of convolutions'. Isn't convolution inherently a local operation? This probably being one of the main reasons that CNNs are biased towards texture [0] and not shapes? [0] https://openreview.net/forum?id=Bygh9j09KX https://openreview.net/forum?id=Bygh9j09KX
  • gwern 8y ago
    Convolutions in a hierarchy of layers, especially with dilated convolutions, provide long-range connections between inputs (handwavily logarithmic). In an RNN, they are separated by however many steps in a linear way, so gradients more easily vanish. Some paper which I do not recall examined them side by side and found that RNNs quickly forget inputs, even with LSTMs, and this means their theoretically unlimited long-range connections between inputs via their hidden state don't wind up being that useful.
    • trott 8y ago
      In an RNN, you could connect each hidden state at time step t, h(t) to h(t-N), instead of, or in addition to, h(t-1), making it analogous to dilated convolutions, but with hidden-to-hidden connections at the same layer. So I don't think RNNs are fundamentally more myopic than CNNs (just that there may be practical advantages to using the latter) Hierarchical RNNs, Clockwork RNNs and Hierarchical Multiscale RNNs and probably others are doing things of this nature.
      • gwern 8y ago
        You could, but it's not equivalent, and no one seems to have been able to use clockwork RNNs or related archs to achieve similar performance, so the differences would seem to make a difference.
        • trott 8y ago
          Right. I'm just saying that this myopia is not a fundamental property of the recurrence any more than of convolution. Clockwork RNNs subsample, BTW, so they are more analogous stride=2 in CNNs than to dilation.
• mrcoder111 8y ago
  How do you handle variable length input without something like an RNN? Even transformers use RNN structures right. I suppose convolutions could technically handle variable length inputs (just slide the window of weights over different length inputs) but I don't think tensorflow or pytorch supports this
  • phowon 8y ago
    >Even transformers use RNN structures right. Nope. >How do you handle variable length input without something like an RNN? Any form of pooling, really. Max, Avg, Sum. The tricky part is how to do the pooling while still taking advantage of the sequential structure of the input information. The Transformer -based models have shown that you can get away with providing very little order information and still go very far.
    • mrcoder111 8y ago
      Samcodes said it above. How do transformers build a shared representation of two input sentences with different lengths? If you convolve them with the same filter, you get two different sized convolution outputs - the embedding dimensions don't align.
      • phowon 8y ago
        Like I said - pooling. You can take the mean over 3 elements or 10 elements just the same. Pooling is lossy, but it seems that if you have the right architecture the model can still learn what it needs to. It's worth noting that the attention mechanism (at least in RNNs) has always been invariant to inputs lengths. It's a weighted sum with weights computed per element, so there's no length constraint at all.
        • mrcoder111 8y ago
          Can you share some paper names or links to architectures that demonstrate the length invariant convolution and attention?