3 ms·

Fergus Lab at NYU. I believe he's across the hall from Yann LaCunn as well ;) Still a long way from a Theory of Biogenesis. But a good next step is using a dif

Fergus Lab at NYU. I believe he's across the hall from Yann LaCunn as well ;) Still a long way from a Theory of Biogenesis. But a good next step is using a differentiable model to predict novel proteins which have no analogue in Nature. Much like Materials Genome researchers searching for stable phases of matter! "Training ever bigger convnets and LSTMs on ever bigger datasets gets us closer to Strong AI -- in the same sense that building taller towers gets us closer to the moon." --François Chollet

eganist 8y ago
> I believe he's across the hall from Yann LaCunn as well ;) I'm having a hard time processing what the wink might possibly mean in this context. No sarcasm intended.
- mkolodny 8y ago
  I'd guess that wink is hinting that Yann LeCun might've had something to do with this research. Whether that's true or not, I have no idea. (Yann LeCun is a Turing award winner for his work in deep learning)
  - slashcom 8y ago
    Yann LeCun did not, otherwise he’d be a coauthor. As it is, this was a collaboration between NYU and Facebook AI Research, with multiple authors working at both institutions.
    - drb91 8y ago
      My understanding is that academic authorship credit is political: authors don’t always contribute, and contributors don’t always get credit. Is this not the case?
      - nl 8y ago
        Not really. Usually the politics goes the other way - people getting an author slot because they are the head of the department or something.
        drb91 8y ago
        I believe I covered that—authors who don’t contribute. Also, I know too many angry grad students to believe that contributors always get credit.
visarga 8y ago
> "Training ever bigger convnets and LSTMs on ever bigger datasets gets us closer to Strong AI -- in the same sense that building taller towers gets us closer to the moon." --François Chollet The Transformer layer has radically leaped over LSTMs and CNNs. While LSTMs can model sequences and CNNs regular grids, they have no efficient long range interaction mechanism. Transformer does. It's a huge leap similar to the one in computer vision from a few years ago. What is needed besides spatial translation invariance (CNN) and temporal invariance (LSTM) is permutation invariance. Whenever the problem can be described as a graph, then the ordering of the vertices and edges should not matter. You can't do that with CNNs and LSTMs, but you can do it with Graph neural nets and Transformers. Apparently Transformers are the best for language modelling (GPT-2), playing games (Dota2 from OpenAI), composing music and possibly now in modelling proteins. I assume they will play a huge role in working with graph structured data, with multiple entities and relations.
- nl 8y ago
  It's not really as clear cut as that. Transformers work well in sequence tasks because both compare well in terms of accuracy but also scale better than a RNNs like a LSTM or a GRU. That means they can be trained on more data. This isn't really the same as CNNs, where they model images by running at different scales. I'm not aware of any cases of Transformers being used particularly successfully on images. They can be used on graphs of course, by translating the problem into a graph walk problem (ala DeepWalk). All the examples you gave (language modelling, Dota2, music and protein modelling) are setup as sequence prediction problems, so are perfect for Transformers.
  - p1esk 8y ago
    https://arxiv.org/abs/1904.09925 https://arxiv.org/abs/1904.09925
    - nl 8y ago
      Nice. I guess I'm 8 days behind on the SOTA... But I'd note that it is build on top of a CNN base (ResNet or RetinaNet) and that the Attention-only system performed slightly worse than the one including the CNN layers. Also, this isn't really a Transformer architecture, even though it uses Attention. But maybe this is too much nitpicking? I agree that Attention is a useful primitive - my point is that the Transformer architecture is too specific. (Also, this is a really nice paper in that it lays out the hyperparameters and training schedules they used. And that Appendix is amazing!)

eganist 8y ago

> I believe he's across the hall from Yann LaCunn as well ;) I'm having a hard time processing what the wink might possibly mean in this context. No sarcasm intended.

mkolodny 8y ago
I'd guess that wink is hinting that Yann LeCun might've had something to do with this research. Whether that's true or not, I have no idea. (Yann LeCun is a Turing award winner for his work in deep learning)
- slashcom 8y ago
  Yann LeCun did not, otherwise he’d be a coauthor. As it is, this was a collaboration between NYU and Facebook AI Research, with multiple authors working at both institutions.
  - drb91 8y ago
    My understanding is that academic authorship credit is political: authors don’t always contribute, and contributors don’t always get credit. Is this not the case?
    - nl 8y ago
      Not really. Usually the politics goes the other way - people getting an author slot because they are the head of the department or something.
      - drb91 8y ago
        I believe I covered that—authors who don’t contribute. Also, I know too many angry grad students to believe that contributors always get credit.

visarga 8y ago

> "Training ever bigger convnets and LSTMs on ever bigger datasets gets us closer to Strong AI -- in the same sense that building taller towers gets us closer to the moon." --François Chollet The Transformer layer has radically leaped over LSTMs and CNNs. While LSTMs can model sequences and CNNs regular grids, they have no efficient long range interaction mechanism. Transformer does. It's a huge leap similar to the one in computer vision from a few years ago. What is needed besides spatial translation invariance (CNN) and temporal invariance (LSTM) is permutation invariance. Whenever the problem can be described as a graph, then the ordering of the vertices and edges should not matter. You can't do that with CNNs and LSTMs, but you can do it with Graph neural nets and Transformers. Apparently Transformers are the best for language modelling (GPT-2), playing games (Dota2 from OpenAI), composing music and possibly now in modelling proteins. I assume they will play a huge role in working with graph structured data, with multiple entities and relations.

nl 8y ago
It's not really as clear cut as that. Transformers work well in sequence tasks because both compare well in terms of accuracy but also scale better than a RNNs like a LSTM or a GRU. That means they can be trained on more data. This isn't really the same as CNNs, where they model images by running at different scales. I'm not aware of any cases of Transformers being used particularly successfully on images. They can be used on graphs of course, by translating the problem into a graph walk problem (ala DeepWalk). All the examples you gave (language modelling, Dota2, music and protein modelling) are setup as sequence prediction problems, so are perfect for Transformers.
- p1esk 8y ago
  https://arxiv.org/abs/1904.09925 https://arxiv.org/abs/1904.09925
  - nl 8y ago
    Nice. I guess I'm 8 days behind on the SOTA... But I'd note that it is build on top of a CNN base (ResNet or RetinaNet) and that the Attention-only system performed slightly worse than the one including the CNN layers. Also, this isn't really a Transformer architecture, even though it uses Attention. But maybe this is too much nitpicking? I agree that Attention is a useful primitive - my point is that the Transformer architecture is too specific. (Also, this is a really nice paper in that it lays out the hyperparameters and training schedules they used. And that Appendix is amazing!)